Chris Welty and I have presented our idea (at WebSci2013) on how to harness the disagreement between crowd text annotators in order to build gold standard data, which is closer to how people interpret relations between medical terms in text. You can read a nice trip reports on the WebSci2013 and CHI2013 conferences by my colleagues Paul Groth and Victor de Boer. Read the reactions on twitter for the event.

Check out also for updates on the data publishing for these results @figshare and a related paper on extraction of events from news papers.

Posted on by Lora Aroyo | Leave a comment

My selection of WebSci2013

Here you can find some of the posters at WebSci2013. You can also read a nice trip reports on the WebSci2013 and CHI2013 conferences by my colleagues Paul Groth and Victor de Boer.


Posted in conference, poster | Leave a comment

Agora Digital Hermeneutics @ WebSci2013

At the WebSci2013 I presented our most recent work on how to use events to provide meaning to cultural collection objects. We designed an evaluation framework for online access to cultural heritage, which enables the assessment of online cultural heritage applications in terms of their provision and support of information and interpretation. It is anchored in digital hermeneutics: the study and theory of the Web as a vehicle of (self)-interpretation. Digital hermeneutics considers the limits of automation and modelling on the one hand, and the interaction of people and technology on the other. We analyzed twelve Web applications, representing the range of current state of the art in this eld. This provides valuable insights into what cultural heritage applications on the Web do, can do, and how distinctive goals are to be achieved. We also reported on three user studies with the Agora demonstrator which made us reconsider a number of assumptions we made about the user’s needs for information and interpretation. Read the reactions on twitter for the event.

You can find the data and the paper @figshare. Check out other related publications.

Posted in conference, paper, presentation | Leave a comment

IBM Tech Report: Harnessing Disagreement in Crowdsourcing a Relation Extraction Gold Standard

The initial results from our October 2012 experiments got accepted as IBM technical report. We have presented updated results at the Web Science 2013 Conference, which cover also experiments until March 2013. Soon we will be publishing a new sets of results for the experiments performed between mainly in the period March – May 2013, which will also include a full set of metrics on them. [link]

Posted in paper | Tagged , , , , | Leave a comment

IBM Faculty Award 2012

Lora Aroyo won a 2012 IBM Faculty Award, which was handed out the 10th of December 2012 by Gerard Smit, CTO IBM Benelux. The Faculty Awards are presented annually by IBM for extraordinary activities in the field of IT research and innovation. Lora Aroyo won the IBM Faculty Award for her innovative work on crowdsourcing the process of (medical) text annotation to further enable AI systems, such as Watson, to deal with diversity in human expressions and interpretations. As part of this event, Chris Welty, IBM Research, NY also talked about “The importance of Metrics for Watson & Healthcare”.

Making Watson smarter, especially in the context of healthcare, is a very exciting opportunity to add our expertise in harnessing the crowd to IBM’s expertise in analytics and NLP” (Lora Aroyo). 

“Interestingly, a lot of the errors made by AI Systems like Watson are very easy for humans to detect. The crowdsourcing work done by Dr. Aroyo helps to overcome these errors, by bringing human knowledge into the machine. This awards stresses both the quality and significance of the research done by Dr. Aroyo at the VU University Amsterdam.” (Robert-Jan Sips, IBM University Relations, NL.

The IBM Faculty Awards are part of IBM’s global University Programs. These awards are intended to stimulate the crossover of academic knowledge to society. The awards are highly competitive. In 2012 IBM granted 200 awards globally, of which 3 in the Netherlands.

Posted in Awards | Tagged , , , | Leave a comment

Harnessing Disagreement for Event Semantics @DERiVE2012

Chris Welty and I presented at the Derive2012 workshop, in conjunction with ISWC2012 Conference our paper focussing on how events can be detected & extracted from natural language text, and how those are represented for use on the semantic web. We draw an inspiration from the similarity between crowdsourcing approaches for tagging and text annotation task for ground truth of events. Thus, we propose a novel approach that harnesses the disagreement between the human annotators by defining a framework to capture and analyze the nature of the disagreement. We expect two novel results from this approach. On the one hand, achieving a new way of measuring ground truth (performance), and on the other hand identifying a new set of semantic features for learning in event extraction.

Other work on events extraction in the history domain with cultural heritage data I am doing in the context of the Agora project.

You can find the data and the paper @figshare

Posted in paper, presentation | Leave a comment