Lora Aroyo is research scientist at Google Research, NYC where she works in the area of Responsible AI specifically focussing on research for Data Excellence, e.g. metrics and strategies to measure quality of human-labeled data in a reliable and transparent way. 

Lora is an active member of the Human Computation, User Modeling & Semantic Web communities. She is president of the User Modeling community UM Inc, which serves as a steering committee for the ACM Conference Series “User Modeling, Adaptation and Personalization” (UMAP) sponsored by SIGCHI and SIGWEB. She is also a member of the ACM SIGCHI conferences board

Prior to joining Google, Lora was a computer science professor at the VU University Amsterdam. Since 2010 she has actively worked towards shaping the concept of “User-Centric Data Science“, which ultimately led to the forming of and her heading the User-centric Data Science group at the Department of Computer Science, Vrije Universiteit Amsterdam, The Netherlands and further extending collaborations on this as a visiting scholar at the Columbia Data Science Institute at Columbia University. 

Her team invented the CrowdTruth crowdsourcing method and applied in various domains such as digital humanities, medical and online multimedia. She is a four times holder of IBM Faculty Award for her work on CrowdTruth: for crowdsourcing ground truth data in the context of adapting the IBM Watson system to the medical domain and applying CrowdTruth for capturing ambiguity for the purpose of understanding misinformation.

logo_crowdtruth-02 (1)

As a Chief Scientist at a NY-based startup Tagasauris (currently seeen.com) Lora guided the human-in-the-loop strategies as part of the hybrid machine learning and human-assisted computing platform for multimedia enrichment (e.g. video, images, and text) with meaningful information about its content, and ultimately improve video search and discovery.

As an expert in user-centric data science, Lora conceived the vision of an user-centric experimental lab for computer science researchers at the VU University Amsterdam. She headed the team that made it possible in 2010 to open VU INTERTAIN Lab – the first of its kind in an academic environment. Throughout her career, Lora was a principal investigator of a large number of research projects, she organized conferences, workshops, and tutorials to bring together methods and tools from human computation, linked (open) data, data science & human-computer interaction with the goal of building hybrid human-AI systems for augmenting both machine and human intelligence for understanding text, images, and videos with humans-in-the-loop and machines-in-the-loop. Her research projects focussing on semantic search, recommendation systems, personalized access to online multimedia collections have a major impact and established her as a recognized leader in human computation techniques for specific domains, such as digital humanities, cultural heritage, and interactive TV.

Tim Berners-Lee visit at the opening of the Interntain Lab at the VU University Amsterdam

understanding ambiguity with humans in the loop:  teaching machines to deal with ambiguity in text, image & video processing;

augmenting intelligence: improving interpretation abilities of text, images, and videos with humans-in-the-loop and machines-in-the-loop;

hybrid human-AI systems: harnessing the power of the crowd and AI to improve recommendation systems, semantic search, access to online multimedia collections in domains like digital humanities, cultural heritage, and interactive TV.

Research projects, where Lora Aroyo was a principal investigator (PI):

  • ReTV: Reinventing the TV for the Digital Age: Re-purposing digital content across Smart TVs, Web and mobile applications, social media and other emerging platforms
  • Capturing Bias: Diversity-aware Analysis of Bias in News Videos: models for bias- and diversity-aware accuracy measures for reliable and explainable data analysis
  • CrowdTruth: disgreement-based metrics for quality beyond ground truth data
    • Dr. Watson: Gamification of Ground Truth Collection for Medical Texts
    • Crowd-Watson: Framework for Crowdsourcing Ground Truth Data
    • Crowd Truth: Metrics to Evaluate Crowdsourced Ground Truth Data
  • CLARIAH: Common Lab Research Infrastructure for the Arts and Humanities
  • DIVE: Event-centric Exploration of Linked Heritage
  • Accurator: Annotating Fashion with Nichesourcing
  • SealincMedia:  Socially-enriched Access to Linked Cultural Media
  • ControCurator: discover and understand controversial topics and events
  • VISTA-TV: Combining LOD and behavioral information for TV analyses
  • PrestoPrime: WAISDA? Crowdsourcing Game for Video Annotation
  • NoTube: integration of Web and TV data with the help of semantics
  • CHIP: Cultural Heritage Information Personalization

Check also:

Posted in crowdsourcing, home, keynote | Leave a comment

Stitch by Stitch: Annotating Fashion at the Rijksmuseum

Fashion can be found everywhere in museums. Fashion heritage collected over centuries: costumes, accessories, paintings, prints and photographs. But while some clothes and accessories are easily found and identified, others are obscure and require a trained eye to describe. What are we looking at? What kind of sleeve is this? Which materials and techniques have been used? More specific descriptions of the images facilitate better use of digital collections and enable users to wander through them in detail.

Modemuze is an online platform and network of 11 Dutch museums, including Rijksmuseum, with a fashion and costume collection: Amsterdam Museum, Centraal Museum Utrecht, Fries Museum Leeuwarden, Gemeentemuseum Den Haag, Museum Rotterdam, Paleis Het Loo, Rijksmuseum, Tassenmuseum Hendrikje, TextielMuseum, Theatercollectie Bijzondere Collecties UvA, Tropenmuseum, Afrika Museum, Museum Volkenkunde.

Annotating the collections

Researchers from VU University Amsterdam, Delft University of Technology and the Centre for Mathematics and Informatics and the Rijksmuseum (in the context of the COMMIT SealincMedia project) have developed Accurator: an online tool to improve the process of annotation of digital collection objects, e.g. being able to find relevant objects to annotate, annotate specific parts of an object, etc. Following ‘Birdwatching in the Rijksmuseum’, this time the Accurator tool will be used to describe fashion related objects from the Modemuze and Rijksmuseum collections.

Participants in the fashion annotation event are also invited to record their findings in the Wikipedia Encyclopedia, Wikimedia Commons and in Wikidata, Wikipedia’s open database. Wikipedia volunteers, as well as staff from the Rijksmuseum and Modemuze, will be present for support throughout the day.

Participation in the event is free, but registration is required. To register, please send an email to accurator@rijksmuseum.nl with your name and your interest in fashion. (We will take your subject preferences into account when setting up the Accurator tool.) If you have any questions regarding the event, please feel free to email them to this address.

Additional information

Posted in crowdsourcing, event, poster | Leave a comment

Exploration is the New Search @ SXSW2017

Posted in event | Leave a comment

Slides from our TEDx Navesink 2015 talk

Posted in conference, crowdsourcing, event, keynote, presentation | Leave a comment

The CrowdTruth Journey @VU Faculty Colloquium

Posted in event | 1 Comment

Posted in event | Leave a comment

A twitter overview of the IUI2015


Posted in Uncategorized | 1 Comment

Crowd Truth, public release v.1

Last week, 13-Jun-2014, we published the first public release of the Crowd Truth framework. The CrowdTruth description document will give you an overall view of the framework components. The CrowdTruth Guidelines document will help you logging in and using the system. On github you can find the CrowdTruth Code and future updates.

Please, be aware that this is still work in progress, and not all things work perfectly – so don’t give up right away and send us email with feedback, so that we can repair the bugs. If you have any problems using it, please contact us Lora Aroyo and Chris Welty and we will be happy to help you through the system.


Posted in Uncategorized | Leave a comment

Crowd-Watson @NLeSC eHumanities Kick-off

Today at the NLeSc, during the “De Geest Uit De Fles” event six new projects kick off their research in the area of eHumanities. As part of this, the Crowd-Watson project team will collaborate with NleSC researchers in the coming 12 months for the development of the next version of the Dr. Watson, a medical detective nichesourcing game. Here you can find my slides on “Crowds & Niches Teaching Machines to Diagnose”:

Posted in Uncategorized | Leave a comment

Posted in Uncategorized | Leave a comment