The task has concluded at the data has been released. Please see MediaEval Datasets.
The 2013 Social Event Detection Task
The Social Event Detection (SED) task of MediaEval 2013 requires participants to discover event-related multimedia and organize them in event-specific clusters, within a collection of Web multimedia.
We define social events as events that are planned by people, attended by people and the media illustrating the events are captured by people. A social event of interest can be specified in terms of event-related metadata (e.g., location, time, venue, and performers), example tags or other social information, example media items (images), or a combination of the above.
Two challenges will be specified as part of the 2013 SED task:
The first challenge will be a completely data-driven one, which involves analysis of a large-scale dataset, requiring participants to produce a complete clustering of the image/video dataset according to events. The task is a supervised clustering task where a set of training event classes is provided. However, these classes are not used during evaluation. This challenge will not specify a particular event or event class of interest. This is a novelty compared to the previous two editions of SED.
The second challenge in 2013 will be a supervised classification task, which requires learning how event-related media items look like (both in terms of visual content and accompanying metadata). More specifically, a set of eight event types are defined, and methods should automatically decide, to which type (if any) an unknown media item belongs.
Target group
The task is of interest to researchers in the areas of information retrieval, multimedia content analysis, social media analysis, event detection and event-based multimedia indexing.
Use Scenario
As social media applications proliferate, an ever-increasing amount of web and multimedia content available on the Web is being created. A lot of this content is related to social events, which we define as events that are organized and attended by people and are illustrated by social media content created by people. For users, finding digital content related to a social event of interest is a difficult task, requiring to search large volumes of data, possibly at different sources and sites. Algorithms that can support humans in this task are clearly needed.
The task thus consists in developing algorithms that can detect event-related media and group them by the events they illustrate or are related to. Such a grouping would provide the basis for aggregation and search applications that foster easier discovery, browsing and querying of social events.
Data
The data set comprises the URLs of Web images (ca. 400.000 images) and possibly also some videos. These media items are accompanied by their metadata. This metadata includes time-stamps, geographic information, tags, title, description, etc. (in an XML format). As it is a real world dataset, there are some features like time-stamps and uploader information which are available for every picture, but there are also features (like geographic information) which are available for only a subset of the images.
Ground truth and evaluation
Ground truth information will record the true media-event associations and will be generated by the organizers. The ground truth is single label, meaning that no image can belong to more than one event. The results of event-related media item detection will be evaluated using Precision-Recall-F-Score and Normalized Mutual Information (NMI). Both will be used to assess the overlap between clusters and classes. Furthermore, all evaluation measures will also be reported in an adjusted form called “Divergence from a Random Baseline” [4], which indicates how much useful learning has occurred and helps detect problematic clustering submissions.
Recommended reading
[1] S. Papadopoulos, E. Schinas, V. Mezaris, R. Troncy, I. Kompatsiaris, "Social Event Detection at MediaEval 2012: Challenges, Dataset and Evaluation", Proc. MediaEval 2012 Workshop, Pisa, Italy, October 2012.
[2| G. Petkos, S. Papadopoulos, Y. Kompatsiaris. “Social Event Detection using Multimodal Clustering and Integrating Supervisory Signals.” In Proceedings of ACM International Conference on Multimedia Retrieval (ICMR), Hong Kong, 2012
[3] Reuter, Timo, and Philipp Cimiano. "Event-based classification of social media streams." Proceedings of the 2nd ACM International Conference on Multimedia Retrieval. ACM, 2012.
[4] De Vries, Christopher M., Shlomo Geva, and Andrew Trotman. "Document clustering evaluation: Divergence from a random baseline." Workshop Information Retrieval, Dortmund, Germany. 2012.
Task organizers
Raphael Troncy, Eurecom, France
Vasileios Mezaris, CERTH-ITI, Greece
Philipp Cimiano, Bielefeld University, Germany
Timo Reuter, Bielefeld University, Germany
Shlomo Geva, Queensland Univ. of Technology, Australia
Symeon Papadopoulos, CERTH-ITI, Greece
Task auxiliaries
Christopher de Vries, Queensland Univ. of Technology, Australia
Task schedule
2 May: Development data release
3 June: Test data release
13 September: Run submission due
20 September: Results returned
28 September: Working notes paper deadline
