What is MediaEval?
MediaEval is a benchmarking initiative dedicated to evaluating new algorithms for multimedia access and retrieval. It emphasizes the 'multi' in multimedia and focuses on human and social aspects of multimedia tasks. MediaEval attracts participants who are interested in multimodal approaches to multimedia involving, e.g., speech recognition, multimedia content analysis, user-contributed information (tags, tweets), viewer affective response, social networks, temporal and geo-coordinates.

How can I get involved?
MediaEval is an open initiative, meaning that any interested research group is free to signup and participate. Groups sign up for one or more tasks, they then receive task definitions, data sets and supporting resources, which they use to develop their algorithms. At the very end of the summer, groups submit their results and in the fall they attend the MediaEval workshop. See also Why Participate? or watch some video on the MediaEval video page.

What is a MediaEval task?
A MediaEval task consists of four parts:
  • A data set provided to the benchmark participants,
  • A task definition that describes the problem to be solved,
  • Ground truth against which participants’ algorithms are evaluated,
  • An evaluation metric.
MediaEval tasks are oriented towards user needs in specific application settings and, to the extent possible, are based on scenarios of use derived from real-world problems.

What is the difference between a main MediaEval task and a "Brave New Task"?
  • Brave New Tasks are "closed tasks", which do not open their sign-up to the research community at large. Rather, they are by invitation only (write an email to the organizers if you are interested).
  • Brave New Tasks are not held to run on the schedule of MediaEval main tasks. The only deadline that they are forced to comply to is the deadline for submitting camera ready working notes papers. (This is the reason for which they are by invitation only.)
  • Brave New Tasks have a single presentation at the MediaEval workshop in the Brave New Tasks session. (In contrast, main tasks have entire sessions.)
Note that if a task runs as Brave New Task in one year, there is no implication that the task will run as a main task the next year. For this reason, we choose the designation 'Brave New Task' rather than 'pilot'. The purpose of Brave New Tasks is to allow MediaEval to take risks, while still ensuring that all main tasks run robustly.

Who runs MediaEval?
MediaEval is a community-drive benchmark that is run by the MediaEval organizing committee consisting of the task organizers of all the individual task in a given year. The overall coordination is carried out by Martha Larson and Gareth Jones, who founded MediaEval in 2008 as VideoCLEF, a track in the CLEF Campaign. MediaEval became an independent benchmarking initiative in 2010 under the auspices of the PetaMedia Network of Excellence. In 2012, it also received support from ICT Labs of EIT. In 2012, MediaEval became associated with the Cubrik project, but enjoys complete autonomy. Cubrik keeps us in Rubik's cubes that we distribute at the end of the workshop to participants who have made particularly outstanding contributions.

Can I get MediaEval data from past years? MediaEval data is generally available from past years. However, the release of the past year's data is delayed until after the next year's tasks have been decided. Currently, we are working on making the 2011 data from selected tasks public. Data that requires a license (for example, Tagging Task Professional 2010, Affect Task 2010 and LInking Task 2009) is difficult for us to release, but please contact us (m.a.larson at tudelft.nl) if you are intent on using this data and we can put you in contact with the licensers.

Is MediaEval the same as TRECVid?
No, not at all. It's true that both benchmarks work with video. But in fact, MediaEval takes great pains to complement rather than duplicate tasks running in the TRECVid video retrieval evaluation. MediaEval runs tasks that emphasize the human and social aspects of multimedia and focuses on exploiting speech, context and user-contributed information.

How are the MediaEval tasks chosen?
We chose tasks by first formulating a list of proposed tasks and then ascertaining which tasks are popular enough to run using a yearly survey. If you or your project would like to propose or organize a task in future years, please contact Martha Larson (m.a.larson at tudelft.nl).

Are there intellectual property rights (IPR) issues with the data?
MediaEval uses Creative Commons data wherever possible. When we need to make an exception we take pains to ensure that participants can license data through the appropriate channels.

For more information contact Martha Larson m.a.larson (at) tudelft.nl.