This task builds on the foundation established by the MediaEval 2012-2014 Search and Hyperlinking tasks. It is not necessary, however, to have participated in Search and Hyperlinking in order to successfully participate in SAVA.
The underlying general use scenario for the MediaEval 2015 SAVA task is a popular information seeking strategy executed millions of times every day: users search for relevant information using a query and then navigate within their target collection starting from the retrieved relevant content. In particular, we envision that users search and navigate within an archive of professional TV broadcasts or semi-professional user generated content.
Two subtasks will be offered to the participants, within the video hyperlinking framework:
- Search for multimedia content: promotes the development of search methods that use multiple modalities (e.g., speech, visual content, speaker’s emotions etc) to answer a search query. The retrieval units are video segments of unrestricted size. Similar to the earlier MediaEval 2013 Search & Hyperlinking edition of this sub-task, participants will be provided with a two-fielded query, where one field refers to spoken content and the other refers to the visual content of relevant segments. Participants can use either or both fields to find video segments within the collection. We will provide automatic speech recognition output and output from visual analysis tools, e.g., detection scores of visual concepts.
- Automatic anchor selection task: explores methods to automatically identify anchors for a given set of videos, where anchors are media fragments (with their boundaries defined by their start and end time) for which users could require additional information. What constitutes an anchor depends on the video, e.g., in a news programme it could be a mention of persons, while and in a documentary it could be the view of particular buildings. Participants will be provided with 5-10 videos of different types and requested to automatically generate anchors within these videos.
Target group
Research groups that are interested in multimedia linking or recommendation applications, personalised storytelling within multimedia (video) collections, or multimedia analysis in the context of these types of applications, for example with a specific focus on segmentation or on new types of features that are relevant for these applications.
Data
The task will again make use of the MediaEval 2014 Search and Hyperlinking dataset. The data is in English, and contains: the videos, manually created BBC metadata, subtitles, automatic speech recognition transcripts, and output of visual analysis. The participants will be required to sign a licensing agreement with BBC in order to access the data that will be available via beehub.nl.
Ground truth and evaluation
We ask users from a media professionals community (e.g. journalists) to generate queries for which relevant videos existing in the collection. For a subset of the relevant segments, these users will define anchors from which they would like to have a hyperlink to further information.
For the search sub-task, based on the top ranked documents in a (prioritized) sub-set of those runs, we will ask Amazon Mechanical Turk workers to judge whether the returned video segments are relevant to the query. We plan to keep using our current set of precision-oriented metrics used in the Search and Hyperlinking task in 2014 (e.g., binned version of precision at rank 10 [3]), but closer to the data release we will adjust.
For the anchor generation sub-task, participants submissions will be compared against a set of anchors defined by target end-users of the application scenario. The evaluation metric is still under development. Submissions will potentially be assessed based on segmentation appropriateness and anchor relevance to the overall video. Optionally, anchors provided by participants are assessed as well, based on e.g., segmentation-appropriateness and anchor-relevance.
Recommended reading
Collection Description:
[1] Eskevich, M. and Aly, R. and Racca, D.N. and Ordelman, R.J.F. and Chen, S. and Jones, G.J.F. The Search and Hyperlinking Task at MediaEval 2014. In Proceedings of the MediaEval 2014 Workshop, Barcelona, Spain, 2014.
Evaluation:
[2] Aly, R. and Eskevich, M. and , Ordelman, R.J.F. and Jones, G.J.F. Adapting Binary Information Retrieval Evaluation Metrics for Segment-based Retrieval Tasks. In Technical report, ArXiv e-prints, 1312.1913
[3] Aly, R. and Ordelman, R.J.F. and Eskevich, M. and Jones, G.J.F. and Chen, S. Linking Inside a Video Collection - What and How to Measure? LiME workshop at the 22nd International Conference on World Wide Web Companion, IW3C2 2013, May 13-17, 2013, Rio de Janeiro, pp. 457-460.
Task history:
[4] Eskevich, M. and Jones, G.J.F. and Aly, R. and Ordelman, R.J.F. and Chen, S. and Nadeem, D. and Guinaudeau, C. and Gravier, G. and Sébillot, P. and de Nies, T. and Debevere, P. and Van de Walle, R. and Galuscakova, P. and Pecina, P. and Larson, M. Multimedia Information Seeking Through Search and Hyperlinking. Proceedings of the 3rd ACM Conference on International Conference on Multimedia Retrieval (ICMR '13), Dallas, Texas, USA, 2013, pp 287--294
Other related work:
[5] Mihalcea, R., Csomai, A. Wikify! Linking Documents to Encyclopedic Knowledge. In Proceedings of ACM CIKM Conference on Information and Knowledge Management. ACM, Lisbon, Portugal, 2007, 233-242.
[6] Larson, M., Newman, E., Jones, G.J.F. Overview of VideoCLEF 2009: New Perspectives on Speech-based Multimedia Content Enrichment. In Proceedings of ACM CLEF International Conference on Cross-Language Evaluation Forum: Multimedia experiments. ACM, Springer-Verlag Berlin, Heidelberg, 2009, 354-368.
[7] Eskevich, M., Jones, G. J.F. Exploring Speech Retrieval from Meetings using the AMI Corpus. In Proceedings of ISCA International Speech Communication Association, Computer Speech and Language, Special Issue Info Extraction & Retrieval, 2014.
Task organizers
Maria Eskevich, EURECOM, France (maria.eskevich@eurecom.fr) (“contact person”)
Robin Aly, University of Twente, Netherlands (r.aly@ewi.utwente.nl)
Shu Chen, Dublin City University, Ireland (shu.chen4@mail.dcu.ie)
David N. Racca, Dublin City University, Ireland (dracca@computing.dcu.ie)
Gareth Jones, Dublin City University, Ireland (Gareth.Jones@computing.dcu.ie)
Roeland Ordelman, University of Twente and Netherlands Institute for Sound & Vision, a.k.a. Beeld & Geluid, Netherlands (ordelman@ewi.utwente.nl)
Task auxiliaries
Benoit Huet, EURECOM, France (benoit.huet@eurecom.fr)
Task schedule
1 May: Development data release
1 June: Test data release
17 July: Run submission due
14 August: Results returned to the participants
21 August: Working notes paper deadline
14-15 September: MediaEval 2015 Workshop
Acknowledgments
CNGL/ADAPT, Ireland
COMMIT, Netherlands