Tutorial: Multimedia Information Retrieval

Instructor: Prof. Dr. Stefan Rüger
Date (Time): 24 Aug 2016 (2:30pm)
Venue: TBD

This tutorial will examine the challenges and opportunities of Multimedia Information Retrieval, ie, image, video and music search engines.

Computer technology has changed our access to information tremendously: We used to search authors or titles (which we had to know) in library cards in order to locate relevant books; now we can issue keyword searches within the full text of unimaginably large book repositories to identify the authors, titles and locations of relevant resources. What about the corresponding challenge of finding multimedia by fragments, examples and excerpts? Rather than asking for a music piece by artist and title, can we hum its tune to find it? Can doctors submit scans of a patient to identify medically similar images of diagnosed cases in a database? Can your mobile phone take a picture of a statue and return to you resources about its artist via a service that it sends this picture to?

Some of the challenges of these questions are given by the semantic gap between what computers can index and high-level human concepts; related to this is an inherent technological limitation of automated annotation of images from pixels alone. Other challenges are given by polysemy, ie, the many meanings and interpretations that are inherent in visual material and the corresponding wide range of a user’s information need.

This tutorial will demonstrate how these challenges can be tackled by automated processing and machine learning and by utilising the skills of the user, for example through harnessing and directing browsing activities with “relevance feedback”, thus putting the user centre stage. Other automated processing methods that discover and utilise world knowledge in the form of wikipedia (an online, linked, multilingual and open content encyclopedia) will be shown to not only improve multimedia retrieval but also give surprising insights into the human nature.

At the end of the tutorial you will be able to

  • Outline challenges brought about by multimedia collections and their interface needs
  • Describe the principle components of a multimedia document access system and how they differ from other retrieval systems, most notably text information retrieval systems commonly used in Digital Library systems
  • Describe and implement simple visual feature vectors commonly used in image search engines
  • Explain the workings of an image search engine and relate this to other mono-medium types