Content-Based Image and Information Retrieval

The scientifically challenging objective of content-based image retrieval (CBIR) is the autonomous adaptation to the user's preferences and conceptions regarding the relevance and similarity of images. This can be achieved by examining the user's reactions on the retrieval results in the form of relevance feedback.

Main research areas

We have recently developed a framework for content-based information retrieval research named the PicSOM system. The research has been and will be three-fold: algorithm development, international evaluations and applications. We will continue to study the basic principles of effective algorithms that implement the Query by Example paradigm in connection with relevance feedback from the user.

We will also study other paradigms such as query by sketch and query by pointing. These modalities will be required in our joint multimodal interfaces initiative. A key concept in the methodological developments will be the utilization of automatic segmentation or location of interest points in the images or other information objects. After the segmentation, relevance feedback from the user and other forms of auxiliary or cross-modal information can be focused on specific subparts of the objects.

Various real-life image retrieval tasks will be studied further. We have already attained very good results in a joint project with ABB, in which the PicSOM system has been applied to recognition of defects in running paper and metal webs.

Future plans

We will continue to broaden the studied domain to include diverse multimodal forms of information, e.g. sound, video, and various multi-part data types. For such experiments we have already incorporated the standardized MPEG-7 visual descriptors in the PicSOM system. Of special interest to us is the interplay between different modalities and how annotations and relevance assessments can be extrapolated over the modality boundary. Specific topics on which we will focus in the years 2006-2008 include the following:

Study a larger variety of indexing techniques including e.g. the Evolving Tree method developed in our laboratory, a variant of the SOM.
Develop methods for using Bayesian ICA models in content-based retrieval.
Develop keyword focusing in which automatic image segmentation is used to find out which specific parts of an image contain the visual objects denoted by the keywords associated to the image.
Study relevance focusing, where the relevance assessments, originally given to entire images, are automatically focused in image segments.
Study sparse annotations, where only a small part of an information database will contain textual annotations or keywords which the content-based information retrieval system will extrapolate to the rest of the database.

Recent publications

2008

Ville Viitaniemi and Jorma Laaksonen. Techniques for image classification, object detection and object segmentation applied to VOC Challenge 2007. Technical Report 2, Department of Information and Computer Science, Helsinki University of Technology (TKK), 2008. [pdf]

2007

Zhirong Yang, Zhijian Yuan, and Jorma Laaksonen. Projective non-negative matrix factorization with applications to facial image processing. International Journal on Pattern Recognition and Artificial Intelligence, 2007.
Zhirong Yang and Jorma Laaksonen. Multiplicative updates for non-negative projections. Neurocomputing, 2007. To appear.
Ville Viitaniemi and Jorma Laaksonen. Improving the accuracy of global feature fusion based image categorisation. In Proceedings of the 2nd International Conference on Semantic and Digital Media Technologies (SAMT 2007), Lecture Notes in Computer Science, pages 1-14, Genova, Italy, December 2007. Springer. To be printed.
Markus Koskela, Mats Sjöberg, Ville Viitaniemi, Jorma Laaksonen, and Philip Prentis. PicSOM experiments in TRECVID 2007. In Proceedings of the TRECVID 2007 Workshop, Gaithersburg, MD, USA, November 2007.
Markus Koskela, Mats Sjöberg, Jorma Laaksonen, Ville Viitaniemi, and Hannes Muurinen. Rushes summarization with self-organizing maps. In Proceedings of the TRECVID Workshop on Video Summarization (TVS'07), pages 45-49, Augsburg, Germany, September 2007. ACM Press.
Jorma Laaksonen, Markus Koskela, Mats Sjöberg, Ville Viitaniemi, and Hannes Muurinen. Video summarization with SOMs. In Proceedings of the 6th Int. Workshop on Self-Organizing Maps (WSOM 2007), Bielefeld, Germany, 2007. Available online at http://biecoll.ub.uni-bielefeld.de/volltexte/2007/154/index_en.html.
Philip Prentis. Galsom - colour-based image browsing and retrieval with tree-structured self-organising maps. In Proceedings of the 6th Int. Workshop on Self-Organizing Maps (WSOM 2007), Bielefeld, Germany, 2007. Available online at http://biecoll.ub.uni-bielefeld.de/volltexte/2007/164/index_en.html.
Zhirong Yang and Jorma Laaksonen. Face recognition using Parzenfaces. In Proceedings of International Conference on Artificial Neural Networks (ICANN'07), volume 4669 of Lecture Notes in Computer Science, pages 200-209, Porto, Portugal, September 2007. Springer.
Markus Koskela, Alan F. Smeaton, and Jorma Laaksonen. Measuring concept similarities in multimedia ontologies: Analysis and evaluations. IEEE Transactions on Multimedia, 9(5):912-922, August 2007.
Zhirong Yang and Jorma Laaksonen. Approximated geodesic updates with principal natural gradients. In Proceedings of The 2007 International Joint Conference on Neural Networks (IJCNN 2007), pages 1320-1325, Orlando, USA, August 2007.
Ville Viitaniemi and Jorma Laaksonen. Evaluating the performance in automatic image annotation: example case by adaptive fusion of global image features. Signal Processing: Image Communications, 22(6):557-568, July 2007.
Markus Koskela and Alan F. Smeaton. An empirical study of inter-concept similarities in multimedia ontologies. In Proceedings of the 6th ACM international conference on Image and video retrieval (CIVR 2007), pages 464-471, Amsterdam, The Netherlands, July 2007. ACM Press.
Zhirong Yang and Jorma Laaksonen. Regularized neighborhood component analysis. In Proceedings of 15th Scandinavian Conference on Image Analysis (SCIA), pages 253-262, Aalborg, Denmark, 2007.
Hannes Muurinen and Jorma Laaksonen. Video segmentation and shot boundary detection using self-organizing maps. In Proceedings of 15th Scandinavian Conference on Image Analysis (SCIA 2007), pages 770-779, Aalborg, Denmark, June 2007.
Ville Viitaniemi and Jorma Laaksonen. Evaluating performance of automatic image annotation: example case by fusing global image features. In Proceedings of Fifth International Workshop on Content-Based Multimedia Indexing (CBMI 2007), pages 251-258, Bordeaux, France, June 2007.
Ville Viitaniemi and Jorma Laaksonen. Empirical investigations on benchmark tasks for automatic image annotation. In Proceedings of the 9th International Conference on Visual Information Systems (VISUAL 2007), Lecture Notes in Computer Science, pages 96-107. Springer, June 2007. To be printed.
Matthieu Molinier, Jorma Laaksonen, and Tuomas Häme. Detecting man-made structures and changes in satellite imagery with a content-based information retrieval system built on self-organizing maps. IEEE Transactions on Geoscience and Remote Sensing, 45(4):861-874, April 2007.
Hannes Muurinen. Video segmentation and shot boundary detection using self-organizing maps. Master's thesis, Laboratory of Computer and Information Science, Helsinki University of Technology, 2007.

2006

Ville Viitaniemi and Jorma Laaksonen. Use of image regions in context-adaptive image classification. In Yannis Avrithis, Steffen Staab, and Noel O'Connor, editors, Proceedings of the 1st International Conference on Semantic and Digital Media Technologies (SAMT 2006), Lecture Notes in Computer Science, pages 169-183, Athens, Greece, December 2006. Springer. [pdf]
Ville Viitaniemi and Jorma Laaksonen. Focusing keywords to automatically extracted image segments using self-organising maps. In Soft Computing in Image Processing: Recent Advances (eds. Mike Nachtegael, Dietrich Van der Weken, Etienne E. Kerre and W. Philips), chapter 5, pages 121-156. Springer Verlag, 2006. [pdf]
Matthieu Molinier, Jorma Laaksonen, and Tuomas Häme. Self-organising maps for change detection and monitoring of human activity in satellite imagery. In Proceedings of ESA-EUSC 2006: Image Information Mining for Security and Intelligence, Madrid, Spain, November 2006.
Mats Sjöberg, Hannes Muurinen, Jorma Laaksonen, and Markus Koskela. PicSOM experiments in TRECVID 2006. In Proceedings of the TRECVID 2006 Workshop, Gaithersburg, MD, USA, November 2006.
Jorma Laaksonen and Ville Viitaniemi. Emergence of ontological relations from visual data with self-organizing maps. In Timo Honkela, Tapani Raiko, Jukka Kortela, and Harri Valpola, editors, Proceedings of the 9th Scandinavian Conference on Artificial Intelligence (SCAI 2006), pages 31-38, Espoo, Finland, October 2006.
Ville Viitaniemi and Jorma Laaksonen. Techniques for still image scene classification and object detection. In Proceedings of 16th International Conference on Artificial Neural Networks (ICANN 2006), volume 2, pages 35-44, Athens, Greece, September 2006. Springer.
Mats Sjöberg, Jorma Laaksonen, Matti Pöllä, and Timo Honkela. Retrieval of multimedia objects by combining semantic information from visual and textual descriptors. In Proceedings of 16th International Conference on Artificial Neural Networks (ICANN 2006), pages 75-83, Athens, Greece, September 2006. Springer. [pdf]
Zhirong Yang and Jorma Laaksonen. A fast fixed-point algorithm for two-class discriminative feature extraction. In Proceedings of 16th International Conference on Artificial Neural Networks (ICANN 2006), Part II, pages 330-339, Athens, Greece, September 2006. Springer. [pdf]
Erkki Oja, Mats Sjöberg, Ville Viitaniemi, and Jorma Laaksonen. Emergence of semantics from multimedia databases. In Gary Y. Yen and David B. Fogel, editors, Computational Intelligence: Principles and Practice, chapter 9. IEEE Computational Intelligence Society, 2006.
Matthieu Molinier, Jorma Laaksonen, and Tuomas Häme. A self-organizing map framework for detection of man-made structures and changes in satellite imagery. In Proceedings of IEEE International Geoscience And Remote Sensing Symposium, Denver, Colorado, August 2006.
Markus Koskela, Alan F. Smeaton, and Georgina Gaughan. Semantic analysis of concept models for news videos. Workshop on Visual Categorisation and Image Management Systems (VCIMS 2006), June 2006.
Markus Koskela and Alan F. Smeaton. Clustering-based analysis of semantic concept models for video shots. In International Conference on Multimedia & Expo (ICME 2006), Toronto, Canada, July 2006.
Mats Sjöberg, Ville Viitaniemi, Jorma Laaksonen, and Timo Honkela. Analysis of semantic information available in an image collection augmented with auxiliary data. In Proceedings of 3rd IFIP Conference on Artificial Intelligence Applications and Innovations, pages 600-608, Athens, Greece, June 2006. IFIP, Springer.
Markus Koskela and Jorma Laaksonen. Semantic concept detection from news videos with self-organizing maps. In Proceedings of 3rd IFIP Conference on Artificial Intelligence Applications and Innovations, pages 591-599, Athens, Greece, June 2006. IFIP, Springer.
Mark Everingham, Andrew Zisserman, and Christopher K. I. Williams et al. The 2005 PASCAL visual object classes challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognizing Textual Entailment, pages 117-176. Springer, 2006.

2005

Markus Koskela, Jorma Laaksonen, Mats Sjöberg, and Hannes Muurinen. PicSOM experiments in TRECVID 2005. In Proceedings of the TRECVID 2005 Workshop, pages 267-270, Gaithersburg, MD, USA, November 2005. [pdf]
Ville Viitaniemi and Jorma Laaksonen. Keyword-detection approach to automatic image annotation. In Proceedings of 2nd European Workshop on the Integration of Knowledge, Semantic and Digital Media Technologies (EWIMT 2005), London, UK, November 2005. [pdf]
Matthieu Molinier, Jorma Laaksonen, Jussi Ahola, and Tuomas Häme. Self-organizing map application for retrieval of man-made structures in remote sensing data. In Proceedings of ESA-EUSC 2005: Image Information Mining - Theory and Application to Earth Observation, Frascat, Italy, October 2005. [pdf]
Mats Sjöberg and Jorma Laaksonen. Content-based retrieval of web pages and other hierarchical objects with Self-Organizing Maps. In Proceedings of 15th International Conference on Artificial Neural Networks (ICANN 2005), pages 841-846 , Warsaw, Poland, September 2005.
Zhirong Yang and Jorma Laaksonen. Partial relevance in interactive facial image retrieval. In Proceedings of 3rd International Conference on Advances in Pattern Recognition (ICAPR 2005), pages 216-225, Bath, UK, August 2005.
Markus Koskela and Jorma Laaksonen. Semantic annotation of image groups with Self-Organizing Maps. In Proceedings of 4th International Conference on Image and Video Retrieval (CIVR 2005), pages 518-527, Singapore, July 2005.
Zhirong Yang and Jorma Laaksonen. Approximated classification in interactive facial image retrieval. In Proceedings of 14th Scandinavian Conference on Image Analysis (SCIA 2005), pages 770-779, Joensuu, Finland, June 2005.
Jorma Laaksonen, Ville Viitaniemi, and Markus Koskela. Application of Self-Organizing Maps and automatic image segmentation to 101 object categories database. In Proceedings of Fourth International Workshop on Content-Based Multimedia Indexing (CBMI 2005), Riga, Latvia, June 2005.[pdf]
Jorma Laaksonen, Ville Viitaniemi, and Markus Koskela. Emergence of semantic concepts in visual databases. In Proceedings of International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning (AKRR 05), pages 127-134, Espoo, Finland, June 2005.[pdf]
Zhirong Yang and Jorma Laaksonen. Interactive retrieval in facial image database using Self-Organizing Maps. In Proceedings of IAPR Conference on Machine Vision Applications (MVA 2005), pages 112-115, Tsukuba Science City, Japan, May 2005.[pdf]

2004

Jorma Laaksonen, Markus Koskela, and Erkki Oja. Class Distributions on SOM Surfaces for Feature Extraction and Object Retrieval. Neural Networks 17(8-9): 1121-1133. 2004.
Jussi Pakkanen, Jukka Iivarinen, and Erkki Oja. The Evolving Tree - a Novel Self-Organizing Network for Data Analysis. Neural Processing Letters 20(3):199-211. 2004,
Jukka Iivarinen, Rami Rautkorpi, Jussi Pakkanen, and Juhani Rauhamaa. Content-Based Retrieval of Surface Defect Images with PicSOM. International Journal of Fuzzy Systems 6(3): 160-166. 2004.
Rami Rautkorpi and Jukka Iivarinen. A Novel Shape Feature for Image Classification and Retrieval. In Proceedings of International Conference on Image Analysis and Recognition. Porto, Portugal. September-October 2004.
Markus Koskela, Jorma Laaksonen, and Erkki Oja. Entropy-based measures for clustering and SOM topology preservation applied to content-based image indexing and retrieval. In Proceedings of 17th International Conference on Pattern Recognition (ICPR 2004). Cambridge, UK. August 2004. [pdf]
Rami Rautkorpi and Jukka Iivarinen. Content-Based Image Retrieval of Web Surface Defects with PicSOM. In Proceedings of International Joint Conference on Neural Networks (IJCNN 2004). Budapest, Hungary. July 2004.
Jussi Pakkanen and Jukka Iivarinen. A Novel Self-Organizing Neural Network for Defect Image Classification. In Proceedings of International Joint Conference on Neural Networks (IJCNN 2004). Budapest, Hungary. July 2004.
Markus Koskela, Jorma Laaksonen, and Erkki Oja. Use of Image Subset Features in Image Retrieval with Self-Organizing Maps. In Proceedings of 3rd International Conference on Image and Video Retrieval (CIVR 2004). Dublin, Ireland. July 2004. Springer-Verlag LNCS series, volume 3115, pp. 508-516. [pdf]
Timo Ojala, Markus Koskela, Esa Matinmikko, Mika Rautiainen, Jorma Laaksonen, and Erkki Oja. Task-based user evaluation of content-based image database browsing systems. In Proceedings of 3rd International Conference on Image and Video Retrieval (CIVR 2004). Dublin, Ireland. July 2004. Springer-Verlag LNCS series, volume 3115, pp. 234-242.
Jukka Iivarinen, Jussi Pakkanen, and Juhani Rauhamaa. SOM-based System for Web Surface Inspection. In Proceedings of SPIE 5303, Machine Vision Applications in Industrial Inspection XII. San Jose, California. January 2004.

2003

Ville Viitaniemi and Jorma Laaksonen. Content-based browsing of mail-order catalogue with PicSOM system. In Proceedings of the 2003 Conference on Visual Information Systems (VIS'2003). September 2003.
Jorma Laaksonen, Markus Koskela, and Erkki Oja. Probability interpretation of distributions on SOM surfaces. In Proceedings of Workshop on Self-Organizing Maps (WSOM'03). Hibikino, Kitakyushu, Japan. September 2003. [pdf]
Jukka Iivarinen, Antti Ilvesmäki, and Jussi Pakkanen. Evaluation of Features for Defect Image Retrieval. In Proceedings of 3rd IASTED International Conference on Visualization, Imaging, and Image Processing. Benalmádena, Spain. September 2003.
Mika Rummukainen, Jorma Laaksonen, and Markus Koskela. An efficiency comparison of two content-based image retrieval systems, GIFT and PicSOM. In Proceedings of International Conference on Image and Video Retrieval (CIVR 2003). Urbana, IL, USA. July 2003. [ps] [pdf]
Jussi Pakkanen, Antti Ilvesmäki, and Jukka Iivarinen. Defect Image Classification and Retrieval with MPEG-7 Descriptors. In Proceedings of 13th Scandinavian Conference on Image Analysis (SCIA 2003). Halmstad, Sweden. June-July 2003.
Mats Sjöberg, Jorma Laaksonen and Ville Viitaniemi. Using Image Segments in PicSOM CBIR System. In Proceedings of 13th Scandinavian Conference on Image Analysis (SCIA 2003). Halmstad, Sweden. June-July 2003. Springer-Verlag LNCS series, volume 2749, pp. 1106-1113. [ps] [pdf]
Markus Koskela, Jorma Laaksonen, and Erkki Oja. Inter-Query Relevance Learning in PicSOM for Content-Based Image Retrieval. In Supplementary Proceedings of 13th International Conference on Artificial Neural Networks / 10th International Conference on Neural Information Processing (ICANN/ICONIP 2003). Istanbul, Turkey. June 2003. [ps] [pdf]
Jussi Pakkanen and Jukka Iivarinen. Content-Based Retrieval of Surface Defect Images with MPEG-7 Descriptors. In Proceedings of SPIE 5132, Sixth International Conference on Quality Control by Artificial Vision. April 2003.
Markus Koskela and Jorma Laaksonen. Using Long-Term Learning to Improve Efficiency of Content-Based Image Retrieval. In Proceedings of Third International Workshop on Pattern Recognition in Information Systems (PRIS 2003). Angers, France. April 2003. Available online here. [ps] [pdf]

2002

Ville Viitaniemi and Jorma Laaksonen. Browsing an electronic mail-order catalogue with PicSOM content-based image retrieval system. In Proceedings of 10th Finnish Artificial Intelligence Conference (STEP 2002). Oulu, Finland. December 2002. [ps] [pdf]
Jukka Iivarinen, Jussi Pakkanen, and Juhani Rauhamaa. Content-Based Image Retrieval in Surface Inspection. In Proceedings of 7th International Conference on Control, Automation, Robotics and Vision. Singapore. December 2002.
Sami Brandt, Jorma Laaksonen, and Erkki Oja. Statistical Shape Features for Content-Based Image Retrieval. Journal of Mathematical Imaging and Vision, 17(2): 187-198, September 2002.
Jukka Iivarinen and Jussi Pakkanen. Content-Based Retrieval of Defect Images. In Proceedings of Advanced Concepts for Intelligent Vision Systems. Ghent, Belgium. September 2002.
Markus Koskela, Jorma Laaksonen, and Erkki Oja. Implementing Relevance Feedback as Convolutions of Local Neighborhoods on Self-Organizing Maps. In Proceedings of International Conference on Artificial Neural Networks (ICANN 2002). Madrid, Spain. August 2002. Available online here. [ps] [pdf]
Markus Koskela, Jorma Laaksonen, and Erkki Oja. Using MPEG-7 Descriptors in Image Retrieval with Self-Organizing Maps. In Proceedings of 16th International Conference on Pattern Recognition (ICPR 2002). Quebec, Canada. August 2002. [ps] [pdf]
Jorma Laaksonen, Markus Koskela, and Erkki Oja. PicSOM--Self-Organizing Image Retrieval With MPEG-7 Content Descriptors. IEEE Transactions on Neural Networks, 13(4): 841-853, July 2002. Available online here.
Markus Koskela, Jorma Laaksonen, and Erkki Oja. MPEG-7 Descriptors in Content-Based Image Retrieval with PicSOM System. In Proceedings of 5th International Conference on Visual Information System. HsinChu, Taiwan. March 2002. [ps] [pdf]

2001

Markus Koskela, Jorma Laaksonen, and Erkki Oja. Self-Organizing Image Retrieval with MPEG-7 Descriptors. In Proceedings of Infotech Oulu International Conference on Information Retrieval (IR'2001). Oulu, Finland. September 2001. [ps] [pdf]
Jorma Laaksonen, Markus Koskela, Sami Laakso, and Erkki Oja. Self-Organizing Maps as a Relevance Feedback Technique in Content-Based Image Retrieval. Pattern Analysis & Applications, 4(2+3): 140-152, June 2001.
Markus Koskela, Jorma Laaksonen, and Erkki Oja. Comparison of Techniques for Content-Based Image Retrieval. In Proceedings of 12th Scandinavian Conference on Image Analysis (SCIA 2001). Bergen, Norway. June 2001. [ps] [pdf]
Sami Laakso, Jorma Laaksonen, Markus Koskela, and Erkki Oja. Self-Organizing Maps of Web Link Information. In Advances in Self-Organising Maps (eds. N. Allinson, H. Yin, L. Allinson, and J. Slack). June 2001. [ps] [pdf]

2000

Jorma Laaksonen, Markus Koskela, Sami Laakso, and Erkki Oja. PicSOM - content-based image retrieval with self-organizing maps. Pattern Recognition Letters, 21(13-14): 1199-1207, December 2000.
Jorma Laaksonen, Erkki Oja, Markus Koskela, and Sami Brandt. Analyzing Low-level Visual Features Using Content-Based Image Retrieval. In Proceedings of 7th International Conference on Neural Information Processing (ICONIP 2000). Taejon, Korea. November 2000. [ps] [pdf]
Markus Koskela, Jorma Laaksonen, Sami Laakso, and Erkki Oja. Evaluating the Performance of Content-Based Image Retrieval Systems. In Proceedings of Fourth International Conference on Visual Information Systems (Visual 2000). Lyon, France. November 2000. [ps] [pdf]
Sami Brandt, Jorma Laaksonen, and Erkki Oja. Statistical Shape Features in Content-Based Image Retrieval. In Proceedings of 15th International Conference on Pattern Recognition (ICPR 2000). Barcelona, Spain. September 2000. [ps] [pdf]
Markus Koskela, Jorma Laaksonen, Sami Laakso, and Erkki Oja. The PicSOM Retrieval System: Description and Evaluations. In Proceedings of Challenge of Image Retrieval (CIR 2000). Brighton, UK. May 2000. [ps] [pdf]

1999

Jorma Laaksonen, Markus Koskela, and Erkki Oja. Application of Tree Structured Self-Organizing Maps in Content-Based Image Retrieval. In Proceedings of 9th International Conference on Artificial Neural Networks (ICANN99). Edinburgh, UK. September 1999. [ps] [pdf]
Erkki Oja, Jorma Laaksonen, Markus Koskela, and Sami Brandt. Self-Organizing Maps for Content-Based Image Database Retrieval. In Kohonen Maps (eds. E. Oja and S. Kaski). July 1999. [ps] [pdf]
Jorma Laaksonen, Markus Koskela, and Erkki Oja. PicSOM: Self-Organizing Maps for Content-Based Image Retrieval. In Proceedings of IEEE International Joint Conference on Neural Networks (IJCNN'99). Washington, DC. July 1999. [ps] [pdf]
Jorma Laaksonen, Markus Koskela, and Erkki Oja. Content-Based Image Retrieval using Self-Organizing Maps. In Proceedings of Third International Conference on Visual Information Systems (Visual'99). Amsterdam, The Netherlands. June 1999. [ps] [pdf]
Jorma Laaksonen, Markus Koskela, and Erkki Oja. PicSOM - A Framework for Content-Based Image Database Retrieval using Self-Organizing Maps. In Proceedings of 11th Scandinavian Conference on Image Analysis (SCIA'99). Kangerlussuaq, Greenland. June 1999. [ps] [pdf]

People


Erkki Oja	Jorma Laaksonen	Markus Koskela	Ville Viitaniemi	Zhirong Yang	Mats Sjöberg	He Zhang