Ph.D. Theses


The doctoral dissertation represents the culmination of the entire graduate school experience. It is a snapshot of all that a student has accomplished and learned about their dissertation topics. While we could post these on our publications page, we feel that they deserve a page of their own. Here are Ph.D. theses from lab members in reverse chronological order.


FOF
Incorporating Boltzmann Priors for semantic labeling in images and videos
by Andrew Kae, May 2014. [pdf]


[IR]

Abstract:

Semantic labeling is the task of assigning category labels to regions in an image. For example, a scene may consist of regions corresponding to categories such as sky, water, and ground, or parts of a face such as eyes, nose, and mouth. Semantic labeling is an important mid-level vision task for grouping and organizing image regions into coherent parts. Labeling these regions allows us to better understand the scene itself as well as properties of the objects in the scene, such as their parts, location, and interaction within the scene. Typical approaches for this task include the conditional random field (CRF), which is well-suited to modeling local interactions among adjacent image regions. However the CRF is limited in dealing with complex, global (long-range) interactions between regions in an image, and between frames in a video. This thesis presents approaches to modeling long-range interactions within images and videos, for use in semantic labeling.

In order to model these long-range interactions, we incorporate priors based on the restricted Boltzmann machine (RBM). The RBM is a generative model which has demonstrated the ability to learn the shape of an object and the CRBM is a temporal extension which can learn the motion of an object. Although the CRF is a good baseline labeler, we show how the RBM and CRBM can be added to the architecture to model both the global object shape within an image and the temporal dependencies of the object from previous frames in a video. We demonstrate the labeling performance of our models for the parts of complex face images from the Labeled Faces in the Wild database (for images) and the YouTube Faces Database (for videos). Our hybrid models produce results that are both quantitatively and qualitatively better than the baseline CRF alone for both images and videos.


FOF
Unsupervised Joint Alignment, Clustering and Feature Learning
by Marwan Mattar, May 2014. [pdf]
[IR]

[IR]

Abstract:

Joint alignment is the process of transforming instances in a data set to make them more similar based on a pre-defined measure of joint similarity. This process has great utility and applicability in many scientific disciplines including radiology, psychology, linguistics, vision, and biology. Most alignment algorithms suffer from two shortcomings. First, they typically fail when presented with complex data sets arising from multiple modalities such as a data set of normal and abnormal heart signals. Second, they require hand-picking appropriate feature representations for each data set, which may be time-consuming and ineffective, or outside the domain of expertise for practitioners.

In this thesis we introduce alignment models that address both shortcomings. In the first part, we present an efficient curve alignment algorithm derived from the congealing framework that is effective on many synthetic and real data sets. We show that using the byproducts of joint alignment, the aligned data and transformation parameters, can dramatically improve classification performance. In the second part, we incorporate unsupervised feature learning based on convolutional restricted Boltzmann machines to learn a representation that is tuned to the statistics of the data set. We show how these features can be used to improve both the alignment quality and classification performance. In the third part, we present a nonparametric Bayesian joint alignment and clustering model which handles data sets arising from multiple modes. We apply this model to synthetic, curve and image data sets and show that by simultaneously aligning and clustering, it can perform significantly better than performing these operations sequentially. It also has the added advantage that it easily lends itself to semi-supervised, online, and distributed implementations.

Overall this thesis takes steps towards developing an unsupervised data processing pipeline that includes alignment, clustering and feature learning. While clustering and feature learning serve as auxiliary information to improve alignment, they are important byproducts. Furthermore, we present a software implementation of all the models described in this thesis. This will enable practitioners from different scientific disciplines to utilize our work, as well as encourage contributions and extensions, and promote reproducible research.


FOF
Improving Text Recognition in Images of Natural Scenes.
by Jacqueline Feild, February 2014. [pdf]
[IR] [IR]

Abstract:

The area of scene text recognition focuses on the problem of recognizing arbitrary text in images of natural scenes. Examples of scene text include street signs, business signs, grocery item labels, and license plates. With the increased use of smartphones and digital cameras, the ability to accurately recognize text in images is becoming increasingly useful and many people will benefit from advances in this area.

The goal of this thesis is to develop methods for improving scene text recognition. We do this by incorporating new types of information into models and by exploring how to compose simple components into highly effective systems. We focus on three areas of scene text recognition, each with a decreasing number of prior assumptions. First, we introduce two techniques for character recognition, where word and character bounding boxes are assumed. We describe a character recognition system that incorporates similarity information in a novel way and a new language model that models syllables in a word to produce word labels that can be pronounced in English. Next we look at word recognition, where only word bounding boxes are assumed. We develop a new technique for segmenting text for these images called bilateral regression segmentation, and we introduce an open-vocabulary word recognition system that uses a very large web-based lexicon to achieve state of the art recognition performance. Lastly, we remove the assumption that words have been located and describe an end-to-end system that detects and recognizes text in any natural scene image.


FOF
Probabilistic Models for Motion Segmentation in Image Sequences.
by Manjunath Narayana, February 2014.
FOF

Abstract:

Motion segmentation is the task of assigning a binary label to every pixel in an image sequence specifying whether it is a moving foreground object or stationary background. It is often an important task in many computer vision applications such as automatic surveillance and tracking systems. Depending on whether the camera is stationary or moving, different approaches are possible for segmentation. Motion segmentation when the camera is stationary is a well studied problem with many effective algorithms and systems in use today. In contrast, the problem of segmentation with a moving camera is much more complex. In this thesis, we make contributions to the problem of motion segmentation in both camera settings. First for the stationary camera case, we develop a probabilistic model that intuitively combines the various aspects of the problem in a system that is easy to interpret and extend. In most stationary camera systems, a distribution over feature values for the background at each pixel location is learned from previous frames in the sequence and used for classification in the current frame. These pixelwise models fail to account for the influence of neighboring pixels on each other. We propose a model that by spatially spreading the information in the pixelwise distributions better reflects the spatial influence between pixels. Further, we show that existing algorithms that use a constant variance value for the distributions at every pixel location in the image are inaccurate and present an alternate pixelwise adaptive variance method. These improvements result in a system that outperforms all existing algorithms on a standard benchmark. Compared to stationary camera videos, moving camera videos have fewer established solutions for motion segmentation. One of the contributions of this thesis is the development of a viable segmentation method that is effective on a wide range of videos and robust to complex background settings. In moving camera videos, motion segmentation is commonly performed using the image plane motion of pixels, or optical flow. However, objects that are at different depths from the camera can exhibit different optical flows, even if they share the same real-world motion. This can cause a depth-dependent segmentation of the scene. While such a segmentation is meaningful, it can be ineffective for the purpose of identifying independently moving objects. Our goal is to develop a segmentation algorithm that clusters pixels that have similar real-world motion. Our solution uses optical flow orientations instead of the complete vectors and exploits the well-known property that under translational camera motion, optical flow orientations are independent of object depth. We introduce a non-parametric probabilistic model that automatically estimates the number of observed independent motions and results in a labeling that is consistent with real-world motion in the scene. Most importantly, static objects are correctly identified as one segment even if they are at different depths. Finally, a rotation compensation algorithm is proposed that can be applied to real-world videos taken with hand-held cameras. We benchmark the system on over thirty videos from multiple data sets containing videos taken in challenging scenarios. Our system is particularly robust on complex background scenes containing objects at significantly different depths.


FOF
Weakly Supervised Learning for Unconstrained Face Processing
by Gary B. Huang, May 2012. [pdf]


[IR]

Abstract:

Machine face recognition has traditionally been studied under the assumption of a carefully controlled image acquisition process. By controlling image acquisition, variation due to factors such as pose, lighting, and background can be either largely eliminated or specifically limited to a study over a discrete number of possibilities. Applications of face recognition have had mixed success when deployed in conditions where the assumption of controlled image acquisition no longer holds. This dissertation focuses on this unconstrained face recognition problem, where face images exhibit the same amount of variability that one would encounter in everyday life.

We formalize unconstrained face recognition as a binary pair matching problem (verification), and present a data set for benchmarking performance on the unconstrained face verification task. We observe that it is comparatively much easier to obtain many examples of unlabeled face images than face images that have been labeled with identity or other higher level information, such as the position of the eyes and other facial features. We thus focus on improving unconstrained face verification by leveraging the information present in this source of weakly supervised data.

We first show how unlabeled face images can be used to perform unsupervised face alignment, thereby reducing variability in pose and improving verification accuracy. Next, we demonstrate how deep learning can be used to perform unsupervised feature discovery, providing additional image representations that can be combined with representations from standard hand-crafted image descriptors, to further improve recognition performance. Finally, we combine unsupervised feature learning with joint face alignment, leading to an unsupervised alignment system that achieves gains in recognition performance matching that achieved by supervised alignment.


FOF
Using Context to Enhance the Understanding of Face Images
by Vidit Jain, September 2010. [pdf]


[IR]

Abstract:

Faces are special objects of interest. Developing automated systems for detecting and recognizing faces is useful in a variety of application domains including providing aid to visually-impaired people and managing large-scale collections of images. Humans have a remarkable ability to detect and identify faces in an image, but related automated systems perform poorly in real-world scenarios, particularly on faces that are difficult to detect and recognize. Why are humans so good? There is general agreement in the cognitive science community that the human brain uses the context of the scene shown in an image to solve the difficult cases of detection and recognition. This dissertation focuses on emulating this approach by using different kinds of contextual information for improving the performance of various approaches for face detection and face recognition.

For the face detection problem, we describe an algorithm that employs the easyto- detect faces in an image to find the difficult-to-detect faces in the same image. For the face recognition problem, we present a joint probabilistic model for image-caption pairs. This model solves the difficult cases of face recognition in an image by using the context generated from the caption associated with the same image. Finally, we present an effective solution for classifying the scene shown in an image, which provides useful context for both of the face detection and recognition problems.


FOF
Unified Detection and Recognition for Reading Text in Scene Images
by Jerod Weinman, May 2008. [pdf]


[IR]

Abstract:

Although an automated reader for the blind first appeared nearly two-hundred years ago, computers can currently “read” document text about as well as a sevenyear- old. Scene text recognition brings many new challenges. A central limitation of current approaches is a feed-forward, bottom-up, pipelined architecture that isolates the many tasks and information involved in reading. The result is a system that commits errors from which it cannot recover and has components that lack access to relevant information.

We propose a system for scene text reading that in its design, training, and operation is more integrated. First, we present a simple contextual model for text detection that is ignorant of any recognition. Through the use of special features and data context, this model performs well on the detection task, but limitations remain due to the lack of interpretation. We then introduce a recognition model that integrates several information sources, including font consistency and a lexicon, and compare it to approaches using pipelined architectures with similar information. Next we examine a more unified detection and recognition framework where features are selected based on the joint task of detection and recognition, rather than each task individually. This approach yields better results with fewer features. Finally, we demonstrate a model that incorporates segmentation and recognition at both the character and word levels. Text with difficult layouts and low resolution are more accurately recognized by this integrated approach. By more tightly coupling several aspects of detection and recognition, we hope to establish a new unified way of approaching the problem that will lead to improved performance. We would like computers to become accomplished grammar-school level readers.


FOF
Image Classification with Bags of Local Features
by Dima Lisin, May 2006. [pdf]


[IR]

Abstract:

Many classification techniques expect class instances to be represented as feature vectors, i.e. points in a feature space. In computer vision classification problems, it is often possible to generate an informative feature vector representation of an image, for example using global texture or shape descriptors. However, in other cases, it may be beneficial to treat images as variable size unordered sets or bags of features, in which each feature represents a localized salient image structure or patch. These local features do not require a segmentation, and can be useful for object recognition in the presence of occlusion and clutter.

The local features are often used to find point correspondences between images to be later used for 3D reconstruction, object recognition, detection, or image retrieval. However, there are many cases when exact correspondences are difficult or even impossible to compute. Furthermore, point correspondences may not be necessary, unless one is interested in recovering the 3D shape of an object. If the correspondences are not computed, then this representation indeed constitutes an unordered set of local features.

In this dissertation we present methods for object class recognition using bags of features without relying on point correspondences. We also show that using bags of features and more traditional feature vector representation of images together can improve classification accuracy. We then propose and evaluate several methods of combining the two representations. The proposed techniques are applied to a challenging marine science domain.