Seamless photography: Using mathematical models for image stitching

(Press-News.org) Philadelphia, PA—A photo captures only as much as the camera in use will allow, and is therefore limited by the field of view of the camera's lens. In the case of smartphones and many advanced cameras, the view from the lens is much smaller than the view from your own eyes.

Panoramic photographs were invented to capture large objects or scenes that could not otherwise fit within the constraints of a single photo. Panoramic photography is achieved through image stitching, a process that combines two or more photographs, seamlessly blending input images with overlapping regions into one picture. A paper published by Wei Wang and Michael Ng in the SIAM Journal on Imaging Sciences this summer aims to develop an algorithm for image stitching.

Image stitching involves two steps: image alignment and image blending. Image alignment finds point pairs in the overlapping region of two images that correspond to one another. Image blending combines the two aligned images seamlessly. This step is important if the pixel intensities in the different images vary enough to produce artifacts such as varying lighting conditions and different exposure settings. In this paper, the authors focus on image blending, assuming that the images have been aligned.

Many different approaches for image blending are seen in the literature. "The traditional method is to search for a curve in the overlapping area in which the differences among the input images are minimal," explains author Michael Ng. "However, the curve may not be determined accurately because of light intensity, color inconsistency, parallax, occlusion, etc."

The approach used in this paper instead minimizes seam artifacts by smoothing the transition between the images. The mosaic image here is a weighted combination of the input images. This means the pixel values from the two overlapping images are combined using a weighted average for qualities such as exposure, local contrast, saturation, etc.

How is this achieved?

Many systems, both natural and man-made, seek out the lowest energy state, such as, a ball rolling down a hill, or a snow-laden tree branch bending to maintain the lowest possible energy in the system. The concept of minimizing the energy of a given system is also used in image processing. For a given image, an energy function is defined and minimized to get a better image (i.e. less noise, better sharpness, higher contrast, etc.). This is the approach the authors use in the paper. Seamless combination of images is achieved by minimizing an energy function based on intensity or gradient differences of the two images.

"According to the model, we construct a weighting function over the overlapping area so that a panoramic image can be generated," says Ng. "The optimal weighting function can be obtained by minimizing the overall energy of the mathematical model." Thus, in the proposed model, both the weighting function and the final blending in the overlapping region are based on solving an energy minimizing problem. The authors show how to define an energy function and develop an algorithm to minimize it.

This variational method—based on achieving the lowest energy or ground state—is seen to produce a more visually appealing photo in comparison to other existing methods.

Future work may extend the scope of this research beyond two-dimensional images. "It is interesting to consider extending the current variational approach to tackle the problem of three-dimensional image stitching in medical imaging applications and stitching video in computer applications," Ng says.



INFORMATION:

Source Article: A Variational Approach for Image Stitching I

http://epubs.siam.org/doi/abs/10.1137/110819871

Wei Wang and Michael K. Ng SIAM Journal on Imaging Sciences, 6(3), 1318 (Online publish date: 11 July 2013).

The source article is available for free access at the link above until January 1, 2014. After that date, or for any single SIAM article, access is available at epubs.siam.org for $25 per article.

To read related research published by the authors about further developing an image stitching algorithm using the gradients of input images, view: A Variational Approach for Image Stitching II: Using Image Gradients http://epubs.siam.org/doi/abs/10.1137/120872140 SIAM Journal on Imaging Sciences, 6(3), 1345

About the authors:

Wei Wang is an associate professor at Tongji University in Shanghai, China, and Michael Ng is a professor at Hong Kong Baptist University. The work was partly supported by RGC grants, HKBU FRG grants, the National Natural Science Foundation of China, and the China Postdoctoral Science Foundation.

About SIAM

The Society for Industrial and Applied Mathematics (SIAM), headquartered in Philadelphia, Pennsylvania, is an international society of over 14,000 individual members, including applied and computational mathematicians and computer scientists, as well as other scientists and engineers. Members from 85 countries are researchers, educators, students, and practitioners in industry, government, laboratories, and academia. The Society, which also includes nearly 500 academic and corporate institutional members, serves and advances the disciplines of applied mathematics and computational science by publishing a variety of books and prestigious peer-reviewed research journals, by conducting conferences, and by hosting activity groups in various areas of mathematics. SIAM provides many opportunities for students including regional sections and student chapters. Further information is available at http://www.siam.org.