User:Paulpop

A Manhattan world scene is a term used in computer vision, that describes a real world scene based on the Cartesian coordinate system . The scene is defined by four types of lines: random lines or lines parallel with one of the X, Y or Z axes.

Introduction

The Manhattan World assumption was first discussed by Coughlan and Yuille. They stated that many random visual scenes are based on a three-dimensional Manhattan grid, which should impose regularities in the image. Urban scenes are more likely to comply with the Manhattan world assumption, because they have high statistical regularities due to the predominant straight lines in the scenes. Although one may expect the Manhattan scenes to be strictly related to urban scenes, the same authors argue that some rural scenes contain enough structure on the distribution of edges to provide a natural Cartesian reference frame for the viewer.

File:Manhattan outside.jpg

Camera orientation estimate, rendered in terms of the corresponding orientations of x and y lines

Theory

Manhattan world

Most urban and some indoor scenes (usually man-made scenes) are based on a Cartesian coordinate system which we can refer to as a Manhattan grid. The ground plane is represented by X and Y axes, while the vertical direction is given by the Z axis. Images are formed by perspective projection (see pinhole camera model) of a three-dimensional scene. Lines that are parallel in the image (such as sides of window frames, sides of buildings etc.) are projected to straight lines that converge to a vanishing point. The structure edges in Manhattan scenes usually occur in the X,Y and Z directions. It comes natural to humans to estimate the three-dimensional structure of the scene based on this information. We also interpret the properties of the images regarding their three-dimensional structure and not their apparent size in the image (e.g. relative position of objects, size of people). Based on our assumption of the Manhattan scene we can observe the Ames room visual illusion, where people appear to be changing size while moving in the camera view.

Calculating relative camera position in Manhattan scenes

Knowing the position of the viewer relative to the frame allows a much easier interpretation of the scene. It becomes much easier to determine the important lines in the scene, such as the street/building boundaries or corridor lines. Having this reference frame allows a much easier and faster detection of outliers in the Manhattan world.

It is assumed that the camera direction lies in the horizontal plane, as it is in most images. Then the Manhattan grid defines an ~i;~j;~k coordinate system. The lines in the ~k direction would map to approximately vertical lines in the projected image. However, there is an ambiguity in the ~i and ~j orientations, because of the compass heading which can only be obtained modulo 90.

Projection to 2D

Project 3-D lines from a Manhattan scene to an UV plane. We consider ${\overrightarrow {a}},{\overrightarrow {b}},{\overrightarrow {c}}$ to be three orthogonal vectors representing the three axes in the Cartesian coordinate system and UV the plane where we want to project the point ${\overrightarrow {r}}=(x,y,z)$, and the focal length of camera is f.

                                          $U=f*{\frac {{\overrightarrow {r}}\cdot {\overrightarrow {a}}}{{\overrightarrow {r}}\cdot {\overrightarrow {c}}}}$ 
                                          $V=f*{\frac {{\overrightarrow {r}}\cdot {\overrightarrow {b}}}{{\overrightarrow {r}}\cdot {\overrightarrow {c}}}}$ 

Camera axes defined by three orthogonal unit vectors which are specified by 3 Euler angles. Ψ = (α, β, γ): azimuth, elevation and twist.

File:Coughan projection.jpeg

Projecting an 3D scene to 2D.

Outliers in Manhattan world

Use Manhattan model to find outliers, pixels that are unaligned to the Manhattan frame. The model was used by Coughlan to identify odd elements in Manhattan images (a bike, a robot)]].

Speculation

Manhattan edge classification could simplify stereo matching Basic idea: edge classification at each pixel constraints possible matchings along epipolar lines. An X pixel in left image should match an X pixel in right image, not a Y or Z pixel such that many possible matches could be ruled out.

Applications

Automatic Camera Calibration from a Single Manhattan Image

• The Manhattan model is used to estimate the camera pose, calibration parameters and focal length.
• Introducing a stochastic search algorithm.

Bayesian algorithms for autonomous vision systems

• The Manhattan algorithm is used to guide an autonomous robot vehicle, using an artificial retina.

Extracting 3D information from single images

• Exploiting the Manhattan assumption to obtain 3D reconstructions from a single image.
• Extracting ambiguous features such as image depth and identify objects.
• Generating views of interior rooms
• Automated model generator of building interiors

Classifying edges

Classifying edges

See also

Manhattan distance

Pinhole camera model

Edge detection

Hough transform

Related research papers

• Building Reconstruction using Manhattan-World Grammars, (Vanegas et al., 2010)

• Automatic camera calibration from a single Manhattan image (Deutscher et al., 2002)

• Manhattan-World Stereo (Furukawa et al., 2009)

• Geometric Reasoning for Single Image Structure Recovery (Lee et al., 2009)

Methods that use edge detection, Hough transforms, not Manhattan models:

• Finding vanishing points in a single image (Brillault-O'Mahony 1991; Lutton, Maitre, Lopez-Krahe 1994; Shufelt 1999.)

• Finding vanishing points in context of Manhattan-type assumption of dominant directions in a scene (J. Kozecká and W. Zhang, “Video Compass.” ECCV ’02)

External links

• CVonline Manhattan world scene

Category:Image processing Category:Manhattan world