A full sphere camera for point of view and VR photography

Dan Slater


Introduction

This paper describes a full sphere camera system that records a scene in all possible viewing directions (4 pi steradians) simultaneously. This camera system is intended for ultrawide angle panoramic photography and virtual reality (VR) applications. Click here for an excellent background material web site written by James Rigg. Click here for a list of examples of VR photography on the web. The Spherecam system consists of two main elements; a compact, highly ruggedized camera assembly that records the scene in all possible viewing directions and a computer based image processing system that produces VR compatible images.

The camera assembly uses a pair of opposing motor driven Nikon cameras equipped with 220 degree fisheye lenses that simultaneously record the scene in all viewing directions. The camera assembly is quite rugged and compact enough that it can be hand or helmet carried on roller coasters, bikes, used inside aircraft cockpits, mounted outside of helicopters, etc.

Unmodified commercially available software is used to first assemble and convert the raw fisheye images into full sphere images and then extract a variety of views in any direction. First the two fisheye images are converted into an equidistant cylindrical projection and combined into a single image in Adobe Photoshop. This intermediate image is then texture mapped onto a 3D sphere, forming the conceptual equivalent of a "virtual Omnimax theater". The viewer is placed in the center of the texture mapped sphere where he can view in any direction. These views can be dynamically formed through the use of VRML.

Since the original publication of this paper in 1996, Helmut Dersch has developed a freely available software package that provides better and higher quality image processing than is described here. The use of his software "Panorama Tools" would be recommended as a higher quality alternative to the image processing described in this paper. The description here though should help to provide a good understanding of processing fisheye images into panoramic equivalents.
 
 

Background

The interest in dynamically extracting scenes from ultra wide angle views is rapidly increasing.  For the purposes of this article, we are defining a VR camera as a system that can record a full sphere image or a major portion of one and the associated image processing that can extract a narrower angle view, often dynamically.

The ideal requirements for a VR camera system are:

  1. Full sphere imaging -The camera should record a full sphere image that includes all possible viewing directions (4 pi steradians).
  2. High resolution -The camera should record the scene to a detail level of what the human eye can resolve. For an unaided eye, this is approximately 1 arcminute. Film based wide angle high resolution imaging systems require large image areas on the film. Currently (in 1996), scanning panoramic and real or synthetic camera clusters are used to record wide angle high resolution imagery. If the VR camera is implemented in digital form, the camera would require one or more sensors and storage devices to capture and record the desired VR image. Approximately 20,000 picture elements are required for one equatorial ring of the image. A full sphere image sampled at a resolution of 66 nanosteradians per pixel (1 arcminute per pixel) produces 190,476,000 pixels per frame. If stored in a standard 24 bit per pixel format, the image would require 571 megabytes per frame. Both the image sensor and the storage device can be single or multiple devices.
  3. Large storage capacity - A VR camera system should be capable of recording numerous images to provide a discrete approximation of time varying viewer positions. As each image could require over 500 megabytes per frame, a combination of good compression algorithms and large storage devices would be required.
  4. POV capabilities - The Point Of View (POV) photography style attempts to place the camera viewpoint at the eyepoint of an actor or other position that is within a scene. The POV camera is within a scene, rather than viewing the scene from a remote location. The POV camera often needs to be rugged and compact as might be required for filming from a motorcycle helmet or skydiving helmet mount. People interested in POV imagery tend also to be interested in VR methods. As such, VR photography requirements are rapidly evolving into a superset of POV requirements. A new generation of VR cameras are starting to be built that are also compatible with the other requirements of POV photography.

Existing VR camera systems

Full sphere images have a number of uses in computer systems, particularly in the area of virtual reality simulation systems. In these systems, the viewer can dynamically select a view from an image data base that contains likely viewing directions. The images are currently recorded with a scanning panoramic camera or with a real or synthetic segmented camera cluster. A film based scanning panoramic camera can provide a coverage of 120 degree (vertical) by 360 degree (horizontal). A digital scanning panoramic camera can provide a complete 4 pi steradian view when using a 180 degree fisheye lens.

Presently there are several vendors that provide VR camera systems. There are three classes of ultrawide angle cameras currently in use; scanning panoramic, synthetic cluster and real cluster. The scanning panoramic designs use a rotating lens to image onto a moving piece of film or a CCD sensor. Camera clusters take a series of images in different directions so as to record all desired viewing directions. By analogy to synthetic aperture radar, the cluster can be real or synthetic. A real cluster means that a separate camera is used for each viewing direction. A synthetic camera cluster means that a single camera is sequentially rotated to different viewing directions. Both film and CCD based cameras can be used. Currently, the film based systems have the advantage of higher resolution and portability although this will likely change in the near future. The film based systems generally either use a single scanning panoramic camera or a series of separate photographs that are stitched together to record a wide angle scene. Several of the more popular and interesting systems are briefly described here.

Apple Quick Time VR - The Apple Quick Time TV (QTVR) system uses either a single scanning panoramic camera shot or a series of still camera shots to record the scene in a circle around a common point. Scanning panoramic cameras commonly used include the Hulcherama,Roundshot and  Globuscope. Alternately, the QTVR system may use a still camera, typically a Nikon equipped with a 15 mm lens to record a series of images at different azimuth angles. A tripod and indexing device are often used to precisely position the camera for this series of images. If the still camera method (synthetic cluster) is used, the separate images are stitched together with a cylindrical mapping correction to form an image equivalent to that produced by a standard scanning panoramic camera. The cylindrical perspective image is then converted into the Apple QTVR file format for viewing with an Apple provided viewer.

Infinite Pictures (iMove) & Kaidan -- This pair of companies manufactures a full sphere imaging system using a full frame fisheye lens and a multiple angle stitching bracket. iMove is apparently also building full sphere video camera systems.

RealVR -- RealVR is a new dynamic viewing system developed by Live Picture. This system an extension of VRML 2.0. The RealVR viewer is freely available as a Netscape / Internet explorer plugin or as a standalone viewer. It is compatible with cylindrical and spherical image mappings (as used by the spherecam). A simple control file is needed to make the system work. Documentation on RealVR available. Using the RealVR viewer is described later in this paper.

Jeffrey Charles (Versacorp) has been building reflective hyperwide optics for many years. Similar optics are available commercially from others including, Tateyama , Be Here and Omnicam. Drexel University is experimenting with other hyperwide mirror systems.  Several other sites (1 , 2) include links to other hyperwide imaging systems.

Ipix - Ipix markets a system that can dynamically extract moderate angle views from any part of fisheye or full sphere image. Initially, their system used a standard Nikon camera equipped with an 8 mm f2.8 fisheye lens. After Nikon discontinued this lens, Ipix has been using lenses manufactured by Coastal Optical and using other Nikon fisheye lenses on the Coolpix 950 camera. In any case, the camera is mounted on a special tripod that insures that the camera is rotated precisely 180 degrees to maintain alignment of the aberrated lens entrance pupil between two adjacent exposures. If the rotation is not accurate, part of the scene will not be recorded. The fisheye image pair is combined with a proprietary software package into a form that is compatible with their viewer. Recently, Coastal Optical has started manufacturing a video equivalent of the Spherecam for IPIX using a pair of fisheye lenses. Within the last few years, Ipix has become quite litiguous and has fallen into disfavor with many photographers (sites explaining the controversy: 1, 2, 3, 4, more information can be found by searching the internet for "ipixfree").

PanoramIX- Another VR system.

Hemispherical imaging home page -- Scientific studies of plant canopies through the use of fisheye hemispherical photography. Also see Delta T Devices.
 

The ruggedized POV / VR full sphere camera (Spherecam)

The design goal of this project was to produce a camera that could record full sphere views in an action VR / POV (point of view) application. The camera would need to be compact, rugged and light weight so it could be used for recording in skydiving, helicopter, motorcycle, aircraft cockpit and other action environments. Because of the hand held / action environment, the camera would need to record all parts of the scene in a very short time interval. This ruled out synthetic cluster designs. The camera would need to be film based as the CCD approach would be too expensive, bulky and complex at this time (1992). A film based scanning panoramic design was ruled out because it could not record over a 4 pi steradian field of view. It is possible to design a refractive lens that records a 4 pi steradian field of view (Slater 1932). This lens would likely be expensive to build and physically large. This drove the design to a real camera cluster using two opposing fisheye lenses.

Spherecam

The camera assembly consists of a pair of opposite viewing motor driven Nikon F camera bodies. The camera bodies are rigidly attached to a machined aluminum skeletal frame. The cameras were stripped of viewfinders and battery packs so as to minimize the size and weight. Both cameras were equipped with Nikon 6 mm f5.6 220 degree fisheye lenses. This particular lens was chosen since it is very compact and provides a large 40 degree overlap region that greatly simplifies the later image splicing operation (Slater 1996). Unfortunately, this lens is no longer in production but can sometimes be found in the used camera marketplace. As the lens entrance pupils can not be coincident as in a synthetic cluster design, the large overlap region is particularly important when objects are close to the camera. A number of accessories were developed for the full sphere camera including a pistol grip, camera extension poles, a remote radio trigger system and a fill lighting system suitable for aircraft cockpit photography.
 

Image processing

Full sphere images can be assembled from separate images that record segments of the scene. As an example, a pair of 180 degree or greater angle fisheye images can be combined into a composite 4 pi steradian image. If all views are recorded simultaneously, the camera cluster is real. If a single camera is used sequentially to record the necessary views, the camera cluster is synthetic. These terms are by analogy to the concepts of real and synthetic aperture radar. A real cluster is required for POV applications.

VR systems operate by extracting part of the image corresponding to the observer's current view port. The basic mathematical concepts behind the conversion of a cylindrical panoramic or spherical image into a view port are well described in books on map making ( Snyder 1989 , Snyder 1987), 3D computer graphics and spacecraft attitude determination (Wertz 1978).

The image processing for this system consists of 3 elements; stitching the two opposing fisheye images into a single 4 pi steradian image, texture mapping the 4 pi steradian image onto the surface of a sphere and then extracting the desired views from the center of the sphere. The method described here uses standard unmodified commercially available software. Better alternatives exist since this paper was originally written. An excellent software program that is freely available was written by Helmut Dersch and is now the software of choice for this application.
 

Step 1: Merge the pair of fisheye photographs into a single full sphere intermediate image.

The first step is to combine the pair of opposing fisheye images into a single image. This is accomplished in standard Adobe Photoshop as follows:
  1. Scan the two images into the computer. Use the same scanner settings for both images. The images should be centered in the frame.
  2. Convert both images into a cylindrical format so that they can be merged. This can be done in Adobe Photoshop by using the polar to rectangular conversion filter. In cartographic terms, the fisheye image is an azimuthal equidistant projection. The Adobe Photoshop polar to rectangular conversion filter converts the fisheye image into an equidistant cylindrical (also known as an equirectangular) projection.
  3. Increase the canvas size of one image vertically by the ratio of 360 to 220 (the ratio of the full sphere angle to the fisheye lens field of view), leaving the image at the top of the frame.
  4. Rotate the second image 180 degrees and paste as a layer with a layer mask onto the first image.
  5. Align the overlap region and blend the layers into a single composite image.
  6. Two fisheye images prior to
          processing
    The two preceding images are a pair of 220 degree fisheye images (step 1). The next two images are the result of step 2:

    Images after step 2

    These two images will be combined by rotating the image on the right by 180 degrees and overlapping.

    Combined image


The previous image combining process can be nicely automated in Photoshop 4.0 by using the Actions feature. The resulting image (called the intermediate image) corresponds to a full sphere image in an equidistant cylindrical projection. It is highly distorted but contains the necessary information from which other views can be extracted. The horizontal axis has 360 degrees of coverage, the vertical axis has 180 degrees of coverage. This projection is similar, but not identical to the central cylindrical projection used by a scanning panoramic camera. The difference is an arctangent function along the Y axis.

The intermediate image can be processed in a number of ways (Slater 1996). As one example, a rectangular to polar conversion (the inverse of the previously used polar to rectangular conversion) can be done to produce a fisheye image corresponding to a 360 degree field of view. As a second example, the intermediate image can be converted into the scanning panoramic format although, because of the arctangent function, the vertical field of view must be less than 90 degrees.
 
 

Step 2: Texture mapping the full sphere image onto a 3D sphere.

We are now going to develop the concept of a "virtual Omnimax theater". A real Omnimax theater consists of a large format 70 mm film projector equipped with a fisheye lens at the center of a hemispherical projection screen. An audience member views the screen, ideally from a point near the center of the theater dome. We will create a similar system in virtual form. The example images produced here were produced with the 3D program "Ray Dream". The screen will be the surface of a sphere created as a 3D model. The projector is simulated by texture mapping the previously assembled intermediate image onto the surface of a 3D sphere.
 
 

Step 3: View the texture mapped sphere

The sphere can be viewed in a conventional manner from the outside. As such, it is similar in appearance to an image of an Earth globe. As in the case of a real Omnimax theater, the more interesting view is often inside the sphere.

Outside view of sphere

The above view is of the texture map applied to the outside of the sphere. This view is different than that of a 360 degree fisheye as shown in the next example. A slight processing misalignment produced the colored region around the spherical image.

360 degree fisheye image

All of the following images are obtained from virtual cameras located at the center of the texture mapped sphere. The next image was produced by a synthetic forward looking camera with a moderate wide angle lens (24 mm 35 mm still format)

Extracted moderate wide angle
          view

The synthetic camera was rotated 80 degrees to the left for this next view. The field of view was similar to that of a very wide 12 mm lens on a 35 mm still format camera.

Extracted wide angle view

The following image is a forward looking rectilinear ultrawide angle view taken with the equivalent of a 3 mm lens on a 35 mm still format camera. There is no real equivalent to this lens.

Extracted ultrawide view

Views can be extracted in any direction and orientation. The amount of available zoom is dependent upon the resolution of the texture map. As an extreme example, the following image shows a cylindrical perspective view with horizontal and vertical fields of view in excess of 360 degrees. Unlike a conventional scanning panoramic camera, this synthetic camera uses a f theta mapping on both axes to achieve the ultrawide vertical field of view. If a narrow horizontal slice were selected in any region other than the zenith or nadir, the resulting view would be similar to that of a conventional scanning panoramic camera.

Panoramic view with greater than
          360 degree field of view

A hyperspherical image made with the Spherecam. The vertical and horizontal fields of view are 720 degrees. The diagonal field of view is 1440 degrees..

Real time image viewing

The images in this paper were produced with Adobe Photoshop and Ray Dream as static views. Can these images be dynamically viewed ? The Apple QTVR program does not support full sphere images, the Omniview system is proprietary so neither of these viewers are readily usable. Virtual Reality Modeling Language (VRML) provides a freely available and open alternative. The method is the same; texture map the full sphere image onto a VRML sphere. Go to the center of the sphere and view. VMRL Internet browsers include the necessary support to display computer derived views at any orientation. In order to make a good viewer, vendors need a minor software change to support object interior views and improve the quality of the texture mapping algorithms. For an example of an exterior VRML view, click here.

The VRML description of the virtual Omnimax is straight forward:

#VRML V1.0 ascii

Separator {

Texture2 {filename "veg3.jpg"}

Sphere {radius 1.0}

}

RealSpace Inc. has developed an extension to VRML 2.0 that provides excellent dynamic viewing of the spherecam images. The process is quite simple: 1. Create a jpg file of the spherecam intermediate image, for example here "vegas.jpg". 2. Create the following file in notepad or a similar text editor. The file name should have an ivr extention.

#VRML V2.0 utf8

Vista {

type "SPHERE"

filename "vegas.jpg"

}

To dynamically view the jpeg image, just open this VRML file with Netscape, IE or directly with the RealVR viewer.
 

Conclusions

This paper has described a system for recording and displaying full sphere images. The full sphere camera assembly is compact, rugged and is suitable for both point of view and VR applications. The image pair produced by the camera assembly is digitally combined into a full sphere image with Adobe Photoshop. The resultant image is texture mapped onto a sphere forming a "virtual Omnimax" theater. Views can be statically extracted with current 3D modeling programs or with minor modifications, dynamically extracted with VRML viewers.
 

References

Slater, J., Photography with the Whole Sky Lens, American Photographer, October (1932)

Describes several simple fisheye optical systems and proposes a lens design concept that provides a 4 pi steradian field of view.

Slater, D., Panoramic Photography with Fisheye Lenses, Panorama, Vol. 13, No. 2, 3, International Association of Panoramic Photographers, (1996)

Describes a camera system and digital processing system that can be used to create ultrawide angle point of view photographs.

Snyder, An Album of Map Projections, U. S. Geological Survey Paper 1453, U. S. Government Printing Office, Washington, DC (1989)

An excellent introduction to the theory of map projections. Highly recommended.

Snyder, Map Projections - A working Manual, U. S. Geological Survey Paper 1395, U. S. Government Printing Office, Washington, DC (1987)

A more detailed discussion of map projections.

Wertz, J., ed., Spacecraft Attitude Determination and Control, D. Reidel Publishing Co., Boston, MA (1978)

Includes a discussion of spherical trigonometry used in spherical image remapping.
 
 

Home > Spherecam


© Copyright 1996 by Dan Slater, All Rights Reserved