How are 360° photos stored and processed?

Question

While looking up 360° photos I found a lack of information about how the images actually work, I found some info suggesting that the images are stored as a regular jpeg but I was unable to find the finner details.

The questions I have are:

• How are 360° photos stored? Do they have a special file format or are they projected onto a regular png/jpeg?
• If they are projected onto a flat image will they lose quality in some areas where the spherical projection has warped the image?
• Are there any non standard ways for storing and processing 360° photos which work differently to the common methods?

Answer 1

There are no finer details.

Let us start from the beginning.

A file

You have a camera with a lens and the file stores what was projected on the sensor.

You can change the lens, for example to a wide angle lens and the file just do the same. No black magic there.

You can then store an image taken with a fisheye lens. The file itself has no idea and stores the image exactly the same way.

A file format has no way to differentiate what is stored in it.

---

360 views

Ok. Now let's take a look at some types of "360" images.

A 360 image for start is one that contains information on what is around you. Behind you, in front, to the left, to the right.

The most primitive way to capture this information is by taking a photo of a metallic reflective sphere. This contains information about (almost) all your surroundings.

360 images from a spherical reflective ball

This is a "Probe Map" projection. But there are several others. Cubic projection, Spherical projection, Cylindrical projection.

What now has popularized these images is the ability to compute these transformations and unwrap them again.

So this simple Jpg with a distorted projection is projected into a 3D "solid" from the inside and then you see a section of it, and you can move around.

This solid can be a sphere or a cube, depending on the application. (Cube is the most common)

Cubic projection

---

Here is a link to a simple paper I wrote some years ago about the basics of different projections for environment maps.

---

The viewing

The trick is that we have now software that can interact with this projection, either on a 3D rendering software for a video or animation, or in real-time like a video game, a VR application, or a simple panorama on a website.

Answer 2

How are 360° photos stored? Do they have a special file format or are they projected onto a regular png/jpeg?

360x180 panos are stored a number of different ways, but most commonly as a single image in the usual visual file formats (TIFF, JPEG, PNG) in equirectangular projection. Equirectangular projection represents the sphere as a 2x1 rectangle, where the cartesian X-Y coordinates correspond to the longitude and latitude of the point on the sphere. As a result, there's a lot of warping around the poles.

When displayed interactively, that equirectangular is most commonly used as the basis for either HTML5 or Flash output; the older Apple QuicktimeVR format was another standard, which used a cubemap rather than an equirectangular projection as a basis. Cubemaps most common are done as a 3x2 grid of the faces, or as a cross layout (e.g., Blender skyboxes and environment maps).

The Google cardboard format is two stacked equirectangulars for 3-d; left-eye view over right.

If they are projected onto a flat image will they lose quality in some areas where the spherical projection has warped the image?

Depends on how you plan to view and rewarp to another projection.

Are there any non standard ways for storing and processing 360° photos which work differently to the common methods?

Tons. There is no standard. And remapping is kind of a sport among those who do 360x180s. Cartographers have figured out a lot of ways to represent a sphere on a flat surface. :) Just looking at the list of Flexify mappings should give a sense of how many possible mappings there are. Some common ones you'll see are stereographic little planets/tunnels. But sky's kind of the limit. I like using drostified quincuncial when I really wanna screw with peoples' heads.

See also: the pano-projections tag.

Answer 3

A lot of these images that are made for commercial purposes, e.g., real estate walk throughs, are either cylindrical or spherical projections, and the method will depend on the requirements of the hosting environment. Many of the online hosting services have propriatary software, and have slightly different requirements. Whether a cylindrical or spherical projection image is chosen depends on the marketing budget for a particular property. A spherical projection image requires triple the number of individual (pre-stitched) images as a cylindrical projection image, and so costs more to produce. The answer mentioning other (non-typical) projections is spot-on. It would be only a matter of working out the mathematics. Virtually any 3d space would be possible once the conversion algorithm is determined. (A projection on a Klein bottle would be an example that might prove more difficult!) Any Euclidean 3d space should be rather straightforward.

Answer 4

I think you are mixing "file format" with "image projection", two completely different topics.

You can use whatever file format you want, they are just images.

But you must use specific image projections, because a 360 panorama represents a sphere (a 3d object), but files represent flat images (2d).

I think the most used image projections for representing 360 images are equirectangular and cubemap.

Equirectangular (2:1 aspect ratio):

Cubemap (6 squared 1:1 images)

As long as the file format is not lossless, you will not loose any detail in the conversion, because while converting from fisheye to cubemap/equirect the pixels are stretched/added where needed.

Answer 5

There is no special format for any kind of 360 degree photo I have ever seen. Everything is done in rectangular images projected and/or stitched in one form or another.

This panorama that I made was done by taking 24 shots with the camera set on the tripod shifted back so the pupil entrance stayed right above the tripod rotation axis. I used a normal rectilinear lens. It was rotated 15 degrees for each shot. The 24 photos were cropped to a vertical in the center then stitched together into one very wide rectangular view and reduced to 3060x360 (8.5:1). The Javascript code I wrote buffers two pictures side-by-side (6120x360) (so only one copy is downloaded) and slices a normal width view (640x360) out of it at various places for various angles, with views of the edges crossing where the two buffers meet. The Javascript inside the HTML is free to use. Just be sure your photo has an exact connection from left to right.

It's not perfect. You can find some stitching errors.

Just click one of the buttons for a rotating view at one of three speeds in the direction you want. The middle button stops the rotation where it is.

http://ipal.net/vance/