How to take pixel-for-pixel aligned pictures with a DSLR and iPhone 7?

Question

The Plan

So I am trying to take two pictures, one with a DSLR and one with an iPhone 7. Why an iPhone 7? That's just what I own currently I want the photos to be aligned so that they are exactly the same. The camera and iPhone will mounted on a tripod using a stereoscopic tripod mount like this, the iPhone using something like this.

The idea is to place the iPhone and the DSLR on the tripod and take the photo, then algorithmically align the photos. I do have a few concerns because I am somewhat new to photography...

The Gear

This is for a research project so I can use cameras on loan from my university. Here are the camera & lens combos I have available with their specs:

• Canon 80D w/ a Canon 18-135mm EFS Lens
• Canon 70D w/ a Tamron 18-200mm Lens
• Canon 80D w/ a Tamron 18-200mm Lens

And it says online that the iPhone 7 has the following specs:

Resolution: 12MP

Sensor: Sony Exmor RS (1.22 µm, 1/3")

Focal Length: 4mm (28mm eff) 8mm (56mm eff)

Max Aperture: F/1.8

The Concerns

So my question is, what variables do I have here that I need to keep in mind. The first one is horizontal displacement seen here in a crude mockup using stock photos. This I think will not be too much of an issue. My one concern is that parallax will make the overlapping part not actually be the same exact photo and there will be some noticeable difference. That would be a problem.

The second problem is the one I have no idea where to start with: lens distortion. This is due to aperture and focal length, right? Theoretically, if I manually set the iPhone's focal length & aperture to be the same as the DSLR's(which is easily done using photography apps), will the pictures turn out to be exactly the same?

How should I start going about doing this? These both are big problems that I really want to know how to address most easily. I want to take lots of photos side by side and I'm trying to figure out a workflow/setup that will make aligning the photos as easy and accurate as possible. I'm just looking for guidance so I can know where to start, what to buy, and how to set things up somewhat professionally.

Feel free to ask followups if I left any important details out.

Edit:

• This is for a ML project where I am trying to use DSLR images as ground truth to improve iPhone images. It's a recreation of this research. What they use is a linear transformation and cropping, which is very similar to @xiota's answer (You are a smart person, I'm glad I got your help :))

Answer 1

Garbage-In, Garbage-Out

A big problem with using DSLR photos as "ground truth" for the improvement of cell-phone photos is the assumption that DSLR photos are necessarily better than cell phone photos.

• It is possible to make a DSLR photo that closely matches a phone photo by closely matching the FOV and DOF.

• The lenses you will be using have relatively slow F3.5-6.3 apertures, which will limit the background blur you can achieve at any given focal length.

• With Canon cameras, I often have difficulty getting well-exposed shadows without blowing-out the highlights.

• Different camera settings can give very different looks.

Another issue is that iPhones are considered to already produce very good images, so the amount of room for improvement isn't as much as with other phones.

Some Neural Net Training Ideas

These are just ideas. They may not actually work.

• Train mini neural networks to improve specific aspects of an image. Noise, resolution, color... can each be trained in separate subnets, which are later combined.

• The subnets don't necessarily need to output an image. An exposure-correcting subnet could return exposure correction settings. A color correction subnet could return adjustment curves.

• For some of the subnets, it may not be necessary to train with images taken on different cameras.

  • To increase image resolution, use a high-res DSLR image along with a down-sampled version.

  • To work on DOF, use DSLR images of the same scene and location with different aperture settings (calculated to match that of the phone).

  • To correct exposure, use images captured on the same device, but with different exposure settings.

  • To reduce noise, use ISO 100 vs other ISO settings taken on the same device. (To correctly capture the phone's noise characteristics, probably best to use the phone. Exif data can be used to select the appropriately trained subnet for processing.)

• Some features, such as color correction, may not need an exact match because the image will be preprocessed (eg, blurring). The match only needs to be good enough. Then pre-existing tools can do the rest (align_image_stack).

• I'd also suggest training some subnets to improve flare and glare because they are particularly difficult to correct. Use images taken with and without lens hoods for training.

---

The following was written before I knew that the purpose of the images is to be used as neural network training data to improve phone photos. Although some of it has been revealed to be counter-productive, I am keeping it because it answers the question as I understood it to be asked.

Factors that Contribute to Image Differences

Obtaining two photos that are "exactly the same" "on a pixel level" with two different cameras and lenses is practically impossible for the following reasons:

• Mismatched output dimensions makes producing pixel-exact images impossible.

• Mismatched sensor ratios makes producing pixel-exact images impossible.

• Other mismatched sensor characteristics, such as light sensitivity and color response.

• Mismatched camera settings, such as demosaicing algorithms and color profiles.

• Even with the same camera and lens on a tripod, it can be difficult to get photos to match with such precision. Even the slightest tap can cause images to be different "on a pixel level" because sensels are only about 5 microns. (Larger for full-frame cameras. Smaller for cell-phone cameras.)

• There may be movement within the scene, such as leaves and branches swaying in the wind.

• Any displacement of the nodal point would result in different images. The dual-camera tripod head will not work.

• Lens field-of-view would need to match. (On DSLR, use phone focal length * crop factor.) Zoom on the DSLR lenses is adjusted manually. It would be difficult to consistently make pixel-precise adjustments manually.

• "Equivalent" apertures of the lenses would need to match. (On the DSLR, use the the phone aperture * crop factor.) Often apertures move to preset levels within a margin of error. The exact values you need may not be available, or the margin of error may be greater than what you need to obtain pixel-level precision.

• The different lenses are likely to have mismatched distortion characteristics, which may include barrel, pincushion, and mustache types. See also:

  • What is the difference between perspective distortion and barrel or pincushion distortion?

  • How difficult is it to correct mustache-style distortion?

• The different lenses may have mismatched chromatic aberration.

Approach that Replicates the Phone Images on the DSLR

To get images to match as closely as possible, you can try the following:

1. Normalize the size and ratio of the images. Perhaps by using a center crop with the same pixel dimensions across all cameras.

2. Calculate the crop factor for the normalized images.

3. Take a photo with the iPhone.

4. Use high-precision motors with a feedback loop to place the DSLR camera into position, with checks and adjustments against the iPhone image.

5. Use high-precision motors with a feedback loop to set DSLR focal length and aperture, again checking and adjusting against the iPhone image.

6. Take photo on DSLR with exposure settings to match iPhone.

7. Adjust contrast, brightness, etc of DSLR image to match iPhone image.

8. Align and correct the images with align_image_stack or similar software.

9. Results may still not be pixel-level identical.

Answer 2

You can not.

You could overcome the displacement of the two lenses using either a prism or a mirror so you have the same position and split the beam of light.

You now then need to decide how to crop for the two sensors. You could probably need to strip the phone and camera from the lenses and use a lens before the light enters the prism so you avoid chromatic aberrations and distortions of the lenses differences, and then use some special optics to focus on the different sensor size and distances and sensitivities.

But even the same brand and model of cameras are made of different silicon plates. They could have the same structure, but they have molecular differences on the surfaces and micro dimensions.

Probably not even the same camera, on a tripod, on a studio controlled environment can take the exact same two pictures because of this quantum level noise. Ok, probably yes, but your example is not the case.

---

I do not understand the reason to take the same shot with two different cameras.