What causes this dark outline produced at the edge of objects in these images?

Question

I'm working on perfecting a green screen algorithm and noticed I was getting a very pronounced dark edge around objects in some scenarios. When trying to figure out what was causing it, I noticed my algorithm was not at fault - the dark outline actually exists in the source images (the background is an actual green screen, there is no material editing involved in the source image at all), look at the subject's nose and lip here:

And at the edge of her face here:

If you look closely, it's not hard to see that the dark outline is in the source images. Is this some kind of natural effect of light? A flaw in the camera? An effect of image compression? Do we perhaps see such effects with our own eyes and just not realize it?

Answer 1

To me it looks like the kind of halo you get in USM sharpening. Note that you get a light outline next to the hair in the second photo.

If those images with the background are camera-produced jpegs, there's not much you can do to remove them. If you have the raw files, test without sharpening and do any required sharpening after removal of the green background (or after final compositing)

Answer 2

No, your algorithm is at fault. Here is a blow-up of a small area of your second example:

You are not properly handling when the foreground subject is partially blended against the green background. Ideally, the original image is point-sampled. Each pixel would then represent either just the green background or just the foreground. Your input images aren't like that, and your algorithm doesn't handle it well.

The best way to deal with this is to start with pictures that are much higher resolution than what you ultimately want. That approximates the point-sampled case more, and any blended transition region will be a smaller fraction of the overall picture dimension.

Proper blue or green screen algorithms do try to deal with this case, although it gets tricky and you can always concoct cases that can fool it. Put another way, the algorithm has to essentially guess about information that is not there.

One way is to do special edge processing. In the transition region between solid foreground and solid background, you have to include some additional logic. At that transition region, you assume that any hue change from the foreground in the direction of the background hue is partial blending with the background, and then remove it. Again, this can be tricky, and it can be fooled, but most of the time it will yield more of what you think it should. Even relatively simple logic will fix the case I show above. Other cases may never be fixed to your satisfaction.

Again, the better answer is to start with a high-res original.

Answer 3

It might be the camera pixel-averaging artifact, as an image from the sensor is usually postprocessed, for example to remove physical sensor pixelation or boost sharpness.

Besides each sensor pixel averages light by default, as it has non-zero size. E.g. on edges some green is mixed with the skin color in those pixels.

Also sensor thickness adds to the effect by slight light pollution from nearby pixels.