This is an image processing question and belongs to stackoverflow I think. But I give it a try to clarify the issue, however, you have to consider my computer vision and image processing knowledge is narrow.
Overlay mode, is a combination of multiply and screen modes. On the other words, if pixel on image A is darker than 50% gray, pixels of A and B are multiplied and otherwise, it would be a screen blend mode behavior which is like an inverse of multiply (image looks brighter). That's why overlaying an image with itself, looks like an increase in contrast.
from wikipedia article of blend modes:

Now having this in mind let me make you an example. I'm going to use two images which are the courtesy of this and this users. I found them on the image of the week page and copied them from their flickers at + and +.
This is the overlayed images:

The point is, human eye can detect there is photo of a flower and a photo of birds but I'm scared CV algorithms are not that powerful. This is the edges extracted from the first image:

If you open these two images on gimp or other photo manipulation softwares and try different blendings, you will see that you can extract too much useful information about the other image.
The problem is that based on definition I quoted above, on overlaying, we multiple some pixel values and this can simply cause the loss of certain data. Here is another example:
This is the overlayed images:

And this is the the first image:

Can you guess how the image under looks like? Here it is:

As you see, even if you have the exact edges of the firs image, you can not certainly decide about the second image. This is due to nature of the overlay blending mode.
__
Disclaimer: What I wrote above is based on my limited knowledge of CV and there might be techniques that can effectively solve your problems that I am not aware of them.
P.S. Here are the original images of the examples above:
http://www.flickr.com/photos/akshaymhatre/6707573177/
http://www.flickr.com/photos/mikenz/5088355393/