Might be a naive question, but I want to fix few subjective metrics / parameters which would let me decide image quality.
I mean, I can use some objective metrics like Mean Squared Error or Mean Absolute Difference, if I have the "golden reference image", but in real life situations, most of time there is no reference image. In those cases, how do I decide the quality of image by viewing it?
I was going to just pop this in as a comment, but what the heck, I think I'll battle a little against the idea of objective metrics. Objective metrics are good things for camera reviews, not so much for photographs because you take the art out of the equation with this concept. People have listed some good, relevant, factors in what makes a "good" image, by definition, and yet you could follow every single measure and still produce an image that is utterly lifeless and dull.
In this century we're often obsessed with hard numbers, quantifiable information. Cameras, today, can operate at speeds and in conditions that cameras in the golden age of film could not and so far beyond the ability of early cameras that you almost can't compare. There are images from these bygone eras that would fail some of these objective tests, such as sharpness, noise, grain, etc. and still they're masterpieces of the photographic art. Sometimes, imperfection is what makes the image something special! Lomography is premised on this very thing and there have been some seriously stunning images taken with toy cameras that will fail all of your objective tests. There's also Lensbaby, creating lenses to do the opposite of what many would cite as objective criteria for a good image.
So, in my opinion, there's really only one objective measurement for the quality of an image in my eyes: does it evoke something for me. That's all. It can evoke wonder, intrigue, the science geek in me, the romantic, whatever, but when an image makes me take a longer look at it, then it's a quality image in my book. I can be objective here, the only emotion that causes me to ignore is "blah" and I know it when I hit it. :)
The quality of an image can be judged in three broad categories:
Technical quality
Here we have factors like sharpness, exposure, color, white balance, depth of field, noise/grain, etc.
Compelling emotion or insight
The photo must elicit some emotion that makes it memorable. A compelling photo provokes an 'aha' experience, hard to describe but easily recognised.
Aesthetically pleasing
This is what we usually call composition. It includes things like line, shape, color, tone, balance, perspective, etc.
A good photo will score well in all three categories. If you must rate photos then I suggest you score them on a scale of 1 to 5 in each category. The big question though, is, does each category have equal weight? My personal preference is to give more weight to 'compelling' photos. My advice is not to get too detailed about individual aspects of the three categories. go by your overall impressions of Quality, Compelling and Aesthetics.
Thomas Carlyle said 'There is nothing more terrible than activity without insight'. He could easily have been talking about photographers. The bottom line, for me at least, is that a good photograph provides a compelling insight.
Just to round things off, here is a counter-intuitive view of the subject
Start out with Stan Rogers's answer to the question "How to capture the scene exactly as my eyes can see?". Think about what exactly that "golden reference image" you describe would look like. Would everything in the scene be in crisp focus? Would colors be as we observe them, or would they be as a perfect sensor would record under a perfect light source? How would the image be framed and cropped? Would vertical lines appear straight, or would they converge? Would it be three-dimensional?
So, I think you're going about things in a very wrong direction. Even if you were to have some sort of magical way to measure against a Perfect Reference of Reality, the quantitative, objective things you could calculate wouldn't relate very well to what makes a good photograph.
Now, that said, there are some things you can consider to assess the technical qualities of a photograph. Unfortunately for your goal, these aren't really measurables — but they are aspects to examine.
There's certainly more, but there's some places to start. As you can see, some of these get into very subjective areas. I've intentionally left "composition" from the list, but it really shouldn't be missing. In fact, it can't be removed from what I've already put there — it's part of all of the above already.
And keep in mind that while technical things are important to making photographs, they're not the heart of it.
Photography is first and foremost an art, not a science. The 'quality' of an image is subjective: if you like a photo, it's a good photo. There are rules of thumb that aid composition, such as the Rule of Thirds, but these aren't hard and fast laws - they won't always produce a good photograph.
Objective quality compares an image to a reference image, and will computationally work out a quality factor based on some sort of algorithm (PSNR, SSIM etc). You need a reference image to measure objective quality.
Subjective measures tend to involve getting people to rate or rank images, and to rank images against each other they don't need to be compared to a reference. Of course it them becomes statistical, so the more data you have the more sure you can be about there being a real pattern behind the results. It also stops being about qualities that can be objectively measured and becomes more about aesthetics or perception.
You set up a bunch of images and get people to rank them according to technical skill, pleasantness, or whatever - you have to decide how to phrase it depending on what you want to gauge.
Anyone remember hotornot.com? That was, believe it not, quite a good example of someone taking a subjective measure of image quality - though it's probably not the 'quality' you are going for.
