Take a look at the CIE 1931 chromaticity diagram shown with the sRGB color space gamut. Why are certain colors intentionally left out of color spaces, like you see below? Why not just include all the colors?
sRGB is a color-space developed by HP and Microsoft in 1996. CRT monitors were common and therefore sRGB was based on the characteristics of these monitors' capabilities. A good write-up of the history and reasons can be found in sRGB: a short history.
The chromaticity coordinates and available colors were chosen on what the phosphors used in CRTs could produce back then. Consider that neither prints nor TFT or CRT monitors can replicate the full visible light spectrum.
A Program on a PC or camera that wants to control a monitor will use discrete values. If you use a larger color space, steps between different colors get coarse unless you use a larger datatype (Example: Adobe RGB with 8 bit). Whereas image information in a larger color space with a larger datatype uses more memory and needs more processing power (Example: Adobe RGB with 16 bit). This digital value will be transformed into an analog signal (usually a voltage) at a certain stage and then to something visible (for CRTs: a phosphorescent screen excited by accelerated electrons).
The resolution for converting a digital input to an analog signal is a further limit due to cost, size and technology.
Therefore fitting sRGB to CRT monitors back then allowed for a good resolution between colors while minimizing hardware requirements.
The CIE 1931 chromaticity diagram represents all colors that the average human eye can see. But just because those colors can be perceived by the average human eye, does not mean that all technologies can produce all the possible colors that the average eye can possibly see. While no tristimulus model can create the entire gamut of human color perception, the various RGB color models cover a very wide range of most of human color perception.
Realize that in the diagram you posted, and indeed any CIE diagram you have on a computer, it is just a model. The actual colors in the diagram outside of the sRGB diagram are actually represented by an RGB value in the image file. But the "pure green" at the top of the labeled sRGB diagram is not actually sRGB "pure green" (i.e., it's not an [R,G,B] value of [0.0, 1.0, 0.0]). The diagram is just a model showing, within the limits of technology, what is in included/excluded in CIE and sRGB color spaces.
For sRGB in particular, it was designed and standardized to accommodate CRT monitors in the mid-90's. CRTs produce color by emitting and combining light from three different phosphor guns (of particular red, green, and blue spectra). Lacking additional phosphor guns of different wavelengths, such CRTs cannot possible emit all the colors that humans can see.
Any color space based on RGB primaries will describe a triangle. Since the CIE diagram is not perfectly triangular, it is impossible to include all of them in a triangle without creating imaginary colors which can't physically exist. In particular the R,G,B values used in any sensor or display must lie within the physical colors. Note that this only applies to physical devices, there are color spaces that use imaginary colors for the RGB points but they're for mathematical manipulation only.
There are other constraints on the RGB points as well. First, it's better if they're achievable with cost-effective current technology. The points for sRGB were taken from Rec. 709 which defined the range to be supported by HDTVs in 1990. Second, spacing the points too far apart leads to problems differentiating between similar colors when your representation is limited, e.g. to 24 bits. It is better to have good representation of common colors than to have representation of colors that are hardly ever seen.
With more than 3 primary colors it would be possible to define a color space that isn't triangular, which would include more of the CIE space. Sony produced an RGBE sensor which included an "Emerald" primary somewhere between blue and green, but they only used it in one camera before abandoning it. I haven't been able to find any information on the CIE coordinates of the filters it uses, but here's a guess at what the gamut might be:
You can see that it covers a much larger area than sRGB, even though I used the 3 sRGB primaries as a starting point. It's hard to say for sure why it never caught on, but we can guess. Since the entire world of software and printing is based on 3-primary color spaces, the gamut has to be squeezed into one of those and any advantages to RGBE are lost in translation.
We normally describe a color by saying it's orange or cherry or pink. Go to a paint store and pick up sample swatches. You will see winter-white and flame red and perhaps candy-apple-red. Names like these fail to classify satisfactorily. One of the earliest and perhaps the best systems is the Munsell System. Developed by Albert H. Munsell, he arranged a three-dimensional solid of all colors that can be represented by actual samples made using stable pigments. I think is the best method.
Following was the CIE System (International Commission on Illumination). Experiments to map the color response of the human eye began in the early 1920’s. Students matched colors that were mixes of the three light primaries which are red, green, and blue. The cells in the human eye responsible for color vision were found to be a triad -- one pigmented to receive red, one green and one blue. It was found that one could mix these three primaries and make all the colors we humans can see.
However, science is unable to make perfect filters or perfect pigments. In every case we slightly miss the mark. The CIE system uses imaginary primaries. These can be intermixed to make all the colors we see. The fact that imaginary primaries are used does not detract from the value of the system. Perhaps you will be the one to make perfect color filters and redo the task.
The CIE system specifies colors in terms of the amount of each of the three primaries. This color mix is for a standard observer as thousands have been tested and the results averaged. A graph of the results is a horseshoe-shaped boundary that represents the position of the colors that have the highest saturation. These are the spectrum colors. The colored areas of the graph are the limits of saturation obtainable with modern printing inks. Near the center is the illumination point which is for daylight conditions.
Note that color as perceived using a Munsell system has a three-dimensional identification: which is hue, brightness, and saturation. The CIE system is two-dimensional. The straight line at the bottom represents magenta and purple of maximum saturation. These colors do not occur in the spectrum or rainbow; their hues are expressed as a wavelength. I can go on and on but maybe we should stick with Munsell.
Each pixel in a monitor display has a horizontal and vertical position on the screen. Within that position are three "colors" in a color monitor which are varied from 0% to 100% intensity.
If you look at the outer edge of the figure's region, then you see the colors that could be formed using all the phosphors which emitted light at pure wavelengths given same visual intensity perception. Within the region are representations of "100%" intensity of light perceived by the (red, blue, and green chromophores) of the human eye at the same visual intensity level. Think of drawing a line between any two pure wavelengths and varying intensity from 0-100% of first color and 100%-0% for the second.
Humans with good color vision have 3 different "color" receptors. So you can fool an eye into thinking that mixtures of three "pure" wavelengths form many different "colors". In such a case the intensity of the light would be varied between 0 and 100% for each of three colors.
Now the inner triangle has three points which mark the "effective color" (color mixture) of the particular phosphor chosen for the monitor. (The phosphors don't emit a pure wavelength of light, but a mixture of colors). So the red phosphor chosen limits how "red" the "pure red color" on the monitor can be. So on for green and blue. You can get an impression of the mixtures of colors which can be obtained with 100% power by using trilinear coordinates.
To get trilinear coordinates, first draw a traingle between the three chosen phosphors. Then draw a perpendicular line from each apex of the inner triangle to the opposite side. The apex of the triangle is 100% intensity and the intersect of the line with the base forms 0% intensity. Doing this for all three apexes will result in three lines meeting at each interior points within the triangle. If each line has 100 divisions, then there will be 10,000 points in the grid. Furthermore, the Red/Green/Blue intensities at each point will sum to 100%.
Notice that the corners of the triangle approach "pure" color of apex. Along the sides of the triangles there is a distinct transition when crossing from outside the triangle to inside. due to the different color mixing.
mattdm has pointed out that you also need to consider the overall "power" for the pixel. If all three phosphors have 0% intensity then the color would be black. If all three color intensities are 100% then the color should be close to white. To get white of course the three phosphors have to be selected judiciously.
There are device spaces and device independent color spaces. sRGB is a device independent color space created by a lady at HP as a space to standardize CRT's back in the day. Chris Cox at Adobe created Adobe 1998. and Kevin Spaulding at Eastman Kodak created RIMM and ROMM color spaces of which RIMM is used as ProPhoto RGB. That space actually does cover the XYZ diagram but is only beneficial to us photogs if our printer gamut is close in volume. (Most high end Epson's with good glossy paper get close to Pro Photo RGB)
The real issue is the end use of the image. The above color space profiles are mathematical models for devices and not actual devices. The benefits to those are they have equidistant primaries and transforms on images contained in these spaces are relatively well behaved.
Having color spaces that are not device spaces and do not contain the noise that device gamuts have. That provides for transforms to the actual device space such as the monitor on your computer or printer that are both predictable and more accurate from device to device. So container spaces are the way to go for quality.
Now to answer your question "Why not just include all the colors?" Well we can if we use ProPhoto RGB, but what we have then are RGB values (0-255) assigned to Lab values that are quite a bit larger than sRGB (the color space of the internet) so the image will not look right if you post ProPhoto RGB files to the web. So images that need to actually look like we want them to look must be converted to an out put referred space. On the internet that happens in your browser. If you have a high end monitor that happens because your computer has a known monitor profile to render the colors into the new Lab space.
It would be partly to do with efficiency of data encoding (not wasting bits/precision), partly historical reasons, and some practical considerations.
There are some colour spaces that do cover all "visible" colours, but we wouldn't normally use them for images/videos. For example, that chart in your question shows colours in the CIE 1931 XYZ space, which is a colour space that covers all colours visible to humans (according to its psychological model).
However, CIE XYZ is not a colour space that would normally be used to actually represent colour data, say in an image or video. The conversion back to an RGB space is relatively complex, it would waste a lot of bits of precision on space outside the range of colours most monitors can produce or sensors can see, even colours outside the space that humans can see. Mathematical operations that are simple to calculate in an RGB space would be highly complex in something like CIE XYZ and in all practicalities would require intermediate conversion anyway.
An RGB colour space makes certain operations a lot easier. Monitors and screens use RGB colour spaces natively. If you're using an RGB colour space because your output medium is inherently RGB based, it initially makes sense to use a colour space that equals or closely matches the red, green and blue primaries that your output medium can do. In the past, colour monitors used phosphors that produced similar red, green and blue primaries, so that RGB space just because the "standard" colour space. Monitors are not all equal, increasingly so, and so inventing a device-independent colour space is a good idea: sRGB is the most common device independent space and it closely matches typical red, green and blue primaries from the CRT monitor era. sRGB has become a de facto standard for monitors, televisions (rec 601 and rec 709, used in digital video, pretty much reproduce it), and now the web and operating systems in general.
So part of sRGB's popularity is its entrenchment in all those areas. As far as colour spaces go, and even as far as just RGB spaces go, it's very limited, and so you get Adobe RGB, ProPhoto, and the other RGB spaces with expanded gamuts. Encoding in them becomes just a little less efficient, necessitating the use of greater than 8 bits per channel in some cases, but they cover a wider gamut that new monitors and display technologies can do, and address a need for a "working colour space", where your input and output colour space can vary according to the device so you may as well use an intermediate space with a really wide gamut so it can convert between them with minimal loss. ProPhoto RGB, often used as a "working" colour space because it's "wide enough" to exceed just about any device colour space you can practically imagine, can cover almost all of the visible colours (according to CIE 1931) with the exception of some super deep greens and violets (again, these are far outside what monitors or other devices can display), but as a result it's fairly inefficient to encode, with many coordinates simply not utilised because they fall outside the range of visible colours. Interestingly its primaries (ie its red, green and blue) are "imaginary" - it is impossible to produce an emitter or sensor with the primaries of ProPhoto RGB because its primaries are impossible colours - they exist mathematically only, as a way to transfer colours to or from other spaces.
Smaller colour spaces are for:
- constrained image transmission. Using smaller colour space will improve colour accuracy compared to huge complete colour space given the same colour depth for both
- pre-rendered images, ready for viewing on target hardware which will apply no conversions before transmitting