An attempt to answer simply:
We cannot practically capture enough information to store a complete breakdown, frequency by frequency, of all the different wavelengths of light present, even just within the visible spectrum. With RGB we can describe the colour of a pixel using just three numbers. If we were to capture the entire frequency spectrum of light, every single pixel would require not 3 numbers, but a graph of data. The data transmission and storage would be immense.
It's not necessary for our eyes. Our eyes don't just see three single wavelengths, but instead each of our "red", "green" and "blue" receptors capture partially-overlapping ranges of light:
The overlap allows our brain to interpret the relative strengths of the signals as varying colours in between the primaries, so our vision system is already pretty good at approximating an actual wavelength given only the relative signal strength of the three primaries. An RGB colour model reproduces adequately this same level of information.