It's definitely possible to determine the relationship between the value of a pixel in a RAW file (measured in analog-to-digital units) and the number of incident photons. Modern CCD and CMOS sensors are highly linear, so in theory there should just be some arbitrary conversion factor.
Actually doing so is a different matter. To understand why, we need to look at what happens between a photon striking the sensor until the final digital value of the pixel is determined. Here's a rough chronological outline:
The photon first encounters the color filter array, which may scatter the photon with a probability dependent on its wavelength.
The energy of the photon is converted to electron-hole pairs in the photosite; the rate of conversion is the quantum efficiency. For sensors manufactured in the past 10-20 years this value usually peaks at anywhere from 40%-90%.
At the end of the exposure, the accumulated charges are read out and digitized. The nature of this process is entirely different between CCD and CMOS sensors. Before digitization the analog signals may undergo amplification. The applied gain is determined by the camera's ISO setting.
A good approximation of gain (ADU per electron) can be made using some basic calculations on the statistics of dark noise. However, the uncertainties introduced by QE and color filter transmittance make translating this value from electrons to photons a fairly difficult task. I am not an expert in this field, but to my understanding you will probably need some sort of calibrated light source to determine these values for certain.
If your intent is to count individual photons, a consumer camera is not going to be a very good option, primarily because of unavoidable read noise. There are technologies built for this purpose like photomultiplier tubes (or EMCCDs if the photon counts must be spatially resolved).
Addressing the concern you expressed in your question:
I'm worried that there are some settings on the camera that will skew the RGB pixel intensities for example like white balance.
White balance settings usually do not affect the outputted data. However, many cameras are known to apply various adjustments to the raw data obtained from the sensor. Some common ones are channel scaling and various spatial filters to combat noise or hot pixels. Nikon's old "star eater" filter is a particularly notorious example. In some cases these adjustments may be disabled using unusual methods like firmware hacks. Check out Roger N. Clark's page on the subject for more info. (But frankly, these adjustments will be the least of your problems.)