While I don't have the code of Photoshop I just can guess.
Exposures are given in logarithmic units, which makes them not so easy to predict as a linear value you may expect. Moreover, Adobe probably applies a gamma curve or preset or allows/prevents posterisation, arithmetic conversion of bits for the screen, etc...
So given log(relative ev)= log(ISO value)+log(aperture)+log(shutter speed)+log(luminance) +other values the software takes into account, you're expecting a value different of calculated. After decreasing/increasing the exp value in photo, some darker/brighter pixels may have clipped, adjacent pixels are changed according to colour gamut, etc... Color also changes in logarithmic values.
Also, supposing you don't have photographic values of shutter speed, aperture, etc... (Some jpg files) when you apply the formula expo=evVal*(2^NewEV) you do it to luminance calculation of the RGB, and all changes are converted back to the pixel in RGB.