There are a number of ways to do such a conversion. I'm assuming you want the user to be able to choose a color that should be neutral in the image, but is not.
My guess for Lightroom is that they probably convert the RGB values into the Lab color space, then adjust the a
and b
values of each pixel by a particular amount. (The amount being the difference between the original color and a zero value for a
and b
.) You would probably want to skew the L axis of the entire color space so that the chosen color lines up with where a
and b
are 0 in the unskewed space.
You can approximate such a change in other ways. One way is to convert from RGB to something simpler like Y'CbCr, and adjust your colors there. Again, you'd skew the Y' axis so that the color the user originally chose is moved in its plane to the origin (where Cb and Cr are 0).
You can mimic this in straight RGB by rotating the gray axis (r=g=b) to be vertical, skewing this new vertical axis so that the chosen color ends up on the axis, and then rotating by the reverse of the first rotation.