I have removed the IR cut filter from a Sony digital compact camera so it has become multispectral (in a way). In addition to that I placed a IR pass filter, so it only captures photos in the Near-Infrared spectrum. I was wondering if there is a way to create 4 different bands (R-G-B-NIR) — instead of the standard 3 RGB bands — by handling the raw files.
Not really. If you want to take a hack at it, the first thing you'll need is the transmittance curves of your (presumably Bayer) color filter. Then,if you're lucky, there's an NIR wavelength at which at least two color filters have some responsivity, and those color filters have no overlap in the visible range. In such a case, you could play with sum and difference values for the pixels corresponding to those two colors and perhaps tease some NIR-only data out of the mess.
I sincerely doubt, based on the Bayer filter curves I'm familiar with, that you'll have any luck doing this. Frankly, the best way to get NIR images would be to take two exposures, one with and one without the NIR filter in place, and difference those.
It's going to be tough putting different bands of IR into the different colour channels of the sensor.
The biggest challenge is that commonly-available IR filters are all low-pass, not band-pass filters.
Bayer filters normally pass a significant amount of IR, so a camera that has been converted to "full spectrum" by replacing the "hot mirror" with clear glass, has a lot of different information available.
The good news is that you can emulate Kodak Infrared Ektachrome film, by mapping the entire infrared band to red, red to green, and green to blue. This is called "IRG to RBG transform," and there's a long, detailed conversation about it on flickr that you can follow.