How can I resize RAW image files (.NEF) from 12 Megapixels to about 5 megapixels without converting them to JPG?
You have to convert them to some other format, but that format doesn't have to be JPEG. For example, you could save the files as TIFF or PNG instead of JPEG. RAW files are data read more or less straight from the sensor, so it doesn't make sense to "resize" such files. You have to instead process them into a useable image format, which you can then manipulate to suit your purposes.
You could use dcraw to convert the raw image data to a "raw" tiff file that only contains the raw image data. You do that by giving the command "dcraw -D filename". This will produce a tiff file without any demosaicing or scaling. Such a tiff file is then smaller than an ordinary tiff file because each pixel is then only either a "red", "green" or a "blue" one. You can then edit the tiff file using e.g. ImageJ and resize it, but you then need to take into account the Bayer pattern. This can be done by considering only the pixels corresponding to each of the colors. You can split the image up in 3 parts such that each new image only contains the pixels of one of the color channels. You then resize each part and then compile a combined image by taking the 3 images and arrange the pixels from them according to a Bayer pattern again.
To understand why you can't do this, it's helpful to understand how RAW works.
A RAW doesn't actually contain colored pixels, it's a single channel ("gray scale") image representing alternating red blue and green pixels called a Bayer pattern. To actually get the "actual" pixels, you have to extrapolate from each of the pixels neighbors using a complex algorithm called de-bayering.
To down-sample a RAW image without first de-bayering ("converting to jpeg"), the only solution is to down-sample by a integer factor in each dimension. This means the image size can shrink only by factors of 4,9, 16 etc.
In your case, this means that the largest 12 MP can be scaled down is 3 MP, assuming you do not want to de-bayer in the process.