What you're seeing is colour moiré. Basically, you're paying the price of having a lens and sensor that can resolve the tiny detail of the weave of the fabric, but not quite the resolution needed to determine the colour because we use a Bayer-pattern colour filter to determine the colour.
Note that the effect is only happening at a limited range of distances from the camera for a given fabric and zoom setting. What that tells you is that the camera can't quite cope with threads that are in a certain range of sizes on the sensor. Any closer, and they're big enough for the "figure out what colour it is" calculations have enough data to work with. Any further away and the threads are too small to be resolved clearly. Get the distance just right, though, and the pattern of threads line up in weird ways with the pattern of green-, red- and blue-filtered pixels. Instead of seeing a more-or-less neutral colour with lighter and darker areas (the shadows and highlights on and between the threads), it sees little areas where the red is missing or very bright, or the blue. (There are twice as many green pixels, so green is rarely a problem.)
The easiest way to fix moiré is to change either your shooting distance or your zoom setting. There are various ways to attack it in post-processing with varying degrees of success, but none of them are perfect. (You can also use a multi-shot camera or one with a full-colour sensor like the Foveon to eliminate colour moiré, which just leaves luminance moiré with even smaller details.) Changing your distance may mean cropping to get "the same picture", but it's better than weird colours that weren't there, or trying to fix weird colours that weren't there, especially if you care about the details.