It's read noise. The same amount of read noise is present in the entire photo, but in the brighter areas there's enough light to overwhelm it and you can't see it.
How to I fix this?
Shoot in brighter light and/or expose appropriately for the shadows.
If there's too wide a range between the brightest areas of the scene and the darkest so that the highlights would be blown before the shadows are exposed well enough, then as the photographer you need to add light to the darker areas.
I don't have it on a high ISO.
Even when shooting at low ISO, when you underexpose the shadows you'll get more noise in the shadows. High ISO isn't really what causes increased noise. The lower amount of light we usually allow into the camera when using higher ISO is what causes noise. If you limit the light to the same level at ISO 100 that you would use to properly expose for, say ISO 3200, and then try to boost those almost black shadows by five stops in post, you'll see the same thing as using ISO 3200 with the same exposure time (Tv) and aperture (Av) when shooting.
For more, please see: