The problem is probably not directly that you scan with colour settings, but that you scan with dust and scratch removal enabled. Epson calls this technology 'Digital ICE' and you should be able to find a setting for it in your scan software.
When scanning colour films, 'Digital ICE' is likely per default enabled. When scanning black and white films, 'Digital ICE' is likely per default disabled. Therefore the problem might seem to be solved by switching between colour and b&w modes, but if you for some reason want to scan b&w film with colours, you should be able to do so by scanning the film as a colour film, but you will then have to disable 'Digital ICE' manually.
The 'Digital ICE' technology for automatic dust and scratch removal scans the film under infrared light. Colour film is almost transparent to infrared light, so whereever the scanner 'sees' a shadow or less light, it interprets it as dust, a scratch or otherwise damaged area of the film and somehow tries to guess what it was supposed to look like by analyzing the surrounding area. The silver in b&w film is however not transparent to infrared light, so if you scan b&w film with ICE enabled, the scanner will see much of the real image as damaged and try to repair it.