In rough terms, each photon that impacts a photosite induces one electron of charge to be stored for that sensel. There's a limit on the number of electrons the site can store, which (in turn) acts as a lower limit on the ISO speed rating. That leaves only a few ways to lower the base ISO.
The first and most obvious would be to reduce the number of electrons that can impact the photosite. The primary way to do that would be to insert filtration in front of the sensor. If you used neutral density filtration, this would only accomplish the same thing you already can by putting an ND filter in front of the lens. The difference would be that you couldn't change or remove the filter -- ever. An ND filter in front of the lens is clearly better in almost all respects: it can be removed when light is low, swapped for a stronger filter when light is particularly high, etc.
Another possibility would be to use stronger color filtration. This would at least potentially provide a small advantage in providing a broader gamut. The problem is that current cameras already support a gamut that's as broad as most people care about. To an extent this seems to already happen anyway. Just for example, Sony sensors appear to use slightly stronger color filters than Canon's sensors. This does (at least according to sites like DxoMark that test such things) give a slightly broader gamut. It also, however, gives noisier pictures at higher ISOs. Based on sales of (for example) the Canon 5DII versus the Sony Alpha 850/900, it would appear that a lot more people find other factors like video and/or lower noise at high ISOs more important than the broader color gamut at low ISOs.
There is yet another possibility (that has also been implemented in real cameras). This is to simply simulate a really low ISO by taking multiple pictures and averaging them together to minimize noise. This has a couple of obvious problems. First of all, it's only good (as low ISOs tend to be in general) for still-life type subjects. Using the ISO 6 you mentioned as an example, consider that the "sunny f/16" rule says the normal daylight exposure should be 1/6th of a second at f/16, or (for example) 1/250th at f/2.8. with a "pro" level zoom, you'd still be shooting wide-open to just edge into the lower end of the range where you can hope to freeze much movement.
I think from most people's viewpoint, there's a simple lack of need: below ISO 200 (or so) most current cameras have so little noise that trying to reduce it further is simply pointless, at least for most people. Quite a few routinely shoot at ISO 400 (or higher), even in broad daylight, just because dropping the ISO below that doesn't gain enough to be worth bothering, at least in most cases.