In principal the base ISO is determined by how much light can hit the sensor before the individual photosites become saturated (i.e. the signal they produce doesn't not increase in response to extra light). This is in turn affected by factors such as the electron well capacity of each sensel (how many electrons can be stored before saturation), the efficiency of the microlenses, transmission rate of UV, IR and low pass filters in front of the sensor etc.
High base ISO is not necessarily a bad thing, it can indicate the sensor is very efficient at gathering light. Likewise a low base ISO (along with allowing longer shutter speeds / larger apertures), may be good as it can indicate a high well depth which will allow more photons to be captured for lower noise / higher dynamic range.
edit: all things being equal fill fraction (the percentage of the CCD or CMOS real estate that contains light gathering circuitry) doesn't affect the base ISO.