Speed and contrast is determined mainly by the recipe used to make the film. The chief ingredient is crystals of silver salts (there are three). These are called silver halides (Swedish for salt maker). The crystals are grown in a gelatin solution. The size and shape of the crystal are carefully controlled. Fast films have a tendency to have larger crystals as these have a higher probability of being hit by photons during the exposure. The three halogens in order of sensitivity, least to greatest, are iodine, chlorine, and bromine.
If pure, these crystals have low sensitivity and respond only to violet and blue. Impurities encourage higher sensitives, plus the shape of the crystals is also modified. The crystals are then dyed to expand their sensitivity to the various colors. The final film is a hodgepodge of different size crystals. Fast films contain a more coarse mixture. Slow films contain tiny crystals.
Films with a broader range of crystal sizes have a broader scale with inherently less contrast. Small slow films thus are softer in contrast, however they have a finer grain structure.
Negative films are only a means to an end -- the end product being a print on paper. Thus the photo paper used to make the final display has a profound effect on the contrast of the displayed image. A simple color film can be constructed using three emulsion layers: one for red, one for green, and one for blue light. A modern color film will have multiple emulsions for each of the primary colors. As an example, a fast red, medium red, and slow red.
The actual ISO and contrast is also a function of the processing chemicals. Because color materials are exceedingly adjustable by the chemicals and timing of the process, the chemicals and the specifications are rigid if the final ISO, color balance and contrast, are to be as specified. Any variation in the processing and printing will induce unpredictable differences.
Digital cameras only roughly use the ISO (International Standards Organization) methods. These standards pertain mainly to photographic films. Digital manufacturers only loosely follow the rules.