The only reason that we even bother with this concept called "crop factor" is because of the absolute prevalence/dominance of 35mm film. Do you think the early shooters, big box view camera's in hand, were conceptualizing their lenses using a crop factor?
The focal length of the lens, such as 50mm, is a characteristic of the lens. This is considered the "normal" lens for 35mm format (What is a normal lens?) and it's one that we are all very used to using. But, when smaller digital formats became prevalent, with many people maintaining the use of their 35mm lenses on these formats, there needed to be a way to communicate to those people what the lens would "look like" on that format.
Enter the crop factor. An easy way to tell someone how their lens would look on a small format. You've been shooting 35mm for decades, know what a 35mm and 50mm and 85mm look like better than the back of your hand. Now you use an APS-C and you toss that 50mm on there and, because of knowing that it's a 1.5x crop factor, you know that your 50mm will "look" like a 75mm on that camera.
The exact thing holds up for medium format. Many, many people start with 35mm and then go up to shooting 120. Conceptualizing the lens use can still be done with the crop factor, it's just that instead of "cropping" you are actually getting more area captured. So, for example, the crop factor for 645 is roughly 0.62 going from 35mm to 120 in the 645 flavor. So, the 50mm lens on a 645 camera would appear to capture like a ~31mm lens on 35mm.
"Normal" by the way, becomes about ~75mm for 645. Take it out to large format - it's ~150mm for a 4"x5".
The key concept here is the conceptualization of the image that will be captured by a given lens on a different format.