Traditionally it has been mostly imagination because the absorption by interstellar dust clouds prevents us from seeing, in visible light, beyond a zone in the Milky Way that is only about 20% of the distance to the center of the galaxy. That is only about 4% of the entire disk of the Milky Way was visible. Of course, we can see indefinitely far when we look out of the plane, but then our lines of sight pass out of the Milky Way disk.
In the mid-20th century, radio astronomy allowed us to see interstellar matter across the entire galaxy, in the form of the 21-cm spectral line (1.4 GHz) of neutral atomic hydrogen (HI). We could then see evidences for giant spiral arms, but distance determinations to the interstellar gas are very approximate, so we could just get a general idea of what it might look like, as viewed by an outside observer. Later, detailed observations of the 2.6-mm spectral line (115 GHz) of interstellar carbon monoxide (CO), mostly located in molecular hydrogen clouds, gave us a more detailed insight since molecular clouds define the spiral arms of our galaxy much more closely than do the more diffuse clouds of neutral hydrogen.
Since the emission of HI and CO is in the form of spectral lines, we can determine the radial velocity (speed towards or away from us) of each emitting cloud from using the Doppler effect. Using a model of galactic rotation, we can then make a map of the spiral arms over the entire galaxy.
But these were still very approximate because (1) each cloud has its own motions which cause a distance error when comparing observations with the model and (2) there are two possible distances corresponding to a given radial velocity for portions of the galaxy closer to the center that we are.
So we knew of large spiral arms and roughly where they were, but the detailed picture was still mostly imagination.
More recently, detailed observations in the infrared have allowed us to see deeply into regions of recent star formation all through the galaxy. These regions of recent star formation define the spiral arms much better than the observations of interstellar gas, but a lot of work involving comparison of both infrared, radio, and microwave observations is needed to come up with a good general picture. See A Map of the Milky Way for a good view of our current understanding of what the Milky Way galaxy might look like as viewed from outside.