In short, pretty much yes!
In detail, there are several issues involved and noise performance is only one.
The point is, in low iso you might not feel the noise difference comparing to a higher sensor but one fact that some people forget is the depth of field. I made it bold because it has bold effect on your photos.
The shallow dof that a big sensor can give you is almost impossible to achieve on a small sensor.
Another important factor is the dynamic range. Again, the dynamic range of a full frame or aps-c sensor is more than a 1/2.3" and if you have accurate eyes, you can detect it easily.
Regarding your question about a small sensor being an advantage for lens design; it is somehow true. When dealing with smaller sensors (at least on dslrs), the rear element of the lens can be closer to the sensor*. From wikipedia:
The proximity of the rear element to the image sensor greatly enhances
the possibilities for wide angle and very wide angle lenses, enabling
them to be made smaller, lighter (containing less glass), faster
(larger aperture) and less expensive.
But the problem is, compact cameras are designed to be cheap and the above statement can not be considered as huge privilege for compact cameras.
Compacts can give you good results, for a compact, but their image quality is not close to a DSLR (specially if you own a full frame) because of many reasons: Noise, dynamic range, depth of field, vignetting, chromatic aberration and more.
So, they are good for an always-in-your-pocket camera and also very good for their cheap prices but you can not compare them to a good DSLR.
that's why you can not mount a ef-s lens on full-frame body: the mirror will hit the rear element of the lens!