There is without a doubt a noticeable difference. The smaller sensor size, as you mentioned, gives a Micro Four-Thirds camera a disadvantage when it comes to low-light performance. The real question is: How much is this difference?
Let me preface this by saying that I have seen and reviewed nearly every Micro-Four Thirds cameras on the market, as well as most APS-C offerings from entry-level ones such as the D3300 to the latest D500 (plus Full-Frame DSLRs and one Medium Format System), so I know what the difference is by experience. You will find most of my reviews on my own Neocamera, although I also review for third party websites and print publications.
What I have been observing, regarding Micro Four-Thirds vs APS-C, is that the performance has been closing until about a year ago when high-end APS-C DSLR took a significant leap. This means that the difference between an Olympus OM-D E-M10 Mark II and a D3400 is less than between an E-M1 Mark II and a D500. Here is an image illustrating the point:
On the left you have the current top-of-the-line Micro Four-Thirds camera, the Olympus OM-D E-M1 Mark II, in the middle the Nikon D5500 which entry-level DSLR and, on the right, the top-of-the-line Nikon D500. As you can see, the first row is taken at ISO 200 and noise levels is very similar. The E-M1 Mark II has a tiny bit more than the others but it would not be noticeable on a typical print. The next row is taken at ISO 1600 where you can see that both the E-M1 Mark II and D5500 become visibly noisy, although you can make out details better on the D5500. On the other hand, the D500 still looks nearly impecable. The final row is taken at ISO 12800 where it is easy to see that the Olympus is way behind the rest. The D5500 got softer which is Nikon's way of hiding its noise. There is some of this on the D500 but its image remains much more usable.
There are different aspects to image-quality and certain circumstances show issues more than others such as handheld low-light photography which requires high sensitivities. When it comes to noise, the current crop of Micro Four-Thirds camera behave similarly to APS-C ones at low sensitivities, showing virtually no noise until ISO 400. As ISO is increased, both sensor sizes start adding noise but APS-C cameras really manage to keep noise low much longer. Once in a while, I do a comparison just to see where the systems are at. Here is an interesting one - although slightly outdated - pitting the Fuji X-T1 (APS-C mirrorless) against a Panasonic GH4 (Micro Four-Thirds mirrorless).
Dynamic-range is always an issue. The top-of-the-line Micro Four-Thirds cameras are still about 1 1/2 stops behind in dynamic-range compared to even older APS-C cameras. This will be a problem when shooting highly contrasting scenes.
The point is that there is a difference and a clear advantage in terms of image-quality of APS-C digital cameras but there are also great advantages to a Micro Four-Thirds camera, most notably size and weight. For the ultimate in image-quality you will need a larger sensor (even Full-Frame) but if image quality from Micro Four-Thirds cameras is good enough for your needs, you might as well consider all the advantages you are getting.
Feel free to read my review of the Olympus OM-D E-M10 Mark II and do check out its gallery. It has full-resolution images at all ISO taken right from the camera. The images will speak for themselves.