Olympus OMD EM5 is a mirrorless camera, hence no PDAF. So you would have very small chances (read: almost impossible) to use CDAF to a correct (spot-on) automatic focus at night.
Most probably you are at some distance from your subject, thing which makes the depth of field to grow a lot at closer apertures.
Hence, most probably, you got the photos at closed apertures "sharp" just because it happened to have your subject in the much greater depth of field given by the closed aperture.
The solution is, if you really want to use a small aperture (eg. to isolate your subject), to use Live View, zoom in (if your camera has a "Zoom in" feature in Live View), and do Manual Focus on your subject.
Manual focus is the by far the most precise focusing method, even in daylight, but you need to have your camera and your subject to stand still during focusing.