Focus issue - possible cause?

Question

I took a whole series of photographs of local listed buildings, so I could add them to wikimedia - until I saw the results.

D5500 with Nikkor 24-120mm VR 3.5/5.6G 50mm ISO 100 1/320s f4.8

Full picture [jpg ⅓ actual size, click-through]

detail - screen-snap at x2 zoom

Pictures taken hand-held, but I'd say that's too much distortion to be motion-blur at 1/320, reinforced by the fact that all my pictures are similarly affected, even faster exposures. I was bracketing so some are at 1/600 or higher. VR was on. Also I had a polariser on to help with the clouds.

Detail of a similar picture taken the same day with just the kit lens at 18mm, 1/125 f4, also with CPL - same 2x magnification as above.
It's not perfect, but it shows nothing like the blurring of the 24-120

I've noticed this 24-120 showing bad colour aberration before, when using it on extension tubes for macro - so even under highly controlled conditions with good lighting & multiple manual exposures on a focussing rail, it still isn't good.

I would have otherwise blamed the CPL, perhaps giving some internal reflection, but I really have to come to the conclusion that this lens doesn't focus properly on my camera.
I have no means to tweak & save the auto-focus in-camera, it's not a feature on the D5500.
I would also perhaps blame the relatively wide aperture, maybe I should have stopped it down - but the kit lens turned out sharper at a wider aperture.

Is this just a "duff lens" in the broadest terms?
[I have to say I've never liked it much, but I was given it, so never look a gift horse in the mouth]

Is there anything in my admittedly newbie technique I can do to improve this, or is it just time to get a 'better lens'?

(Not used for this particular comparison, but I also have a 50mm 1.4 & a Tamron 70-300mm f4-5.6 neither of which shows such glaring focus-fail)

After comments:
I went back to the location to try some further experiments. I removed the CPL from the equation, but added a variable ND to my 50mm to reduce light & get longer exposures, just for comparison. These are [very] quickly assembled from my results...
_{[again, click to see larger]}

24-120mm
a) one of the bracket -1 EV shots from the first session, with filter & VR [to eliminate long exposure shake from the equation]
b) through d) no filter, no VR
b) tighter aperture
c)/d) original aperture, fast exposure

50mm for comparison
a) 1/40 to show how steady I can hold the camera - not perfect but OK-ish.
b) similar settings to my problem pics
c) comparison at tighter aperture
d) comparison at wide open

Answer 1

I believe it is motion blur from being handheld, as your sample crop of the sign shows a double image with a sort of drop shadow effect. I've encountered nearly the same issue when a shot is taken while the camera was still being steadied- I steady my camera with my left hand and guide the camera up to the left of my left eye when I shoot handheld. This creates a diagonal blur like what I see in the image.

Otherwise, the image is in focus, about as best as it can be for your chosen aperture, and that crop is of details that are pretty close to the resolution capability of your camera. It would be hard to discern any faults other than global softness, as you point out.

Here's your image again, with the area of interest marked in pink (Note the text: 616 WS or 616 W8).

It's a little easier to see the symptoms with a little increase in brightness and big increase in contrast (10% brightness and 40% contrast in Paint.NET).

Aside from the motion blur, everything else is okay with the image for what it is. I recommend testing again in a similar scenario with a bracket of shots, some with a tripod and and some without; and some with VR on and some with VR off (set VR to tripod mode if available when testing on the tripod too). There's a chance you may see continued softness or reproducible blur with VR on, which may suggest that VR is to blame on the lens instead.