Could someone please explain the difference between the two?
Also are there cases where the one technique is more preferable over the other or a general rule of thumb?
HDR is a multi-image technique that takes a sequence of images of different exposures to produce one that shows a greater dynamic range than a camera can capture.
HDR is used for scenes of excessive dynamic-range. It extremely difficult to do without a tripod and moving subjects cause serious problems because pixels have to be perfectly aligned to be merged correctly.
Adaptive D-Lighting works from a single capture and serves to reduce contrast so that details are more visible, particularly in shadow areas.
From a RAW file you can do this yourself by using tone-curves and otherwise changing contrast non-uniformly. You can also partly do this from a JPEG but will see banding in shadows. Regardless, this tends to increase noise in dark parts and make images duller.
There are very few cases when I have seen good results from Adaptive D-Lighting and enabling the feature makes results less predictable.