How can I take the photo with the focus on the main object and blurred background using the Nikon D5100? What kind of mode and settings I should use? Thanks.
This is called "shallow depth of field", there are 3 factor (that you control) that affect depth of field:
Distance to subject - as you get closer less will be in focus, also, you will get more background blur if the background is far away
Focal length - longer focal length = less in focus
Aperture - this is the easiest to use because it's a setting you dial into your camera but it's the "weakest" factor of the 3, to get less in focus use a bigger aperture (smaller F number)
The effect you are looking for is called Bokeh and it is a result of using a narrow depth of field. To achieve the look, you need to decrease your depth of field to put the background out of focus. You can minimize the depth of field by using the widest open aperture possible and if that does not accomplish what you are looking for sufficiently, then you can try to increase the focal length (zoom in) and shoot from further away. It's worth noting however that on many telephoto lenses, the aperture will actually stop down as you get in to longer focal lengths, so the best result will probably be on the longest focal length you can reach while maintaining a wide open aperture.
It is worth noting that while technically decreasing the distance to the subject or increasing the focal length has a larger change to the DoF than aperture, if you want to be able to maintain a consistent shot composition, getting closer to a subject will requires a smaller focal length and vice versa, so the change in depth of field is reduced some for those. Generally, the focal length will make the depth of field shorter than the distance will, so being far away with a long focal length will give you the smallest depth of field for a particular shot if aperture isn't sufficient.
You can use this blur calculator to figure out the best way to maximize your background blur.