I think both cameras are wrong for you:
Nails are small, to fill the frame with a nail you need a lens with macro capability, the 18-55 kit lens can't do macro.
The most important thing for getting images just right is getting the lighting right, the FZ45 does not have a flash hotshoe and can't use external flashes.
You can get the D5100 (or any other DSLR) with a macro lens instead of the kit lens (or in addition to it, the kit lens can still be useful).
You can get a mirrorless camera with a a standard hotshoe and a macro lens, this will be cheaper than a DSLR but you will not have such an extensive selection of cheap Chinese accessories like you can find for a Canon or Nikon DSLR.
And you can also find a fix lens compact camera with an hotshoe, but high-end compacts tend to cost as much as a mirrorless camera or an entry level DSLR making them a bad deal.
Once you start taking pictures with your new camera you are likely to find that for exactly the image you want to make you'll also need an external flash and some flash modifiers, for video you'll probably need a tripod.