I am a dentist and I have Nikon D5300 camera and a 105 mm macro lens. I tried to use them as a microscope for my work in treating people teeth (to magnify inside tooth pulp canals by more than 25x to 40x). But on doing that and with magnification, the details become lost and blurred vision occurs and picture sharpness goes away.
What shall I do please to have high magnification inside pulp canals with high magnification more than 25x?
what shall i do please to have high magnification inside pulp canals with high magnification more than 25x
Buy a suitable microscope. No DSLR lens will provide the level of magnification you need, particularly at a focal distance that makes sense for dentistry. Macro lenses get a lot of their ability to magnify just from being able to focus at very short distances -- it's like looking closely at something by holding it close to your eye. Beyond the obvious physical and hygienic problems with sticking a lens into a patient's mouth, getting close enough to get the view you want inside a small space will make it impossible to get enough light on the subject.
For dentistry, you need something that'll give you good magnification from far enough away to be hygienic, and also far enough to give you room to work. There are microscope models made specifically for dentistry that can give you the magnification you want from a distance of 200-300 mm. They often have their own light source, and they sometimes include a camera as well. I expect that the high cost of some of those microscopes can be a problem, but it looks like there are also models that aren't so expensive.
I am understanding several issues here. I am going to separate them and you can choose an option.
Then "Blurriness" could mean that either you are not focusing well or that you have motion blur. If that is the case you probably need a ring flash.
https://www.google.com/search?q=ring+flash+macro
If you are not focusing well either you need to practice or you need to take a look at your lens, for example, if it is autofocus.
But it can also mean that you need to practice your "macro photography skills" like holding your breath, holding the camera firmly, etc.
It can also mean that you need to understand about motion blur if you are using a slow speed, so probably you need to increase the sensitivity of your camera, increasing the ISO.
So in optics terms, 25x means A LOT! of magnification, and I mean a LOT.
But as you are a dentist you probably do not really mean that.
In this other question: What kind of lens to photograph a 1 mm object?
I posted this image of some sugar grains. 
The optical magnification is not that great. It is less than 2x (1.8)
But if you see the image for example on a FullHD 23" monitor, at full size you can see the sugar grain of 1mm at about 11cm. This is 110 times larger.
So you need to see what is in reality what you need.
It has a specific magnification, a fixed focal distance, it can be easier to clean, it's compact and probably has a built-in light, and live view, so you can use it to see in real time.
Let's look at the capabilities of what you are using.
Your Nikon D5300 is an APS-C camera with a sensor size of 23.5x15.6 millimeters and a resolution of 6000x4000 pixels.
Your AF-S VR Micro-Nikkor 105mm f/2.8G IF-ED lens has a maximum reproduction ratio of 1:1 at the the lens' minimum focus distance of 0.314 meters (1 foot).
This means that the image of your patient's tooth projected on the sensor of the camera will be life-sized when the lens is focused at its nearest focusing position and the camera's sensor is exactly one foot from the tooth.
To view that image at 25X magnification means you would need to display the image, or a portion of it, at a size of 25X the size of the camera's sensor. This works out to 587.5x390 millimeters, or about the size of a 28-29 inch monitor. You can use a smaller monitor and zoom in on only the part of the image that has the tooth of interest. But you'll still need a monitor large enough to show the image at 100% (one image pixel per screen pixel) for the area that contains the tooth.
To get an image clear enough to see sharp details at such magnification, you will not be able to handhold the camera, no matter what shutter speed you use. You're going to need a rig that can position the camera precisely at the correct distance above the tooth you wish to view. Your rig will need to be able to remain rock steady with absolutely no movement while the exposure is captured, probably with the assistance of a fairly powerful flash unit so that the exposure time can be short enough that patient movement does not affect the clarity of the image. You'll also want to use a fairly narrow aperture, say around f/8 to f/11, to get as much depth of field as possible without giving up very much sharpness to diffraction. This narrower aperture also requires more light to get a proper exposure.
Even then, a single molar crown with an 8mm width will only occupy about one half of the image height and one third of the image width, or about one sixth of the total image area. That will give you a relatively low resolution image of the crown of the tooth of about 2000x2000 pixels or 4 MP. To view that on a monitor at 100% will require about 4K resolution (3840x2160 pixels).
It’s not likely that you will achieve high magnification. If you do, likely the final contraption will be too complicated --- thus awkward. So the question is: do you have an optical device that allows you work at this high magnification? If so, likely it has an eyepiece that you peer through. Such a lash-up presents to your eye, parallel rays. You can aim your camera so that it is peering through the eyepiece. From the camera’s viewpoint, the image it sees is coming from infinity; it’s the same image you were viewing. Mounting the camera to peer through the eyepiece is called the “afocal” position. Try it, it works.
Sorry, you cannot turn your present configuration into an endoscope which is purpose-made for what you wish to do.
An endoscope (fibre pack optics) has its own axial illumination in addition to a capability for high magnification. You know that, already.
The/an endoscope might seem to be overkill for external use; but, in terms of asepsis, it will be easier to sterilize after each patient than draping your present gear with surgical towels would be.
All of that said, there may be an endoscopic lens attachment available for a Nikon mount. It may not perform to the level of a committed device however.
