I bought a Nikon D5300 and a Yongnuo YN565-EX. Can I use the flash off-camera using any mode or something without a separate trigger?
You can use the Yongnuo YN565EX in
Optical Slave mode, where it will detect and be triggered by another flash (for example the built in flash of the Nikon D5300). You would need to have the built in flash firing, and you can enable optical slave mode on the flash (S1 and S2, depending on whether you want it to ignore the TTL pre-flash). Take a look here: http://www.davidpartington.com/m43/2014/12/16/yn568ex-ii-remote-slave-mode/ for information on the optical slave mode.
Remember that the master flash on the camera will need to be bright enough that it can be detected by the Yongnuo slave, and also if the camera is using TTL metering and you've set the Yongnuo slave to ignore the pre-flash, the camera is not going to be able to meter properly, so you will need to separately take that into account.
As an aside, you might consider investing in a wireless trigger. If you want to do it cheaply, Yongnuo triggers are very good for the price.