Formatting is best, unless you're pressed for time.
Writing to a previously used location of flash memory requires that it be erased first. By formatting, you are pre-erasing the entire space, ready for immediate writing. Formatting also resets any fragmentation. If you rarely or never format, fragmentation can also develop, which can effect performance:
1) read speed (nice to have): reviewing pictures, and transferring them off the device.
Unlike mechanical hard drives, fragmentation does not affect the read speed of flash memory.
2) write speed (critical): taking pictures.
Even though there are no mechanical latencies for flash memory, "write amplification", (page-level rewriting), can eventually cause significant slow downs: https://en.wikipedia.org/wiki/Write_amplification
This happens when existing files need to be moved while trying to store a new file. This involves erasing and rewriting, and that has to be done in large chunks. Imagine swapping houses with your neighbor that otherwise has the same floor-plan as you.
Note that if you keep the picture resolution setting constant, picture files are all nearly the same size, which goes a long way to minimizing fragmentation in the first place. However, the effects of fragmentation can eventually result in noticeable slow downs. Here are some interesting test results for SSD hard drives: http://www.lagom.nl/misc/flash_fragmentation.html