I am using studio flashes and I'm firing them using the built-in flash of my Canon 77D on custom wireless mode. The flash seems to work fine, however, there is no sign of flash lighting in the final photo. I have tried playing with the various shutter speeds as well. How do I fix this?
The built-in flashes of modern cameras (as well as most hotshoe strobes) use a "pre-flash" to measure the scene. A small burst of light happens right before (or right as) the shutter opens, and that is evaluated to calculate the power for the full flash.
This works pretty well, but it also means that traditional secondary flashes that trigger when they notice a burst of light are fooled and can fire too soon. Some have a mode where they ignore the first burst for exactly this reason — check if your studio light does.
Don’t use Wireless mode on your 77D. It will cause your flash to emit additional light pulses to communicate with the Canon wireless slaves flashes. These extra light pulses will cause your studio strobes to fire too early.
Use Manual flash mode on your 77D to avoid the ETTL preflash which will also cause your studio strobes to fire too early.
Even a low manual power setting like 1/4 or 1/8 should be enough to trigger the studio strobes.
Wireless mode in a Canon pop-up flash is for the Canon "smart" optical wireless system, where settings from the camera to the flash that are normally done over the contacts/pins of the camera hotshoe/flash foot, are done by a series of light pulses from the flash ("pre-flashes") before the shot is taken. Think of it like sending messages via Morse code by with light blips. But for it to work, the receiving flash has to not only receive the signals via a sensor, but also know how to interpret the blip code.
Most studio strobes can't. They use a "dumb" optical slave: one that simple fires the strobe whenever its sensor "sees" a flash burst. Most of these slaves will typically fire on the first flash burst they see. But, that means that they'll be tripped by the command communication of Canon's "wireless" mode with its pop-up flashes.
So, while it doesn't make a lot of logical sense terminology-wise, you have to take the Canon pop-up flash out of wireless mode. And further, you have to set the pop-up flash to be in M / Manual mode (not TTL), because TTL uses a preflash for metering, and may also set off your strobes too early. Red-eye reduction is a typical third source of preflashes, but probably isn't an issue for you.