Neither approach—with or without hoods—is necessarily "right" or "wrong". It's a matter of shooting style, personal preference, lens quality, tolerance for flare, and myriad other factors.
Personally, I carry all those lens hoods with me in my camera bag, but I can count the number of times I've used them on zero hands. Most modern lenses (or at least modern Canon lenses) have pretty good rejection of incident light (anti-flare coatings), and I rarely find myself shooting nearly into the sun anyway (and as others have mentioned, that's what your second hand is for). In the rare situations when I do, a little bit of flare is more tolerable than the inconvenience of having a lens that's harder to carry around in a fanny pack.
I routinely carry a 16–35L II, a 24–105L, and a 70–300L, in a one-on, two-off configuration. Typically, I use the stretch parts that normally hold water bottles. That way, the lenses are easily reachable for a quick change. Lens hoods would either make that configuration impossible (if I kept the hoods on) or would triple the amount of time it takes to switch lenses (if I took them off). So from my perspective, a lens hood has a big downside and not a lot of upside.
In contrast, I use lens caps religiously, both to avoid accidental fingerprints and because I change lenses pretty frequently. The caps are attached by a tether so I can flip them off of the front of the lens and take a picture within about a second if I see a shot I want to snag (assuming I don't decide to do a lens swap, which adds a few seconds).
I also use lens filters religiously to add a small amount of bump protection and environmental sealing. I'm careful to use only high-end filters to minimize the IQ impact, though I acknowledge that there is still some loss.
Basically, my advice would be "Try it and see." If you find that the hoods on your particular lenses mostly just get in the way, don't use them. If you find that they make a lot of shots salvageable, use them. If it is somewhere in between, flip a coin. Either way, the important thing is not the decision, but rather that you took the time to think about it and make the decision that best suits your particular shooting style.