Disclaimer: this is not exactly an answer to the question, as it doesn't rely on calculation, but experimentation.
I used a do-it-yourself approach for checking lens hoods (and vignetting, to some extent), back in the analog times.
From cardboard, I cut out a 24*36mm rectangle and glued together a box to hold that cutout at the correct distance from the lens mounting flange.
When looking from the back through the cutout into the lens, no part of the lens hood should be visible. If it's visible, estimate the percentage of area obscured by the hood, with 50% covered resulting in 1.0EV vignetting.
Regarding lens vignetting: when looking along the center line, the shape of the lit-up area always looks circular, and that's the reference. Lenses with a vignetting problem can typically be identified when looking from a corner angle: the shape no longer is circular, but looks like the intersection of two overlapping circles: like an ovoid with two sharp edges.
If you aren't afraid of spending some time with cardboard, glue and sharp knives: build a cardboard box with depth equal to the flange focal distance of your camera system, with a front cut out to match the camera/lens mount, and a back coutout resembling your sensor size.
2020's high-tech-version would be to design and print it in 3D.