If you need to see a preview of the image in square format to aid in composition using the rear LCD, the easiest thing to do is what you suggest near the end of the question: mask the rear LCD screen into a square, taking care to make the width of the mask equal to the image height of preview images displayed on the LCD, rather than the height if the LCD screen itself.
For the viewfinder, selecting 'On' as the option under custom setting 'd7:Viewfinder Grid Display' enables a set of grid line overlays to be displayed. The two vertical lines on either side of the frame appear to be about 5% taller than the width between each one. This is as measured from a drawing of the viewfinder on page 10 of the Nikon D750 User Manual
I've added yellow lines to show where an actual square crop centered on the exact middle of the frame would be:
That's going to get you very close, though you would need to crop the image manually later on.
Is there a way to install D850 firmware on the D750 maybe?
Nope. If you could manage to load it, it would most likely brick the camera permanently. Even if it didn't permanently damage the camera, it would not work correctly. Different sensor pixel dimensions would scramble the photos. Other hardware differences, both internal and external, would make most things things non-functional.