I am looking at 8x11, 11x8.5, and 12x12 inch photobooks (other sizes are available but are mostly obtained by rescaling these). I want to include both portrait and landscape photos. Which overall dimension of the photobook will be the best?
The format you choose is ultimately down to your personal preference. However, there are basic, and fairly obvious, design principles you can follow. If the vast majority of your images are "landscape", then choose a format that reflects that. The same is true if the majority of your images are "portrait".
You then have to decide how to best fit in the non-conforming images: for example, you could place two portrait images on a single page of a landscape book. Or, if you have text, you can place the text in the spaces around the non-conforming images.
My favourite solution is to go for a square format, which accommodates both landscape and portrait images equally well and leaves plenty of space for captions and other text. But, obviously, a square format only really makes sense if you have a roughly equal mix of landscape and portrait images. And, as printers like using standard paper sizes, then square formats are often more expensive, as they are made by printing on a standard paper size, which is then trimmed square, which increases wastage.