This answer is in addition to Michael C's answer.
The other part of the story is that lenses are sometimes priced based on what the market will bear. Fuji lenses are expensive compared to some Canon lenses, yes, but they are rather cheap compared to others.
For instance look at pairs of standard lenses: a cheaper slightly slower one and a faster generally more expensive one. For Fuji, by 'standard' I mean 'about 35mm' while for full-frame by 'standard' I mean 'about 50mm'. Prices rounded to nearest £10, all from the same shop.
- 35mm f/2 £370
- 35mm f/1.4 £500
- Zeiss M mount
- 50mm f/2 Planar) £650
- 50mm f/1.5 (C-Sonnar) £920
- Leica M mount
- 50mm f/2.4 (Summarit-M) £1,330
- 50mm f/1.4 (Summilux-M) £3,100
And if you want to be really silly: Leica 50 f/2 Summicron ... £6,000 in black or £6,350 in silver (the silver finish on the lens costs as much as the whole 35mm f/2 Fuji lens). And that's not the most expensive Leica 50mm lens, even without considering absurdly fast ones.
Now of course, you could argue that these lenses are hugely better than the Fuji equivalents. Well, I own some of them, and the ones I own are not. I have both of the Zeiss lenses I mention above and the C-Sonnar, which I love dearly, is frankly not objectively a great lens: it famously has focus shift, and many of them, including mine, are really annoyingly rattly. It's neither better made nor objectively optically better than the Fujis. I don't own the Leica lenses I mention above (I own a much older Summicron), but I seriously doubt they are either that much optically better or that much mechanically better than the Fujis, especially than the 35 f/1.4 which I understand to be a really, really good lens.
None of the M-mount lenses have autofocus which presumably costs more than nothing to provide in a lens.
This price difference is not covered by R & D costs. I don't know the product history of the Leica lenses mentioned, but I bought both of the Zeiss lenses I mention above before there were X-mount Fujis, and they're both variants on fairly (in fact very) old designs.
It might be that the difference is accounted for by the relatively tiny market for M-mount lenses. I don't personally think this is true, but I don't have evidence.
What I think is reasonably clear however is that people who buy M-mount cameras expect to pay a lot for their lenses. Poor (relatively) people who buy M-mount cameras, like me, spend more on their lenses than they would on objectively better Fuji lenses. Rich people spend much, much more.