The new Zeiss Otus 55mm f/1.4 is nearly $4000.
With faster lenses from Canon (50mm f/1.2L) with autofocus, why is the zeiss lens more than twice that of the Canon counterpart?
There is a clear difference in intent and design philosophy. The Canon 50 f/1.2L is a bokeh machine, offering not only shallower depth of field (due to the ultra wide f/1.2 aperture) but also a smoother background blur on account of the decision to leave a certain amount of spherical aberration in the design. It's a lens with character and a distinct look, which is at least partially due to its imperfections. As a result, it's not the sharpest lens in the Canon lineup, by a long shot.
The Zeiss Otus 55mm f/1.4, on the other hand, is an all-out no-compromise lens*. Speed, very high sharpness across the frame at every aperture, flatness of field, absence of coma, near absence of longitudinal chromatic aberration (bokeh fringing) were all key design criteria.
Its construction is very different from the Canon 50L — which, like almost every 50 ever made for the 35mm format, is based on a classic double Gauss. The Zeiss Otus is a Distagon, which is the Zeiss term for a retrofocus lens, a technique employed in wide angle lenses with a reversed telephoto group at the back to move the rear element far enough from the sensor to clear the DSLR mirror. Clearance is not an issue with a longer focal length like 55mm, however this design was chosen despite its increased complexity because it allowed better control of certain aberrations.
It's also no-compromise mechanically, with actual bearings for the moving parts. Ultimately to move past a certain level of performance you need to eliminate or at least severely limit any variation in the position of lens elements, otherwise all the effort that went into the design and the precision grinding of each lens surface is wasted. This all comes at a price.
The Canon lens, although it is still expensive due to the wide aperture, use of fluorite, low volumes, etc., is a lens of trade-offs; the Otus is a level above all round. The Canon is the latest evolution of a long series of fast 50s; the Otus is a new design that looks forward to ever increasing demands on resolution from 35mm sensors — it's really competing against medium format, which makes the price tag actually quite reasonable.
* unless you count weight / cost