The problem is mostly solved now, objdump saved me :) It turns out ldc passes 32-bit parameters in EDI, ESI and 64-bit parameters in RDI, RSI. The result is in EAX or RAX, depending on the size. The segfaults I encountered are caused by using ESP as stackpointer, but in 64 bit mode this should of course have been RSP.
I think this solves all my practical problems for now, but it would be nice to know the detailed calling conventions, or are they maybe generated on-the-fly by llvm to get the best calling convention for each single function? I also tried to enforce the documented D calling convention with extern(D), and also tried extern(C), but that didn't change anything.