Hi Sébastien,
put your .emacs on diet :) More seriously, I would start by checking org-agenda-to-appt documentation. It should be on your system. If it is not, check online. That's what really slows down your config. Then, when this is done, I would try to use only those hooks that are really necessary. My 2 cts of course, -- Bastien