Hi Marcin, I can't advise as to profiling to find out what really bogs down agenda building.
I found that log messages do bog it down. I have a lot of recurring tasks, which accumulate log entries for every closing (which in fact means rescheduling to the next day). Every two to three months I prune my org files of those log entries. This significantly speeds up agenda building. HTH, Julius
