Are you returning enough data to cause the system to go into paging - check the system monitoring tools, not sure what you are running on, to see if there is a large amount of VM activity involving paging to disk while this is running. That would really kill performance. The entire data returned by the query will be turned into a Rows object so it all has to live in memory at the same time.
Another odd clue is the second time takes longer than the first time. Normally it should take less time because the caching in the system such as in the database engine is warmed up. If VM activity is high, caching will be defeated because those assumed high speed memory caches saved in case the are needed again in a short time might have been rolled out onto the disk. Ron