On Thu, Aug 20, 2015 at 11:16 PM, Peter Geoghegan <p...@heroku.com> wrote: > It could reduce seek time, which might be the dominant cost (but not > I/O as such).
No I didn't quite follow the argument to completion. Increasing the run size is a win if it reduces the number of passes. In the single-pass case it has to read all the data once, write it all out to tapes, then read it all back in again.So 3x the data. If it's still not sorted it needs to write it all back out yet again and read it all back in again. So 5x the data. If the tapes are larger it can avoid that 66% increase in total I/O. In large data sets it can need 3, 4, or maybe more passes through the data and saving one pass would be a smaller incremental difference. I haven't thought through the exponential growth carefully enough to tell if doubling the run size should decrease the number of passes linearly or by a constant number. But you're right that seems to be less and less a realistic scenario. Times when users are really processing data sets that large nowadays they'll just throw it into Hadoop or Biigquery or whatever to get the parallelism of many cpus. Or maybe Citus and the like. The main case where I expect people actually run into this is in building indexes, especially for larger data types (which come to think of it might be exactly where the comparison is expensive enough that quicksort's cache efficiency isn't helpful). But to do fair tests I would suggest you configure work_mem smaller (since running tests on multi-terabyte data sets is a pain) and sort some slower data types that don't fit in memory. Maybe arrays of text or json? -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers