On Sat, Oct 31, 2020 at 12:09:32AM +0200, Heikki Linnakangas wrote:
On 30/10/2020 22:56, Tomas Vondra wrote:
I agree this design looks simpler. I'm a bit worried about serializing
the parsing like this, though. It's true the current approach (where the
first phase of parsing happens in the leader) has a similar issue, but I
think it would be easier to improve that in that design.
My plan was to parallelize the parsing roughly like this:
1) split the input buffer into smaller chunks
2) let workers scan the buffers and record positions of interesting
characters (delimiters, quotes, ...) and pass it back to the leader
3) use the information to actually parse the input data (we only need to
look at the interesting characters, skipping large parts of data)
4) pass the parsed chunks to workers, just like in the current patch
But maybe something like that would be possible even with the approach
you propose - we could have a special parse phase for processing each
buffer, where any worker could look for the special characters, record
the positions in a bitmap next to the buffer. So the whole sequence of
states would look something like this:
EMPTY
FILLED
PARSED
READY
PROCESSING
I think it's even simpler than that. You don't need to communicate the
"interesting positions" between processes, if the same worker takes
care of the chunk through all states from FILLED to DONE.
You can build the bitmap of interesting positions immediately in
FILLED state, independently of all previous blocks. Once you've built
the bitmap, you need to wait for the information on where the first
line starts, but presumably finding the interesting positions is the
expensive part.
I don't think it's that simple. For example, the previous block may
contain a very long value (say, 1MB), so a bunch of blocks have to be
processed by the same worker. That probably makes the state transitions
a bit, and it also means the bitmap would need to be passed to the
worker that actually processes the block. Or we might just ignore this,
on the grounds that it's not a very common situation.
Of course, the question is whether parsing really is sufficiently
expensive for this to be worth it.
Yeah, I don't think it's worth it. Splitting the lines is pretty fast,
I think we have many years to come before that becomes a bottleneck.
But if it turns out I'm wrong and we need to implement that, the path
is pretty straightforward.
OK. I agree the parsing is relatively cheap, and I don't recall seeing
CSV parsing as a bottleneck in production. I suspect that's might be
simply because we're hitting other bottlenecks first, though.
regard
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
