Dear Imran,
I feel your idea is very interesting. Here are very elementary comments from my
point of view.
01.
I feel the patch looks large and it is bit difficult to be reviewed. Can you
separate it into several parts, e.g., producer and consumer parts?
02.
The file name seems to be "xlogpipeline.{h|c}" per others.
03.
Can you add tests? it's also helpful to understand the patch.
04.
Some Assert() are commented out but they won't be accepted. Can you added a
function like AmWalPipelineProducer() and call there?
05.
Producer is implemented as the bgworker process, but we have another option to
be as a background process. Which one is better and why?
06.
Can we enable the pipeline with the streaming replication? I've tried but it
did not work well - the producer exit and recovery finished without any promote
signals. My expectation was that producer waits till new record was arrived.
```
...
LOG: starting backup recovery with redo LSN 0/02000028, checkpoint LSN
0/02000080, on timeline ID 1
LOG: entering standby mode
LOG: [walpipeline] started.
LOG: redo starts at 0/02000028
LOG: completed backup recovery with redo LSN 0/02000028 and end LSN 0/02000128
LOG: consistent recovery state reached at 0/02000128
LOG: database system is ready to accept read-only connections
LOG: [walpipeline] consumer: received shutdown message from the producer
LOG: [walpipeline] producer: exiting: sent=4 received=4
LOG: WAL pipeline stopped
LOG: redo done at 0/02000128 system usage: CPU: user: 0.00 s, system: 0.02 s,
elapsed: 0.03 s
...
```
07.
As mentioned by Jakub, it is better to profile per process (startup and
producer).
08.
I suggest to benchmark on larger environment. Do let me know if difficult - I
can
try on my env instead.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
