sodonnel commented on PR #12061:
URL: https://github.com/apache/ozone/pull/12061#issuecomment-6915837722

   Are these new "chained pipelines" intended to be long lived, like the Ratis 
ones, or short lived like the EC ones?
   
   From the Jira:
   
   ```
   What it reuses. A CHAINED pipeline is created like a RATIS/THREE pipeline:
   datanode choice through PipelinePlacementPolicy;
   a leader from LeaderChoosePolicy;
   CreatePipelineCommand(pipelineID, CHAINED, THREE, dns, leader) sent to each 
datanode;
   ClosePipelineCommand on close.
   ```
   
   It sounds like the DNs know about the pipelines etc, like with Ratis 
pipelines?
   
   I am concerned that long lived Ratis pipelines have turned out to be a 
really really bad idea in Ozone and they are at the root of many hard to deal 
with problems. We should not build on that IMO.
   
   The EC mode forms a pipeline for a single container and then forms a new one 
for the next one, much more like HDFS. It allows for much more dynamic space 
balance and load sharing.
   
   I checked the chained design doc, which doesn't really answer the above 
questions.
   
   Also have we considered a model like EC, where the client concurrently 
writes to all 3 DNs directly rather than chaining them? This is how EC works 
and it lets the client coordinate consensus and it feels like that would be 
faster than a forwarding approach as you have added latency as the DNs have to 
forward to each and ACK back all the way to the client.
   
   Is it possible to adapt the EC code to write replicated data to see how it 
performs before implementing this totally new way?
   
   Also will this new approach open the door for 2 or 4 way replicated 
containers, which are not possible with Ratis right now?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to