Re: Issue: Data-inconsistency during segment push

2020-04-06 Thread Mayank Shrivastava
Yes, the high level idea works. We will need to iron out the details a bit. Few examples: 1. For refresh use cases with large data size though, even the servers may non-trivial amount of time to download segments. We need to wait until all new version is ONLINE in external view before

Re: Issue: Data-inconsistency during segment push

2020-04-06 Thread kishore g
One suggestion - high level Put the time boundary in ZK (may be in /propertystore/table/routingInfo). This can be updated via one of the following - The upload job can set this via controller API after everything is pushed - Controller can do this periodically - Anything else Brokers watch for