2018-12-05 10:12:07 UTC - Tobias Gustafsson: Hi all, new Pulsar user here. I have a question about bookkeeper replication. What I want to achieve is to have one copy of the ledger in three places but only require two bookies to ACK before returning and ACK to the client. Basically the same semantics that I would achieve with a Cassandra keyspace configured with a relication factor of 3 and writes in quorum (at least 2 acks). The first thing I tried was setting my conf to the following managedLedgerDefaultEnsembleSize: 3 managedLedgerDefaultWriteQuorum: 3 managedLedgerDefaultAckQuorum: 2 I believe this will give me availability problems though if I have 3 keepies in total one of the keepers goes missing. Am I right? So to use the above config I would have to increase the number of bookies to 6 and add some kind of rack awareness to distribute the data across DCs. Is that a correct interpretation? ---- 2018-12-05 10:14:37 UTC - Tobias Gustafsson: Or would you rather recommend me to change the settings to managedLedgerDefaultEnsembleSize: 3 managedLedgerDefaultWriteQuorum: 2 managedLedgerDefaultAckQuorum: 2 And rely on auto recovery for fault tolerance? ---- 2018-12-05 10:59:37 UTC - Tobias Gustafsson: Also, as a follow up on the above. How do I let a bookie know which rack it is in? I cannot seem to find any documentation covering this. Skimming the code I find some tests calling a management commando like this, set-bookie-rack, is that the way to go? ---- 2018-12-05 11:18:53 UTC - Karthik Palanivelu: But those are would be considered to be one cluster right, got it. Missed that. In this set up how would I configure synchronous replication- will rackAwarePlacement policy alone will help? Or need to set up some other properties? ---- 2018-12-05 11:23:35 UTC - Karthik Palanivelu: And more question is Cluster A is behind ELB-A who’s is part of cluster metadata and Cluster B is behind ELB- B. Will this be a problem that ELB-B is not part of cluster metadata? ---- 2018-12-05 13:09:52 UTC - jia zhai: @Tobias Gustafsson Besides the code, Here contains more info about the placement policy: <https://twitter.github.io/distributedlog/html/implementation/storage.html#ensemble-placement-policy> ---- 2018-12-05 13:12:38 UTC - jia zhai: ``` managedLedgerDefaultEnsembleSize: 3 managedLedgerDefaultWriteQuorum: 3 managedLedgerDefaultAckQuorum: 2 ``` This would keep 3 copies in 3 bookie nodes, and for each write, it will only wait the fastest 2 acks return. ---- 2018-12-05 13:41:02 UTC - Tobias Gustafsson: OK, thanks, and what would the expected behaviour be if one of the bookies disappeared? Is the cluster still supposed to be able to handle writes? ---- 2018-12-05 13:42:35 UTC - Ivan Kelly: right now, you need if ensemble size is 3, you need 3 nodes up ---- 2018-12-05 13:42:55 UTC - Ivan Kelly: so to tolerate failure, you should have 4 in total ---- 2018-12-05 13:44:30 UTC - Ivan Kelly: really, you only need 2 replicas though, so set ensemble, write quorum and ack quorum to 2 ---- 2018-12-05 13:44:37 UTC - Ivan Kelly: and still work with 3 nodes ---- 2018-12-05 13:45:03 UTC - Ivan Kelly: the only tradeoff there will be a latency hit if one of the two fails ---- 2018-12-05 13:46:14 UTC - Ivan Kelly: we really should allow to have fewer than ensemble nodes available, as long as ack quorum is satisfied ---- 2018-12-05 13:47:15 UTC - Ivan Kelly: actually, we do ---- 2018-12-05 13:47:21 UTC - Ivan Kelly: was looking at wrong code ---- 2018-12-05 13:47:46 UTC - Ivan Kelly: you need to have delayEnsembleChange enabled. I'm not sure that's available through managed ledger though ---- 2018-12-05 13:58:38 UTC - Tobias Gustafsson: OK, so basically rely on the recovery to make sure that two replicas exist at all times, is that your suggestion? ---- 2018-12-05 14:00:24 UTC - Tobias Gustafsson: If you would like to run bookies in three DCs I guess you would want six of them if aiming for an ensemble of three and make sure that ledgers are evenly distributed over the DCs using rack-awareness? ---- 2018-12-05 14:01:40 UTC - Ivan Kelly: yes, recovery should keep you at two replicas ---- 2018-12-05 14:03:23 UTC - Ivan Kelly: I don't have much experience running a cluster across DCs, perhaps @Matteo Merli can help ---- 2018-12-05 14:08:29 UTC - Tobias Gustafsson: OK, thanks! ---- 2018-12-05 14:10:08 UTC - jia zhai: :+1: ---- 2018-12-05 15:55:50 UTC - Bogdan BUNECI: Hi All ! I think I found a small bug in pulsar sql. ---- 2018-12-05 15:56:33 UTC - Bogdan BUNECI: The sql table is always a record behind until the next record arrives. ---- 2018-12-05 15:58:23 UTC - Bogdan BUNECI: I’m running pulsar-2.2.0 in standalone mode. ---- 2018-12-05 16:21:38 UTC - Matteo Merli: @Bogdan BUNECI the sql worker is tailing the log, though it has no visibility on the last committed entry (or entries, when pipelining more entries) on the log. For that reason, the last few entries will not be visible. If you have more entries being added all the time that won’t be an issue.
It’s not difficult to add the support to read up to the last entry, we just need to follow the same BookKeeper protocol to establish the last entry ---- 2018-12-05 16:44:55 UTC - Bogdan BUNECI: Thanks ! Is not a big issue, I guess. I just wanted to make sure you know about this. ---- 2018-12-05 16:47:40 UTC - Ivan Kelly: @Matteo Merli you mean explicitLAC? ---- 2018-12-05 16:53:29 UTC - Bogdan BUNECI: No data is lost. The only issue is that the last/few record/s are not visible until new records are arriving. ---- 2018-12-05 17:38:42 UTC - Matteo Merli: Yes, since the ledgers are still open, we only only read up to the LastAddConfirmed (basically, it’s a safepoint where the client has received ack from the required number of replicas). @Ivan Kelly I think we should add in BK client the method to do something like `ReadHandle.getLastCommittedEntry()` that follows the same logic as recovery, just in read-only mode ---- 2018-12-05 17:42:50 UTC - Sijie Guo: @Matteo Merli explicitLAC is better to be used in this case. Following the recovery logic but doesn’t do the recovery can potentially volatile consistency issue ---- 2018-12-05 17:43:28 UTC - Sijie Guo: I mean it is trickier to get things right ---- 2018-12-05 17:43:48 UTC - Sijie Guo: ExplicitLAC is safe ---- 2018-12-05 18:13:31 UTC - Matteo Merli: never used explicit lac… need to take a look at that :slightly_smiling_face: ---- 2018-12-05 19:32:01 UTC - Addison Bair: @Addison Bair has joined the channel ---- 2018-12-05 19:49:39 UTC - Ivan Kelly: @Matteo Merli getLastCommittedEntry could break TOAB. the recovery logic only gives you a candidate lastEntryId. this candidate could change, depending on which bookies respond ---- 2018-12-05 19:52:07 UTC - Matteo Merli: Yes, though we could do N-reads after the LAC. with explicit lac, it would be the writer to periodically flush that info, right? ---- 2018-12-05 19:59:34 UTC - Ivan Kelly: that's different to the recovery logic. and you could still end up reading entries that didn't make it into the final ledger ---- 2018-12-05 20:00:17 UTC - Matteo Merli: Yes ---- 2018-12-05 22:45:49 UTC - Shalin: I have a usecase where I am using pulsar functions with a custom `SerDe`. The `topic` contains messages which can belong to one of two different schemas. I was planning on storing the schema type in the message `properties` and using that to Serialize or deserialize a message. How do I access the message `properties` in my custom `SerDe` class? ---- 2018-12-05 23:15:41 UTC - Sanjeev Kulkarni: I don’t see any way of doing that using the current SerDe interface which only takes in message content. What I would suggest would be to move the deserialization into the function process method where you can access the current message using the Context interface ---- 2018-12-05 23:28:18 UTC - Shalin: Thanks for clarifying that @Sanjeev Kulkarni. It makes sense to only pass in the bytes to the `SerDer` interface. Couple of questions around that - Are messages with multiple schemas for a single topic not supported/encouraged in `pulsar`? - Or it is supported, but handling it using message metadata is not the right way to do in `pulsar`? ---- 2018-12-05 23:51:56 UTC - Sanjeev Kulkarni: Typically data in a particular topic represents some notional entity that is described by its schema. This schema may evolve over time, however at any point of time there is only one schema version. Thus using a topic with two completely different types of schemas is sort of an anti-pattern ---- 2018-12-06 03:59:32 UTC - drkna: @drkna has joined the channel ----
