2020-05-24 12:05:04 UTC - Guilaaume: @Guilaaume has joined the channel ---- 2020-05-24 12:19:20 UTC - Julius S: Hey Matej, I’ve done the math vs. MSK on AWS and Pulsar comes out cheaper, even with top level managed support added. I can DM you. We currently run Pulsar on EKS. ---- 2020-05-24 12:44:50 UTC - Matej Šaravanja: @Julius S yes, please dm me with all the details you have :) ---- 2020-05-24 21:16:17 UTC - Andy Teijelo: Hello, guys! First of all, kudos to everyone involved in Pulsar. It's an amazing piece of software with a huge potential. On to my question. We've been playing with Pulsar for a few days (we come from Kafka). We have a Debezium setup for a multitenant environment where there are about 300 tenants and each tenant's database has around 300 tables. On the default Debezium setup, that translates to 90000 topics. You can imagine Kafka was not happy about that. But even with Pulsar's architecture, where that number of topics is not insane, having every topic under the same tenant/namespace still feels wrong. A seemingly obvious solution is to run 300 Debezium connectors, but for the source database (e.g. MySQL), that's 300 slave servers following the binary log, which I don't think is a good idea (I may be wrong on that). Would there be a way of having a single Debezium connector on Pulsar create topics on different namespaces, so that each db schema becomes its own Pulsar namespace and each table is then a topic? ---- 2020-05-25 02:49:59 UTC - Luke Stephenson: Hello. I've just discovered that it is not possible to have multiple consumers reading from a compacted topic with a shared subscription. <https://github.com/apache/pulsar/blob/v2.5.2/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PulsarClientImpl.java#L309-L315>. My planned use case for pulsar included large compacted topics which I had wanted to have multiple consumers reading from. Is there a reason to not support this? If not, and it's just a case of it not being implemented yet, I'll raise a feature request. ---- 2020-05-25 04:40:35 UTC - Luke Stephenson: Raised the feature request <https://github.com/apache/pulsar/issues/7028> ---- 2020-05-25 05:47:06 UTC - Adelina Brask: Hej Guys, do you have an example of how to implement and handle DLQ on a Pulsar configuration that uses (netty) sources and (Elastic) sinks? I don't know JAVA and the only guide I found is using a Java client. Do you have a guide maybe, or even better, a setting you can set on the consumer/broker side for turning DLQ on? ---- 2020-05-25 07:28:21 UTC - Sijie Guo: Currently source and sink don’t support DLQ yet. ---- 2020-05-25 07:28:32 UTC - Sijie Guo: only functions support DLQ. ---- 2020-05-25 07:28:50 UTC - Sijie Guo: Can you create a Github issue for us? So we can prioritize it. ---- 2020-05-25 07:30:49 UTC - Sijie Guo: > Would there be a way of having a single Debezium connector on Pulsar create topics on different namespaces, so that each db schema becomes its own Pulsar namespace and each table is then a topic? Yes. With code changes to the debezium connectors, it should be fairly easy to achieve that. ---- 2020-05-25 07:43:22 UTC - Sijie Guo: Typically a compacted topic is used for representing a “state” for a topic. The most common case is to read the compacted topic in sequence. Can we understand a bit more about your requirements of having multiple consumers reading from one compacted topic? ---- 2020-05-25 07:57:39 UTC - Adelina Brask: <https://github.com/apache/pulsar/issues/7032> ---- 2020-05-25 07:58:51 UTC - Adelina Brask: Hope it will be prioritized - as I can't really use Pulsar with ElastisSearch sink without a DLQ, as we are loosing logs and overcharging the brokers. ---- 2020-05-25 08:14:59 UTC - Adelina Brask: @Sijie Guo Could you advice on an alternative solution? Maybe there is and I can't see it? Or is waiting for 7032 the only solution? ---- 2020-05-25 08:54:56 UTC - Sijie Guo: I don’t think there is a simple workaround at this moment. /cc @Penghui Li to see if it is easy to add a flag for sink. ---- 2020-05-25 08:55:29 UTC - Penghui Li: ok ---- 2020-05-25 09:03:04 UTC - Frans Guelinckx: I have a very similar requirement on my project. Good to know here is that the debezium team is working towards decoupling debezium from kafka in order to improve integration with other platforms like pulsar, kinesis,… They’re currently in the beta stage with this, but they’re making good progress. More information can be found <https://debezium.io/blog/2020/05/19/debezium-1-2-beta2-released/|here> and <https://debezium.io/documentation/reference/operations/debezium-server.html|here>. ----
