2020-03-24 11:15:06 UTC - ikeda: Hi, I’m researching Pulsar because I’m interested in geo replication.
Is there any limitation to the number of regions(clusters)? The architecture which has 5 to 10 clusters is highly feasibile architecture? Please let me know if you have any ideas or use case. thanks ---- 2020-03-24 11:22:47 UTC - Ali Ahmed: there are no built in limitations on number of clusters. ---- 2020-03-24 12:19:46 UTC - ikeda: ahmed, I understood. thanks! ---- 2020-03-24 15:13:42 UTC - David Kjerrumgaard: @Abhilash Mandaliya Each connector has to have a unique name. Try running the second `create` command using a different value for the `--name` switch ---- 2020-03-24 20:00:38 UTC - Pradeesh: have not changed the username/password -its pulsar/pulsar ---- 2020-03-24 20:11:22 UTC - Yosi Attias: hey! I am checking out the pulsar functions, and have some questions. I want my functions to contain list of applicative subscription (not pulsar subscription) and filter by those subscriptions. • Is there any control API that I can expose from a function? like http api or something like this? if no, I thought about sending event in a application_subscriptions topic and have the function consume it too. • What’s the the life-cycle of the functions guaranteed? can I store cache their? or I need to use the state api? ---- 2020-03-24 20:13:01 UTC - Yosi Attias: Looks I can solve my both questions with functions state and use pulsar admin api to update the state - <https://pulsar.apache.org/docs/en/admin-api-functions/#put-state-associated-with-a-function> is that makes sense? ---- 2020-03-24 20:15:48 UTC - Yosi Attias: The only problem is that I can’t append to this state ---- 2020-03-24 20:41:52 UTC - David Kjerrumgaard: @Yosi Attias You can have your function consume from multiple topics and perform filtering logic on the messages it consumes from each. If you are looking for a way for downstream consumers to "self" register with a function, then you can implement a "control channel" mechanism as outlined in the <https://www.enterpriseintegrationpatterns.com/patterns/messaging/DynamicRouter.html|Dynamic Router Pattern> ---- 2020-03-24 20:44:24 UTC - Yosi Attias: @David Kjerrumgaard this is actually what I thought about. sending to topic named “subscriptions” operations like add/remove and have my function subscribe to to “events” topic and “susbscriptions” topic. I am just afraid about paralellism here, if I’ll process in parallel both topics I can have some issues here. ---- 2020-03-24 20:44:43 UTC - Yosi Attias: since the state is buffer and not list that I can append/remove atomicly ---- 2020-03-24 20:52:22 UTC - David Kjerrumgaard: Yes, parallelism makes it a bit more difficult. However, the `context.putState("key", buf);`does allow you to store a any serialized Java object, such as a List in a ByteBuffer. See <https://dzone.com/articles/serialization-using-bytebuffer|this> ---- 2020-03-24 20:53:13 UTC - David Kjerrumgaard: It would be much easier of the API used an Object rather than a ByteBuffer, but there are ways around that :smiley: ---- 2020-03-24 20:54:23 UTC - Yosi Attias: The ByteBuffer is not limited me, I will just serialize the list. I am not sure how tackle the parallelism here, do I need to use Java Locks here? or this is something we can to add `putState` the ability to safely modify a key? +1 : David Kjerrumgaard ---- 2020-03-24 20:56:12 UTC - David Kjerrumgaard: I believe that you would need to replicate the behavior of a concurrent list where each function instance "takes" some of the subscribers from the list to service, correct? ---- 2020-03-24 20:57:49 UTC - David Kjerrumgaard: Not sure if that is doable across processes. So you would have to pre-distribute the subscribers among the instances. ---- 2020-03-24 20:58:27 UTC - David Kjerrumgaard: Then you would also have to implement some sort of heartbeat to ensure that the subscribers get messages in the event their primary instance dies. ---- 2020-03-24 21:00:56 UTC - Yosi Attias: Sounds complicated.. If I will create function per service (consumer group), so each consumer-group will have it’s own state and I’ll run the function as thread. will it suffice? or it will limit in terms of scalability and I will need multiple processes here ---- 2020-03-24 21:04:36 UTC - Yosi Attias: I am starting to think, maybe storing this state in redis where I have lists their and transactions will be simpler ---- 2020-03-24 22:24:32 UTC - Gilles Barbier: regarding your parallelism issue, is it similar to <https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1584012254251600> ? ---- 2020-03-24 23:43:13 UTC - Ali Ahmed: <https://www.ovh.com/blog/announcing-kafka-on-pulsar-bring-native-kafka-protocol-support-to-apache-pulsar/> +1 : Sijie Guo, Karthik Ramasamy, Ali Ahmed, Yuvaraj Loganathan, Dennis Yung, Gilles Barbier tada : Sijie Guo, Addison Higham, Ali Ahmed, Yuvaraj Loganathan bouquet : Sree Vaddi, Ali Ahmed, Yuvaraj Loganathan ---- 2020-03-25 01:25:48 UTC - Joel Cressy: @Joel Cressy has joined the channel ---- 2020-03-25 04:18:50 UTC - Dennis Yung: Can we set the default number of partition for automatically created topics , like how kafka does? ---- 2020-03-25 04:33:11 UTC - Sijie Guo: <https://github.com/apache/pulsar/blob/master/conf/broker.conf#L98> ---- 2020-03-25 04:43:35 UTC - Dennis Yung: Thanks! This can only be set per cluster right? no namespace level override? ---- 2020-03-25 05:30:19 UTC - Sijie Guo: there is a pull request adding namespace override ---- 2020-03-25 05:30:30 UTC - Sijie Guo: it will come in 2.6.0 release. ---- 2020-03-25 07:05:30 UTC - Abhilash Mandaliya: It is working but both the connectors consume same messages. I want the messages to be distributed among instances. Looks like they both create different subscription ---- 2020-03-25 07:16:38 UTC - Abhilash Mandaliya: I think what I am looking for is --parallelism option. If I assign it a value same as my broker nodes, it distributes among them. Am I correct? ---- 2020-03-25 07:18:04 UTC - Dennis Yung: That's great! Looking forward ---- 2020-03-25 08:52:01 UTC - Roman Ananyev: @Roman Ananyev has joined the channel ---- 2020-03-25 09:09:06 UTC - Pierre-Yves Lebecq: @Sijie Guo Hello. This is an interesting thread. Is there a way to specify the schema type to use for a function? ----
