2018-12-17 09:27:17 UTC - Lena Gregor: I’m really sorry to bother you again, but it doesn’t have to be the “one and only perfect answer”. Just a few classes that you would personally recommend to a new developer to look at first would already help me a lot! So if somebody of you could offer a few minutes of their time, that would be great. (You can also send your answer to <mailto:[email protected]|[email protected]> if you’re afraid that your answer could be judged by the others). So thank you very much in advance. ---- 2018-12-17 09:30:51 UTC - Ali Ahmed: @Lena Gregor there is no simple answer to your question it depends what areas you are interested in (storage , compute, co-ordination etc) and what your level of understanding of Pulsar’s architecture is ---- 2018-12-17 09:30:59 UTC - Ali Ahmed: How much do you know about Pulsar ? ---- 2018-12-17 09:48:46 UTC - Lena Gregor: @Ali Ahmed Close to nothing, to be honest. I was just looking for projects that are big enough so my computation would make sense/ could be useful. But if it’s hard to name classes because the parts are so independent and you don’t really need a “general overlook” but it’s better for a new developer to start working in their specific region, than that’s an answer too. We could also do it the other way around: I could compute the most central classes and you could tell me if you would recommend some of them. ---- 2018-12-17 09:59:10 UTC - Ali Ahmed: You should read all blogs / video talks relevant to pulsar’s architecture, then you ask for areas of interest and we can guide you through the code ---- 2018-12-17 10:28:13 UTC - linxin: Why Pulsar schema is topic dimension, unlike Kafka, each message gets a schema id? What happens if the schema makes incompatible changes (such as delete a Topic schema and then re-creating one) and then register a new consumer with new schema trying to consume the old message with old schema? 1. Topic#setSchema(SchemaA); 2. send a messageA under SchemaA; 3. delete SchemaA and Topic#setSchema(SchemaB); 4. Consumer#subscribeTopicWith(SchemaB); How the consumer handles the messageA? Is there anyone can solve my doubts？ ---- 2018-12-17 10:58:24 UTC - Victor Li: @Victor Li has joined the channel ---- 2018-12-17 11:02:23 UTC - Victor Li: Hello Everyone! :slightly_smiling_face: I am a newbie at Pulsar. We are using NiFi + Kafka+ Storm(HDF by hortonworks) for our business streaming process. ---- 2018-12-17 11:03:03 UTC - Victor Li: So is there any nifi connector for pulsar? ---- 2018-12-17 11:03:43 UTC - Victor Li: we don't want to use storm any more ---- 2018-12-17 11:07:31 UTC - Ali Ahmed: @Victor Li yes there is ---- 2018-12-17 11:07:44 UTC - Ali Ahmed: @David Kjerrumgaard can help you with it ---- 2018-12-17 11:52:03 UTC - Guangzhong Yao: @linxin Currently, pulsar may not support schema evolution, so your steps will cause exceptions. I think you can ask @Sijie Guo, @Matteo Merli, @Ivan Kelly for the pulsar schema design considerations. ---- 2018-12-17 11:53:28 UTC - linxin: @Guangzhong Yao thanks. ---- 2018-12-17 11:56:33 UTC - linxin: Maybe it’s jet lag. They probably haven’t seen the message yet. ---- 2018-12-17 12:53:56 UTC - Sijie Guo: the schema is stored in topic level. each message has a field for schema version : <https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L100>
however as what @Guangzhong Yao pointed out, we are not fully leveraging the schema version yet. there are still some work items for fully leveraging the schema version ---- 2018-12-17 12:59:14 UTC - linxin: so at current stage, Pulsar can not handle the situation I proposed? ---- 2018-12-17 13:00:24 UTC - Sijie Guo: correct. that’s the missing piece at this moment ---- 2018-12-17 13:00:40 UTC - Sijie Guo: feel free to contribute :slightly_smiling_face: since we already have the schema version :slightly_smiling_face: ---- 2018-12-17 13:06:17 UTC - linxin: :ok_hand: ---- 2018-12-17 13:42:30 UTC - Sijie Guo: <https://github.com/apache/nifi/pull/3178> ---- 2018-12-17 13:44:28 UTC - Sijie Guo: @David Kjerrumgaard I think it might be making sense to port this connector back to Pulsar repo. if nifi is moving slowly at merging this. ---- 2018-12-17 14:35:25 UTC - Ivan Kelly: @linxin generally you don't delete the schema before adding a new one. when you add a new one you can set a compatibility level (backward, forward, full) ---- 2018-12-17 16:57:20 UTC - David Kjerrumgaard: @Sijie Guo Agreed ---- 2018-12-17 16:57:58 UTC - David Kjerrumgaard: They are starting to really irk me with their lack of effort on getting this code merged. I have been at it since the 1.6 release back in March ---- 2018-12-17 17:02:03 UTC - David Kjerrumgaard: Given the message retention periods afforded by our tiered-storage, this may become an issue on longer-lived topics. ---- 2018-12-17 17:12:17 UTC - David Kjerrumgaard: @Victor Li What version of NiFi are you currently using? ---- 2018-12-18 00:58:25 UTC - Balamurugan Muthusamy: @Balamurugan Muthusamy has joined the channel ---- 2018-12-18 01:44:36 UTC - 高天赐: @高天赐 has joined the channel ---- 2018-12-18 05:04:25 UTC - Victor Li: @David Kjerrumgaard @Ali Ahmed Thanks. our NiFi Cluster using 1.5. Edge side using 1.7 ---- 2018-12-18 05:15:21 UTC - Victor Li: @David Kjerrumgaard thanks for your information about nifi pulsar processor. ---- 2018-12-18 05:20:02 UTC - Roger Yates: @Roger Yates has joined the channel ----
