Ã…pen informasjon / Public information Hi,
Sorry for being offline on this matter.... Claus, please do not get political on this. 😊 Confluent Kafka is also open source - at least the stuff we are talking about here. And I am pretty sure Confluent is one of, if not the, main contributor to Apache Kafka. Enterprise open source is the "new way", and you cannot blame others for doing the same as Red Hat has been doing successfully for many years! Camel should, in my option, provide the extensibility mechanisms to fulfill Camel users requirements - even if you consider them to be "proprietary".... And no, I have absolutely no personal relationship to Confluent. I was also a bit surprised when our Kafka Ops tells me I have to use a "schema registry" to produce Kafka messages. But I have got used to it, and I even like the idea of having some API/schema on the Kafka data streams.... A Maven plugin would not solve my original issues - even if it was in place when I needed to configure some Confluent specific Kafka properties. Or did I misunderstood something? I think we really need a way for users to set (additional) Kafka consumer/producer properties that are not known to Camel. And other "complex" Camel components do have that. That is my main point of view. Do you want me to create an issue for this? Erik -----Opprinnelig melding----- Fra: Claus Ibsen <claus.ib...@gmail.com> Sendt: søndag 13. oktober 2019 18:19 Til: users@camel.apache.org Emne: [SUSPECT URL !]Re: Configuring custom Kafka producer/consumer properties on Camel Kafka component/endpoint? Hi Omar Yeah the maven plugin idea is really good. But maybe we need both. The maven plugin will only get you the options of the kafka version that was used for that release. With having a way for end users to set extra options then they could work against newer Kafka versions without Camel have to upgrade. And maybe also to proprietary options on Confluent. On Sun, Oct 13, 2019 at 2:31 PM Omar Al-Safi <omars...@gmail.com> wrote: > > Hi Claus, > > I also thought about this, however since Kafka producers/consumers options > increase and some of them may gets deprecated, I was thinking we can have a > maven plugin that generate these configurations for us without the need to > do it manually, it will generate using the configDef (e.g: > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fblob%2Ftrunk%2Fclients%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkafka%2Fclients%2Fconsumer%2FConsumerConfig.java%23L555&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788184830&sdata=S%2BJyTAx8u7V0jow9oCELojh9af5dsgDtxYjRzRz89Lg%3D&reserved=0). > I did this for DBZ as I built a maven plugin that scrap all these config > and generate the camel configurations, you can take a look at my progress > here: > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fomarsmak%2Fcamel%2Fblob%2FCAMEL-14009%2Fcomponents%2Fcamel-debezium%2Fcamel-debezium-maven-plugin%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fcamel%2Fmaven%2FTestMojo.java%23L24&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788184830&sdata=hTRBEetZ7zBt%2BoS9fSfbQV94G0KRIudY0WlkDeo50Fs%3D&reserved=0. > Please take a look and share your thoughts about it. > > Regards, > Omar > > On Sun, 13 Oct 2019 at 14:12, Claus Ibsen <claus.ib...@gmail.com> wrote: > > > Hi > > > > Yeah this has come up a few times that Kafka is adding new options, or > > more particularly Confluent adds propriety options that they dont give > > back to the community. > > > > Can you create a JIRA ticket. > > > > Then we can look at adding a way to add custom options. You are of > > course welcome to try to implement this. We love contributions. > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fcamel%2Fblob%2Fmaster%2FCONTRIBUTING.md&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=jPZhXs6oPXeBZKVQn54EQhA2kTqGLQHm5CqUmCDaXo8%3D&reserved=0 > > > > On Tue, Oct 8, 2019 at 9:57 AM Erik Boye <erik.b...@statnett.no> wrote: > > > > > > Hi, > > > > > > Is there any way to configure custom Kafka producer/consumer properties > > on the Camel Kafka component/endpoint (without "breaking in")? > > > I specifically want to configure the Confluent Kafka property with key > > 'value.subject.name.strategy': > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.confluent.io%2Fcurrent%2Fschema-registry%2Fserializer-formatter.html&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=XyX9RJL%2Fp2CMp%2Fg0enZ7dTrRQ5oalOIcLuy6nWxr%2BcA%3D&reserved=0 > > > > > > Other Camel components (i.e Camel ActiveMQ; destination.consumer) are > > made extensible by providing users a way to set "additional properties". > > Why does not Camel Kafka provide that? Until schemaRegistryURL was added as > > a configurable property, we had to make our own Camel Kafka Component to > > make it work ("breaking in"). There should definitely be a way to set > > additional/custom properties on Kafka producers/consumers. > > > > > > Just have a look at the Camel Kafka documentation and do a text search > > for Confluent: > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcamel.apache.org%2Fcomponents%2Flatest%2Fkafka-component.html&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=YTimYbe%2FcTksQUj3wmG4pUwbncNj48jalsJjffC9hwY%3D&reserved=0 > > ;-) > > > Having "native" support for schemas on Kafka records is definitely a > > good idea, and I am pretty sure Apache Kafka will move in that direction at > > some point.... > > > > > > Erik > > > > > > "This email with attachments is solely for the use of the individual or > > entity to which it is addressed. It may contain confidential or privileged > > information. If you are not the addressee, please notify the sender and > > delete this message and all attachments from your files." > > > > > > > > -- > > Claus Ibsen > > ----------------- > > https://eur04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsecure-web.cisco.com%2F1xpy_T_WkPmbFg4TwkqwyaZahfxz1MfSaZ02psLmX-wfBWpgZpJz4jaGHNVqayjeYuMD-08GpzTHiKX85nBQ7VriDB8JXKS9tyJ3dvESd-znuqFVxMitkW42oaBTUu4gqGeFoVQwf8l3QH6wMJbv6htaA_F4KOkpDFeMYULKjzViY3hK4YoEy_m9TPo2nNoxakhfpMjlKA8dDyp1ZlBiD1wS-dUqcXhHJ2ArG576KT-hHbjqd0XDf8UIsAVMvTApxiEksX8BnSMf5ZFNWIlmwL9b6OLmI1zLw7_ySidwEDXgbWb64GBEHXbS0h1eGP0F1NnQxQjY4wARrZk1EimDwfw%2Fhttp%253A%252F%252Fdavsclaus.com&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=Cxi9L9c1McgnkrcBQmcLLjpoi7RimVB%2FusqwR2e%2BugY%3D&reserved=0 > > @davsclaus > > Camel in Action 2: > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.manning.com%2Fibsen2&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=uZ3SSQeCEF7YH%2FRgBqjqwLcvSufu39Py3EDY0DllIb4%3D&reserved=0 > > -- Claus Ibsen ----------------- https://eur04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsecure-web.cisco.com%2F1xpy_T_WkPmbFg4TwkqwyaZahfxz1MfSaZ02psLmX-wfBWpgZpJz4jaGHNVqayjeYuMD-08GpzTHiKX85nBQ7VriDB8JXKS9tyJ3dvESd-znuqFVxMitkW42oaBTUu4gqGeFoVQwf8l3QH6wMJbv6htaA_F4KOkpDFeMYULKjzViY3hK4YoEy_m9TPo2nNoxakhfpMjlKA8dDyp1ZlBiD1wS-dUqcXhHJ2ArG576KT-hHbjqd0XDf8UIsAVMvTApxiEksX8BnSMf5ZFNWIlmwL9b6OLmI1zLw7_ySidwEDXgbWb64GBEHXbS0h1eGP0F1NnQxQjY4wARrZk1EimDwfw%2Fhttp%253A%252F%252Fdavsclaus.com&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=Cxi9L9c1McgnkrcBQmcLLjpoi7RimVB%2FusqwR2e%2BugY%3D&reserved=0 @davsclaus Camel in Action 2: https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.manning.com%2Fibsen2&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=uZ3SSQeCEF7YH%2FRgBqjqwLcvSufu39Py3EDY0DllIb4%3D&reserved=0 "This email with attachments is solely for the use of the individual or entity to which it is addressed. It may contain confidential or privileged information. If you are not the addressee, please notify the sender and delete this message and all attachments from your files."
