Hi Erik, To be honest I will keep confluent discussion separate as I personally don't 100% agree with you and I am afraid this will need entirely separate thread :P.
Regardimg the maven plugin, this will solve different problem, the problem of maintaining the configurations with more recent versions. Yes as Claus mentioned, please create a ticket for it in order to allow the component to set custom configurations. Regards, Omar On Fri, Oct 18, 2019, 08:59 Erik Boye <erik.b...@statnett.no> wrote: > Åpen informasjon / Public information > > Hi, > > Sorry for being offline on this matter.... > > Claus, please do not get political on this. 😊 Confluent Kafka is also > open source - at least the stuff we are talking about here. And I am pretty > sure Confluent is one of, if not the, main contributor to Apache Kafka. > Enterprise open source is the "new way", and you cannot blame others for > doing the same as Red Hat has been doing successfully for many years! Camel > should, in my option, provide the extensibility mechanisms to fulfill Camel > users requirements - even if you consider them to be "proprietary".... > And no, I have absolutely no personal relationship to Confluent. I was > also a bit surprised when our Kafka Ops tells me I have to use a "schema > registry" to produce Kafka messages. But I have got used to it, and I even > like the idea of having some API/schema on the Kafka data streams.... > > A Maven plugin would not solve my original issues - even if it was in > place when I needed to configure some Confluent specific Kafka properties. > Or did I misunderstood something? I think we really need a way for users to > set (additional) Kafka consumer/producer properties that are not known to > Camel. And other "complex" Camel components do have that. That is my main > point of view. Do you want me to create an issue for this? > > Erik > > -----Opprinnelig melding----- > Fra: Claus Ibsen <claus.ib...@gmail.com> > Sendt: søndag 13. oktober 2019 18:19 > Til: users@camel.apache.org > Emne: [SUSPECT URL !]Re: Configuring custom Kafka producer/consumer > properties on Camel Kafka component/endpoint? > > Hi Omar > > Yeah the maven plugin idea is really good. > But maybe we need both. > > The maven plugin will only get you the options of the kafka version > that was used for that release. > With having a way for end users to set extra options then they could > work against newer Kafka versions without Camel have to upgrade. > And maybe also to proprietary options on Confluent. > > > On Sun, Oct 13, 2019 at 2:31 PM Omar Al-Safi <omars...@gmail.com> wrote: > > > > Hi Claus, > > > > I also thought about this, however since Kafka producers/consumers > options > > increase and some of them may gets deprecated, I was thinking we can > have a > > maven plugin that generate these configurations for us without the need > to > > do it manually, it will generate using the configDef (e.g: > > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkafka%2Fblob%2Ftrunk%2Fclients%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkafka%2Fclients%2Fconsumer%2FConsumerConfig.java%23L555&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788184830&sdata=S%2BJyTAx8u7V0jow9oCELojh9af5dsgDtxYjRzRz89Lg%3D&reserved=0 > ). > > I did this for DBZ as I built a maven plugin that scrap all these config > > and generate the camel configurations, you can take a look at my progress > > here: > > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fomarsmak%2Fcamel%2Fblob%2FCAMEL-14009%2Fcomponents%2Fcamel-debezium%2Fcamel-debezium-maven-plugin%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fcamel%2Fmaven%2FTestMojo.java%23L24&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788184830&sdata=hTRBEetZ7zBt%2BoS9fSfbQV94G0KRIudY0WlkDeo50Fs%3D&reserved=0 > . > > Please take a look and share your thoughts about it. > > > > Regards, > > Omar > > > > On Sun, 13 Oct 2019 at 14:12, Claus Ibsen <claus.ib...@gmail.com> wrote: > > > > > Hi > > > > > > Yeah this has come up a few times that Kafka is adding new options, or > > > more particularly Confluent adds propriety options that they dont give > > > back to the community. > > > > > > Can you create a JIRA ticket. > > > > > > Then we can look at adding a way to add custom options. You are of > > > course welcome to try to implement this. We love contributions. > > > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fcamel%2Fblob%2Fmaster%2FCONTRIBUTING.md&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=jPZhXs6oPXeBZKVQn54EQhA2kTqGLQHm5CqUmCDaXo8%3D&reserved=0 > > > > > > On Tue, Oct 8, 2019 at 9:57 AM Erik Boye <erik.b...@statnett.no> > wrote: > > > > > > > > Hi, > > > > > > > > Is there any way to configure custom Kafka producer/consumer > properties > > > on the Camel Kafka component/endpoint (without "breaking in")? > > > > I specifically want to configure the Confluent Kafka property with > key > > > 'value.subject.name.strategy': > > > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.confluent.io%2Fcurrent%2Fschema-registry%2Fserializer-formatter.html&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=XyX9RJL%2Fp2CMp%2Fg0enZ7dTrRQ5oalOIcLuy6nWxr%2BcA%3D&reserved=0 > > > > > > > > Other Camel components (i.e Camel ActiveMQ; destination.consumer) are > > > made extensible by providing users a way to set "additional > properties". > > > Why does not Camel Kafka provide that? Until schemaRegistryURL was > added as > > > a configurable property, we had to make our own Camel Kafka Component > to > > > make it work ("breaking in"). There should definitely be a way to set > > > additional/custom properties on Kafka producers/consumers. > > > > > > > > Just have a look at the Camel Kafka documentation and do a text > search > > > for Confluent: > > > > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcamel.apache.org%2Fcomponents%2Flatest%2Fkafka-component.html&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=YTimYbe%2FcTksQUj3wmG4pUwbncNj48jalsJjffC9hwY%3D&reserved=0 > ;-) > > > > Having "native" support for schemas on Kafka records is definitely a > > > good idea, and I am pretty sure Apache Kafka will move in that > direction at > > > some point.... > > > > > > > > Erik > > > > > > > > "This email with attachments is solely for the use of the individual > or > > > entity to which it is addressed. It may contain confidential or > privileged > > > information. If you are not the addressee, please notify the sender and > > > delete this message and all attachments from your files." > > > > > > > > > > > > -- > > > Claus Ibsen > > > ----------------- > > > > https://eur04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsecure-web.cisco.com%2F1xpy_T_WkPmbFg4TwkqwyaZahfxz1MfSaZ02psLmX-wfBWpgZpJz4jaGHNVqayjeYuMD-08GpzTHiKX85nBQ7VriDB8JXKS9tyJ3dvESd-znuqFVxMitkW42oaBTUu4gqGeFoVQwf8l3QH6wMJbv6htaA_F4KOkpDFeMYULKjzViY3hK4YoEy_m9TPo2nNoxakhfpMjlKA8dDyp1ZlBiD1wS-dUqcXhHJ2ArG576KT-hHbjqd0XDf8UIsAVMvTApxiEksX8BnSMf5ZFNWIlmwL9b6OLmI1zLw7_ySidwEDXgbWb64GBEHXbS0h1eGP0F1NnQxQjY4wARrZk1EimDwfw%2Fhttp%253A%252F%252Fdavsclaus.com&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=Cxi9L9c1McgnkrcBQmcLLjpoi7RimVB%2FusqwR2e%2BugY%3D&reserved=0 > @davsclaus > > > Camel in Action 2: > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.manning.com%2Fibsen2&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=uZ3SSQeCEF7YH%2FRgBqjqwLcvSufu39Py3EDY0DllIb4%3D&reserved=0 > > > > > > > -- > Claus Ibsen > ----------------- > > https://eur04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsecure-web.cisco.com%2F1xpy_T_WkPmbFg4TwkqwyaZahfxz1MfSaZ02psLmX-wfBWpgZpJz4jaGHNVqayjeYuMD-08GpzTHiKX85nBQ7VriDB8JXKS9tyJ3dvESd-znuqFVxMitkW42oaBTUu4gqGeFoVQwf8l3QH6wMJbv6htaA_F4KOkpDFeMYULKjzViY3hK4YoEy_m9TPo2nNoxakhfpMjlKA8dDyp1ZlBiD1wS-dUqcXhHJ2ArG576KT-hHbjqd0XDf8UIsAVMvTApxiEksX8BnSMf5ZFNWIlmwL9b6OLmI1zLw7_ySidwEDXgbWb64GBEHXbS0h1eGP0F1NnQxQjY4wARrZk1EimDwfw%2Fhttp%253A%252F%252Fdavsclaus.com&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=Cxi9L9c1McgnkrcBQmcLLjpoi7RimVB%2FusqwR2e%2BugY%3D&reserved=0 > @davsclaus > Camel in Action 2: > https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.manning.com%2Fibsen2&data=02%7C01%7CErik.Boye%40statnett.no%7C512ca3c98977460f22d908d74ff923f3%7Ca8d61462f25244b2bf6ad7231960c041%7C0%7C0%7C637065803788194819&sdata=uZ3SSQeCEF7YH%2FRgBqjqwLcvSufu39Py3EDY0DllIb4%3D&reserved=0 > "This email with attachments is solely for the use of the individual or > entity to which it is addressed. It may contain confidential or privileged > information. If you are not the addressee, please notify the sender and > delete this message and all attachments from your files." >
