Re: Schema Registry
And looks like with NiFi 2.0 though we will drop the Hortonworks Schema Registry as shown here [1]. If maintenance becomes more active for it we could revisit as needed. [1] https://issues.apache.org/jira/browse/NIFI-11095 On Wed, Jul 19, 2023 at 12:32 PM Joe Witt wrote: > Hello > > You will want to check with Cloudera for the status of the Schema Registry > in question here. From a NiFi point of view we will continue to integrate > with schema registries we see in common usage and it should be set up well > to plug in new ones and phase older ones as we go. Support for the > Hwx/Cldr one is there. Support for Confluent is there as is our built-in > simple one. There is a new registry supported by Redhat that looks > interesting, etc.. > > Thanks > > On Wed, Jul 19, 2023 at 12:28 PM Dennis Suhari > wrote: > >> Hi, >> as far as I know the Schema registry has developed into >> https://docs.cloudera.com/runtime/7.2.16/schema-registry-overview/topics/csp-schema_registry_overview.html >> with >> features like authorization and proper get schema (without proprietary >> envelope) methods. >> >> Br, >> >> Dennis >> >> Von meinem iPhone gesendet >> >> Am 19.07.2023 um 02:56 schrieb Jennifer L Tress : >> >> >> Hello, >> >> Is the Hortonworks Schema Registry still actively developed and >> supported? I would like to use RecordReaders and Writers in my NiFi data >> flow for transformations between schemas. Since Hortonworks was bought >> by Cloudera, I wanted to check that record serialization was still a use >> case supported by NiFi and verify what schema registry to use. >> >> Thank you! >> >>
Re: Schema Registry
Hello You will want to check with Cloudera for the status of the Schema Registry in question here. From a NiFi point of view we will continue to integrate with schema registries we see in common usage and it should be set up well to plug in new ones and phase older ones as we go. Support for the Hwx/Cldr one is there. Support for Confluent is there as is our built-in simple one. There is a new registry supported by Redhat that looks interesting, etc.. Thanks On Wed, Jul 19, 2023 at 12:28 PM Dennis Suhari wrote: > Hi, > as far as I know the Schema registry has developed into > https://docs.cloudera.com/runtime/7.2.16/schema-registry-overview/topics/csp-schema_registry_overview.html > with > features like authorization and proper get schema (without proprietary > envelope) methods. > > Br, > > Dennis > > Von meinem iPhone gesendet > > Am 19.07.2023 um 02:56 schrieb Jennifer L Tress : > > > Hello, > > Is the Hortonworks Schema Registry still actively developed and supported? > I would like to use RecordReaders and Writers in my NiFi data flow for > transformations between schemas. Since Hortonworks was bought by > Cloudera, I wanted to check that record serialization was still a use > case supported by NiFi and verify what schema registry to use. > > Thank you! > >
Re: Schema Registry
Hi, as far as I know the Schema registry has developed into https://docs.cloudera.com/runtime/7.2.16/schema-registry-overview/topics/csp-schema_registry_overview.html with features like authorization and proper get schema (without proprietary envelope) methods. Br, Dennis Von meinem iPhone gesendet > Am 19.07.2023 um 02:56 schrieb Jennifer L Tress : > > > Hello, > > Is the Hortonworks Schema Registry still actively developed and supported? I > would like to use RecordReaders and Writers in my NiFi data flow for > transformations between schemas. Since Hortonworks was bought by Cloudera, I > wanted to check that record serialization was still a use case supported by > NiFi and verify what schema registry to use. > > Thank you! >
Schema Registry
Hello, Is the Hortonworks Schema Registry still actively developed and supported? I would like to use RecordReaders and Writers in my NiFi data flow for transformations between schemas. Since Hortonworks was bought by Cloudera, I wanted to check that record serialization was still a use case supported by NiFi and verify what schema registry to use. Thank you!
Re: How to use Confluent schema-registry with Publish/ConsumeKafka_2_0?
Hello, In your record readers/writer you need to use a Schema Access Strategy that uses a Schema Registry, like "Schema by Name" or in your case "Confluent Content-Encoded Schema Reference", which then goes to whichever schema registry is configured and asks for the schema with the given name or id. In your case you would setup a Confluent Schema Registry services and select that in your record reader/writer. Thanks, Bryan On Fri, May 1, 2020 at 2:01 PM David Hoffer wrote: > We are using the standard PublishKafka_2_0 and ConsumeKafka_2_0 NiFi > processors but we would now like to use Confluent/Kafka schema-registry as > well. How can I integrate this with these or other processors? > > Thanks, > -Dave >
How to use Confluent schema-registry with Publish/ConsumeKafka_2_0?
We are using the standard PublishKafka_2_0 and ConsumeKafka_2_0 NiFi processors but we would now like to use Confluent/Kafka schema-registry as well. How can I integrate this with these or other processors? Thanks, -Dave
NIFI - Kafka & confluent Schema registry example
Hi, I am looking for an example where I need to push data to kafka topic and at the same time use schema registry of confluent to use the schema for this topic. can anyone please share examples/templates if anyone has it. That would be really helpful. Thanks, Asmath
Re: Unable to connect to Confluent Schema Registry with credentials
Thanks, Bryan. NiFi-6576 <https://issues.apache.org/jira/browse/NIFI-6576> is created. Thanks, Ali On Wed, Aug 21, 2019 at 12:10 AM Bryan Bende wrote: > I believe it would require a code change to the > ConfluentSchemaRegistry service to have properties for basic auth > username and password, and then pass them down to the client calls. > > Feel free to create a JIRA to capture this improvement. > > On Mon, Aug 19, 2019 at 8:17 PM Ali Nazemian > wrote: > > > > I would like to use Confluent Schema Registry processor to retrieve a > schema and use it accordingly. SSL is enabled for the confluent schema > registry with basic authentication. I am not able to get the Confluent > Schema Registry processor to work with the credentials. If I disable > credentials everything is fine, but once I enabled it I don't have any > option in the processor to provide credentials. I tried to use the > credential in the URL, but it looks like NiFi ignores that as I still see > the 401 Unauthorized exception. I would appreciate it if someone could help > me to resolve this issue. > > > > P.S: I am using this processor: > > > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-confluent-platform-nar/1.9.0/org.apache.nifi.confluent.schemaregistry.ConfluentSchemaRegistry/index.html > > > > Thanks, > > Ali > -- A.Nazemian
Re: Unable to connect to Confluent Schema Registry with credentials
I believe it would require a code change to the ConfluentSchemaRegistry service to have properties for basic auth username and password, and then pass them down to the client calls. Feel free to create a JIRA to capture this improvement. On Mon, Aug 19, 2019 at 8:17 PM Ali Nazemian wrote: > > I would like to use Confluent Schema Registry processor to retrieve a schema > and use it accordingly. SSL is enabled for the confluent schema registry with > basic authentication. I am not able to get the Confluent Schema Registry > processor to work with the credentials. If I disable credentials everything > is fine, but once I enabled it I don't have any option in the processor to > provide credentials. I tried to use the credential in the URL, but it looks > like NiFi ignores that as I still see the 401 Unauthorized exception. I would > appreciate it if someone could help me to resolve this issue. > > P.S: I am using this processor: > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-confluent-platform-nar/1.9.0/org.apache.nifi.confluent.schemaregistry.ConfluentSchemaRegistry/index.html > > Thanks, > Ali
Unable to connect to Confluent Schema Registry with credentials
I would like to use Confluent Schema Registry processor to retrieve a schema and use it accordingly. SSL is enabled for the confluent schema registry with basic authentication. I am not able to get the Confluent Schema Registry processor to work with the credentials. If I disable credentials everything is fine, but once I enabled it I don't have any option in the processor to provide credentials. I tried to use the credential in the URL, but it looks like NiFi ignores that as I still see the 401 Unauthorized exception. I would appreciate it if someone could help me to resolve this issue. P.S: I am using this processor: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-confluent-platform-nar/1.9.0/org.apache.nifi.confluent.schemaregistry.ConfluentSchemaRegistry/index.html Thanks, Ali
Re: Weird behavior w/ HWX schema registry new builds
Yes, it works after that. We are stuck with Nifi 1.5.0 (it uses 0.3.0), so after upgraded the client library (to 0.5.1) it works. Probably you are facing a similar problem. Notice you also need NIFI-4893 <https://issues.apache.org/jira/browse/NIFI-4893>applied for some uses cases. Juan On Mon, 15 Apr 2019 at 13:46, Michael Pearce wrote: > Sounds like a client compatibility issue, maybe raise a bug on HWX schema > registry, pretty bad to be breaking api calls. > > > > *From: *Juan Pablo Gardella > *Reply-To: *"users@nifi.apache.org" > *Date: *Monday, April 15, 2019 at 5:44 PM > *To: *"users@nifi.apache.org" > *Subject: *Re: Weird behavior w/ HWX schema registry new builds > > > > I had to rebuild it using a newer version of HWX client. > > > > On Mon, 15 Apr 2019 at 13:24, Mike Thomsen wrote: > > We deployed 0.7.0 and started running into issues with it saying it could > not find a schema. Really weird since we can see the schema in the UI and > get the metadata about it with the same call that returns a failure message. > > > > Anyone seen similar behavior or have any idea what might be happening? > Just trying to figure out if there are any known issues before I start > poking around at our code and the registry's code. > > > > Thanks, > > > > Mike > > The information contained in this email is strictly confidential and for > the use of the addressee only, unless otherwise indicated. If you are not > the intended recipient, please do not read, copy, use or disclose to others > this message or any attachment. Please also notify the sender by replying > to this email or by telephone (+44 (0)20 7896 0011) and then delete the > email and any copies of it. Opinions, conclusions (etc) that do not relate > to the official business of this company shall be understood as neither > given nor endorsed by it. IG Group Holdings plc is a company registered in > England and Wales under number 04677092. VAT registration number 761 2978 > 07. Registered Office: Cannon Bridge House, 25 Dowgate Hill, London EC4R > 2YA. Listed on the London Stock Exchange. Its subsidiaries IG Markets > Limited and IG Index Limited are authorised and regulated by the Financial > Conduct Authority (IG Markets Limited FCA registration number 195355 and IG > Index Limited FCA registration number 114059). IG Europe GmbH is authorised > and regulated by the Bundesanstalt für Finanzdienstleistungsaufsicht (BaFin > registration number 148759) and the Deutsche Bundesbank. The Swedish > branches of IG Markets Ltd and IG Europe GmbH are regulated by the > Finansinspektionen.- >
Re: Weird behavior w/ HWX schema registry new builds
Sounds like a client compatibility issue, maybe raise a bug on HWX schema registry, pretty bad to be breaking api calls. From: Juan Pablo Gardella Reply-To: "users@nifi.apache.org" Date: Monday, April 15, 2019 at 5:44 PM To: "users@nifi.apache.org" Subject: Re: Weird behavior w/ HWX schema registry new builds I had to rebuild it using a newer version of HWX client. On Mon, 15 Apr 2019 at 13:24, Mike Thomsen mailto:mikerthom...@gmail.com>> wrote: We deployed 0.7.0 and started running into issues with it saying it could not find a schema. Really weird since we can see the schema in the UI and get the metadata about it with the same call that returns a failure message. Anyone seen similar behavior or have any idea what might be happening? Just trying to figure out if there are any known issues before I start poking around at our code and the registry's code. Thanks, Mike The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44 (0)20 7896 0011) and then delete the email and any copies of it. Opinions, conclusions (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG Group Holdings plc is a company registered in England and Wales under number 04677092. VAT registration number 761 2978 07. Registered Office: Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Listed on the London Stock Exchange. Its subsidiaries IG Markets Limited and IG Index Limited are authorised and regulated by the Financial Conduct Authority (IG Markets Limited FCA registration number 195355 and IG Index Limited FCA registration number 114059). IG Europe GmbH is authorised and regulated by the Bundesanstalt für Finanzdienstleistungsaufsicht (BaFin registration number 148759) and the Deutsche Bundesbank. The Swedish branches of IG Markets Ltd and IG Europe GmbH are regulated by the Finansinspektionen.-
Re: Weird behavior w/ HWX schema registry new builds
Worked fine after that? If so, I'll write up a PR. Thanks, Mike On Mon, Apr 15, 2019 at 12:44 PM Juan Pablo Gardella < gardellajuanpa...@gmail.com> wrote: > I had to rebuild it using a newer version of HWX client. > > On Mon, 15 Apr 2019 at 13:24, Mike Thomsen wrote: > >> We deployed 0.7.0 and started running into issues with it saying it could >> not find a schema. Really weird since we can see the schema in the UI and >> get the metadata about it with the same call that returns a failure message. >> >> Anyone seen similar behavior or have any idea what might be happening? >> Just trying to figure out if there are any known issues before I start >> poking around at our code and the registry's code. >> >> Thanks, >> >> Mike >> >
Re: Weird behavior w/ HWX schema registry new builds
I had to rebuild it using a newer version of HWX client. On Mon, 15 Apr 2019 at 13:24, Mike Thomsen wrote: > We deployed 0.7.0 and started running into issues with it saying it could > not find a schema. Really weird since we can see the schema in the UI and > get the metadata about it with the same call that returns a failure message. > > Anyone seen similar behavior or have any idea what might be happening? > Just trying to figure out if there are any known issues before I start > poking around at our code and the registry's code. > > Thanks, > > Mike >
Weird behavior w/ HWX schema registry new builds
We deployed 0.7.0 and started running into issues with it saying it could not find a schema. Really weird since we can see the schema in the UI and get the metadata about it with the same call that returns a failure message. Anyone seen similar behavior or have any idea what might be happening? Just trying to figure out if there are any known issues before I start poking around at our code and the registry's code. Thanks, Mike
Re: Preferred schema registry
Isn't AvroSchemaRegistry an embedded one to NiFi? I.e. it's not a comparable alternative to an external dedicated schema registry. Andrew On Wed, Jan 16, 2019 at 7:43 AM Mike Thomsen wrote: > I think I figured out what it was. We backed out a change and it was > fungible with an earlier version of the schema. Therefore it appears that > it didn't create a new version of the schema. When I made a more meaningful > change, it created a new 4th version. > > Thanks, > > Mike > > On Wed, Jan 16, 2019 at 10:39 AM Bryan Bende wrote: > >> Mike, >> >> Do you have anymore details about why the Hortonworks schema registry >> stopped working? >> >> I have used it before and didn’t have any issues. >> >> Thanks >> >> Bryan >> >> On Tue, Jan 15, 2019 at 8:38 PM dan young wrote: >> >>> We used the AvroSchemaRegistry >>> >>> Dano >>> >>> On Tue, Jan 15, 2019, 12:51 PM Mike Thomsen >> wrote: >>> >>>> What schema registry are others using in production use cases? We tried >>>> out the HortonWorks registry, but it seemed to stop accepting updates once >>>> we hit "v3" of our schema (we didn't name it v3, that's the version it >>>> showed in the UI). So I'd like to know what others are doing for their >>>> registry use since we're looking at either Confluent or going back to the >>>> AvroSchemaRegistry. >>>> >>>> Thanks, >>>> >>>> Mike >>>> >>> -- >> Sent from Gmail Mobile >> >
Re: Preferred schema registry
I think I figured out what it was. We backed out a change and it was fungible with an earlier version of the schema. Therefore it appears that it didn't create a new version of the schema. When I made a more meaningful change, it created a new 4th version. Thanks, Mike On Wed, Jan 16, 2019 at 10:39 AM Bryan Bende wrote: > Mike, > > Do you have anymore details about why the Hortonworks schema registry > stopped working? > > I have used it before and didn’t have any issues. > > Thanks > > Bryan > > On Tue, Jan 15, 2019 at 8:38 PM dan young wrote: > >> We used the AvroSchemaRegistry >> >> Dano >> >> On Tue, Jan 15, 2019, 12:51 PM Mike Thomsen > wrote: >> >>> What schema registry are others using in production use cases? We tried >>> out the HortonWorks registry, but it seemed to stop accepting updates once >>> we hit "v3" of our schema (we didn't name it v3, that's the version it >>> showed in the UI). So I'd like to know what others are doing for their >>> registry use since we're looking at either Confluent or going back to the >>> AvroSchemaRegistry. >>> >>> Thanks, >>> >>> Mike >>> >> -- > Sent from Gmail Mobile >
Re: Preferred schema registry
Mike, Do you have anymore details about why the Hortonworks schema registry stopped working? I have used it before and didn’t have any issues. Thanks Bryan On Tue, Jan 15, 2019 at 8:38 PM dan young wrote: > We used the AvroSchemaRegistry > > Dano > > On Tue, Jan 15, 2019, 12:51 PM Mike Thomsen >> What schema registry are others using in production use cases? We tried >> out the HortonWorks registry, but it seemed to stop accepting updates once >> we hit "v3" of our schema (we didn't name it v3, that's the version it >> showed in the UI). So I'd like to know what others are doing for their >> registry use since we're looking at either Confluent or going back to the >> AvroSchemaRegistry. >> >> Thanks, >> >> Mike >> > -- Sent from Gmail Mobile
Re: Preferred schema registry
We used the AvroSchemaRegistry Dano On Tue, Jan 15, 2019, 12:51 PM Mike Thomsen What schema registry are others using in production use cases? We tried > out the HortonWorks registry, but it seemed to stop accepting updates once > we hit "v3" of our schema (we didn't name it v3, that's the version it > showed in the UI). So I'd like to know what others are doing for their > registry use since we're looking at either Confluent or going back to the > AvroSchemaRegistry. > > Thanks, > > Mike >
Re: Preferred schema registry
We are using AvroSchemaRegistry on our side. On Wed, 16 Jan 2019 at 1:21 AM, Mike Thomsen wrote: > What schema registry are others using in production use cases? We tried > out the HortonWorks registry, but it seemed to stop accepting updates once > we hit "v3" of our schema (we didn't name it v3, that's the version it > showed in the UI). So I'd like to know what others are doing for their > registry use since we're looking at either Confluent or going back to the > AvroSchemaRegistry. > > Thanks, > > Mike >
Preferred schema registry
What schema registry are others using in production use cases? We tried out the HortonWorks registry, but it seemed to stop accepting updates once we hit "v3" of our schema (we didn't name it v3, that's the version it showed in the UI). So I'd like to know what others are doing for their registry use since we're looking at either Confluent or going back to the AvroSchemaRegistry. Thanks, Mike
Re: Kafka Schema registry
https://issues.apache.org/jira/browse/NIFI-1763 Please feel free to add your thoughts to that JIRA. On Wed, Apr 13, 2016 at 2:17 PM, Joe Witt wrote: > Ok will look into a bit and put in a JIRA for this idea. Will send > that on this thread to ensure it captures your thoughts and of course > please do add/augment it as you like. Are you interested in helping > contribute to this from a coding perspective as well? > > Thanks > Joe > > On Wed, Apr 13, 2016 at 2:09 PM, Madhukar Thota > wrote: >> Hi Joe, >> >> We are using Confluent version Kafka and using its schema registry to store >> Avro schema. we would like continue same with Nifi writing avro file to >> Confluent Kafka Schema registry. >> >> http://docs.confluent.io/2.0.0/schema-registry/docs/index.html >> >> -Madhu >> >> On Wed, Apr 13, 2016 at 1:48 PM, Joe Witt wrote: >>> >>> Madhu, >>> >>> Do you have any information you can point to for the registry? I know >>> of the Confluent one but I am not sure of its interfaces. If there >>> are open source friendly ones available it certainly would be a fine >>> thing to support. Can you point us to what you are looking at >>> specifically? >>> >>> Thanks >>> Joe >>> >>> On Wed, Apr 13, 2016 at 1:34 PM, Madhukar Thota >>> wrote: >>> > Friends, >>> > >>> > Is it possible to use Schema registry with Kafka Processors to store and >>> > retrive Avro schema? >>> > >>> > -Madhu >> >>
Re: Kafka Schema registry
Ok will look into a bit and put in a JIRA for this idea. Will send that on this thread to ensure it captures your thoughts and of course please do add/augment it as you like. Are you interested in helping contribute to this from a coding perspective as well? Thanks Joe On Wed, Apr 13, 2016 at 2:09 PM, Madhukar Thota wrote: > Hi Joe, > > We are using Confluent version Kafka and using its schema registry to store > Avro schema. we would like continue same with Nifi writing avro file to > Confluent Kafka Schema registry. > > http://docs.confluent.io/2.0.0/schema-registry/docs/index.html > > -Madhu > > On Wed, Apr 13, 2016 at 1:48 PM, Joe Witt wrote: >> >> Madhu, >> >> Do you have any information you can point to for the registry? I know >> of the Confluent one but I am not sure of its interfaces. If there >> are open source friendly ones available it certainly would be a fine >> thing to support. Can you point us to what you are looking at >> specifically? >> >> Thanks >> Joe >> >> On Wed, Apr 13, 2016 at 1:34 PM, Madhukar Thota >> wrote: >> > Friends, >> > >> > Is it possible to use Schema registry with Kafka Processors to store and >> > retrive Avro schema? >> > >> > -Madhu > >
Re: Kafka Schema registry
Hi Joe, We are using Confluent version Kafka and using its schema registry to store Avro schema. we would like continue same with Nifi writing avro file to Confluent Kafka Schema registry. http://docs.confluent.io/2.0.0/schema-registry/docs/index.html -Madhu On Wed, Apr 13, 2016 at 1:48 PM, Joe Witt wrote: > Madhu, > > Do you have any information you can point to for the registry? I know > of the Confluent one but I am not sure of its interfaces. If there > are open source friendly ones available it certainly would be a fine > thing to support. Can you point us to what you are looking at > specifically? > > Thanks > Joe > > On Wed, Apr 13, 2016 at 1:34 PM, Madhukar Thota > wrote: > > Friends, > > > > Is it possible to use Schema registry with Kafka Processors to store and > > retrive Avro schema? > > > > -Madhu >
Re: Kafka Schema registry
Madhu, Do you have any information you can point to for the registry? I know of the Confluent one but I am not sure of its interfaces. If there are open source friendly ones available it certainly would be a fine thing to support. Can you point us to what you are looking at specifically? Thanks Joe On Wed, Apr 13, 2016 at 1:34 PM, Madhukar Thota wrote: > Friends, > > Is it possible to use Schema registry with Kafka Processors to store and > retrive Avro schema? > > -Madhu
Kafka Schema registry
Friends, Is it possible to use Schema registry with Kafka Processors to store and retrive Avro schema? -Madhu
