Re: Code review for NIFI-7865

[Nandor Soma Abonyi]

Hi Seda!

I’d like to help, but I only can in the evening. If nobody hops in until then, 
I’ll do it.

Best regards,
Soma

> On Nov 15, 2021, at 11:32 AM, Seda Dogan  wrote:
> 
> Hello together,
> 
> I have a pull request. https://github.com/apache/nifi/pull/5458 . I have 
> already an approve. I need one more.
> Can you help?
> 
> 
> Kind Regards
> Seda Dogan
> [ABAS ERP]
> abas Software GmbH | Gartenstraße 67 | 76135 Karlsruhe | Germany
> Phone: +49(0)721-96723-0 | Fax: +49(0)721-96723-100
> E-Mail: seda.do...@abas.de | Web: 
> https://abas-erp.com/de
> 
> [ABAS ERP 20 
> Release]
> 
> Managing Directors / Geschäftsführer: Richard Furby, Paul Smolinski
> Registered Office / Sitz der Gesellschaft: Karlsruhe
> Commercial Register / Handelsregister: HRB 734651 Amtsgericht Mannheim
> 
> [ISO 9001][BITMi Software Made in Germany]
> 
> Informationen zum Umgang mit Ihren personenbezogenen Daten nach den Art. 13 
> und 14 EU-DSGVO
> finden Sie unter https://abas-erp.com/de/datenschutz.
> Further information according to Art. 13 and 14 GDPR about processing your 
> personal data you can find
> here: https://abas-erp.com/en/privacy-copyright.

RE: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0)

[Bilal Bektas]

Hi Pierre,

Thank you for quick response.

You can find JIRA:
https://issues.apache.org/jira/browse/NIFI-9380

Thank you in advance,

--Bilal


-Original Message-
From: Pierre Villard  
Sent: 16 Kasım 2021 Salı 12:59
To: dev@nifi.apache.org
Subject: Re: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0)

Hi Bilal,

Can you please file a JIRA? I won't have time to dig into this before some time.

Le mar. 16 nov. 2021 à 09:06, Bilal Bektas  a écrit :

> Hi Pierre,
>
> I have tested this situation on NiFi 1.15.0-RC3. PutParquet processor 
> (Compression Type: SNAPPY) does not work correctly and gives the same 
> error :(
>
> To summarize the situation, this error only happens in NiFi 1.14.0 and
> 1.15.0 versions.
>
> Could you help in order to resolve this situation?
>
> Thank you in advance,
>
> --Bilal
>
>
>
> -Original Message-
> From: Pierre Villard 
> Sent: 15 Eylül 2021 Çarşamba 19:00
> To: dev@nifi.apache.org
> Subject: Re: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi 
> 1.14.0)
>
> OK so we upgraded the version of parquet-avro in [1]. The snappy-java 
> version between both is not the same so I *guess* there could be 
> something weird in terms of classloader with another dependency conflicting. 
> Checking.
>
> [1]
>
> https://github.com/apache/nifi/commit/5108d7cdd015ef14e98f2acfb61b0213
> 972fe29e#diff-c2fdc63240d48e73764ca171426b65fc143249bd466b9ba71eb76664
> cceb4410
>
> Le mer. 15 sept. 2021 à 15:53, Bilal Bektas  a 
> écrit :
>
> > Hi Pierre,
> >
> > Thank you for your return.
> >
> > In order to do lean testing, the default configuration was used
> generally:
> > * nifi-env.sh file has the default configuration.
> > * bootstrap.conf file has the default configuration.
> > * nifi.properties file has the default configuration except security 
> > configuration.
> > * PutParquet Processor has the default configuration except 
> > configuration which I mentioned in the previous post. (But SNAPPY 
> > compression is not
> > working)
> > * ConvertAvroToParquet Processor has the default configuration.
> > (SNAPPY compression is working correctly)
> > * There is no custom processor in our NiFi environment.
> > * There is no custom lib location in Nifi properties.
> >
> > As I mentioned in my previous email, this error happens only in NiFi
> > 1.14.0 version.
> >
> > Thank you for helping,
> >
> > --Bilal
> >
> >
> > -Original Message-
> > From: Pierre Villard 
> > Sent: 15 Eylül 2021 Çarşamba 13:50
> > To: dev@nifi.apache.org
> > Subject: Re: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi
> > 1.14.0)
> >
> > Hi,
> >
> > Is there something you changed on how the Snappy libraries are 
> > loaded in the NiFi JVM?
> > I'm not familiar with this specifically but you may need more recent 
> > versions of the native libraries.
> > Also, anything related to a snappy-java lib you added somewhere? /lib?
> > custom processor? etc.
> >
> > I believe it may be related to
> > https://github.com/apache/nifi/pull/4601
> > where we're adding snappy-java and maybe there is a version conflict
> there.
> >
> > Pierre
> >
> > Le mer. 15 sept. 2021 à 12:38, Bilal Bektas  
> > a écrit :
> >
> > > Hi Dev Team,
> > >
> > > To be able to detect the problem, is there anything you expect 
> > > from us or is there anything we can help with
> > >
> > > Thank you in advance,
> > >
> > > --Bilal
> > >
> > >
> > > -Original Message-
> > > From: Bilal Bektas 
> > > Sent: 1 Eylül 2021 Çarşamba 12:43
> > > To: dev@nifi.apache.org
> > > Subject: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi
> > > 1.14.0)
> > >
> > > Hi Dev Team,
> > >
> > > I have tested different compression types which is a feature of 
> > > PutParquet and ConvertAvroToParquet Processors on different NiFi
> > versions.
> > >
> > > To avoid confusion, i will give a summary information:
> > >
> > > * Compression types (UNCOMPRESSED, GZIP, SNAPPY) of PutParquet 
> > > Processor works correctly on NiFi 1.12.1 and 1.13.2
> > > * Compression types (UNCOMPRESSED, GZIP) of PutParquet Processor 
> > > works correctly on NiFi 1.14.0; SNAPPY gives an error in the 
> > > previous email
> > >
> > > * Compression types (UNCOMPRESSED, GZIP, SNAPPY) of 
> > > ConvertAvroToParquet Processor works correctly on NiFi 1.12.1,
> > > 1.13.2
> > and 1.14.0.
> > >
> > >
> > > PutParquet - Properties:
> > > * Hadoop Configuration Resources: File locations
> > > * Kerberos Credentials Service: Keytab service
> > > * Record Reader: AvroReader Service (Embedded Avro Schema)
> > > * Overwrite Files: True
> > > * Compression Type: SNAPPY
> > > * Other Properties: Default
> > >
> > > Error Log (nifi-app.log):
> > > Class org.xerial.snappy.SnappyNative does not implement the 
> > > requested interface org.xerial.snappy.SnappyApi
> > >
> > >
> > > Could you help in order to resolve this situation?
> > >
> > > Thank you in advance,
> > >
> > > --Bilal
> > >
> > >
> > >
> > > -Original Message-
> > > From: Bilal Bektas 
> > >

Re: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0)

[Pierre Villard]

Hi Bilal,

Can you please file a JIRA? I won't have time to dig into this before some
time.

Le mar. 16 nov. 2021 à 09:06, Bilal Bektas  a
écrit :

> Hi Pierre,
>
> I have tested this situation on NiFi 1.15.0-RC3. PutParquet processor
> (Compression Type: SNAPPY) does not work correctly and gives the same error
> :(
>
> To summarize the situation, this error only happens in NiFi 1.14.0 and
> 1.15.0 versions.
>
> Could you help in order to resolve this situation?
>
> Thank you in advance,
>
> --Bilal
>
>
>
> -Original Message-
> From: Pierre Villard 
> Sent: 15 Eylül 2021 Çarşamba 19:00
> To: dev@nifi.apache.org
> Subject: Re: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0)
>
> OK so we upgraded the version of parquet-avro in [1]. The snappy-java
> version between both is not the same so I *guess* there could be something
> weird in terms of classloader with another dependency conflicting. Checking.
>
> [1]
>
> https://github.com/apache/nifi/commit/5108d7cdd015ef14e98f2acfb61b0213972fe29e#diff-c2fdc63240d48e73764ca171426b65fc143249bd466b9ba71eb76664cceb4410
>
> Le mer. 15 sept. 2021 à 15:53, Bilal Bektas  a
> écrit :
>
> > Hi Pierre,
> >
> > Thank you for your return.
> >
> > In order to do lean testing, the default configuration was used
> generally:
> > * nifi-env.sh file has the default configuration.
> > * bootstrap.conf file has the default configuration.
> > * nifi.properties file has the default configuration except security
> > configuration.
> > * PutParquet Processor has the default configuration except
> > configuration which I mentioned in the previous post. (But SNAPPY
> > compression is not
> > working)
> > * ConvertAvroToParquet Processor has the default configuration.
> > (SNAPPY compression is working correctly)
> > * There is no custom processor in our NiFi environment.
> > * There is no custom lib location in Nifi properties.
> >
> > As I mentioned in my previous email, this error happens only in NiFi
> > 1.14.0 version.
> >
> > Thank you for helping,
> >
> > --Bilal
> >
> >
> > -Original Message-
> > From: Pierre Villard 
> > Sent: 15 Eylül 2021 Çarşamba 13:50
> > To: dev@nifi.apache.org
> > Subject: Re: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi
> > 1.14.0)
> >
> > Hi,
> >
> > Is there something you changed on how the Snappy libraries are loaded
> > in the NiFi JVM?
> > I'm not familiar with this specifically but you may need more recent
> > versions of the native libraries.
> > Also, anything related to a snappy-java lib you added somewhere? /lib?
> > custom processor? etc.
> >
> > I believe it may be related to
> > https://github.com/apache/nifi/pull/4601
> > where we're adding snappy-java and maybe there is a version conflict
> there.
> >
> > Pierre
> >
> > Le mer. 15 sept. 2021 à 12:38, Bilal Bektas  a
> > écrit :
> >
> > > Hi Dev Team,
> > >
> > > To be able to detect the problem, is there anything you expect from
> > > us or is there anything we can help with
> > >
> > > Thank you in advance,
> > >
> > > --Bilal
> > >
> > >
> > > -Original Message-
> > > From: Bilal Bektas 
> > > Sent: 1 Eylül 2021 Çarşamba 12:43
> > > To: dev@nifi.apache.org
> > > Subject: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi
> > > 1.14.0)
> > >
> > > Hi Dev Team,
> > >
> > > I have tested different compression types which is a feature of
> > > PutParquet and ConvertAvroToParquet Processors on different NiFi
> > versions.
> > >
> > > To avoid confusion, i will give a summary information:
> > >
> > > * Compression types (UNCOMPRESSED, GZIP, SNAPPY) of PutParquet
> > > Processor works correctly on NiFi 1.12.1 and 1.13.2
> > > * Compression types (UNCOMPRESSED, GZIP) of PutParquet Processor
> > > works correctly on NiFi 1.14.0; SNAPPY gives an error in the
> > > previous email
> > >
> > > * Compression types (UNCOMPRESSED, GZIP, SNAPPY) of
> > > ConvertAvroToParquet Processor works correctly on NiFi 1.12.1,
> > > 1.13.2
> > and 1.14.0.
> > >
> > >
> > > PutParquet - Properties:
> > > * Hadoop Configuration Resources: File locations
> > > * Kerberos Credentials Service: Keytab service
> > > * Record Reader: AvroReader Service (Embedded Avro Schema)
> > > * Overwrite Files: True
> > > * Compression Type: SNAPPY
> > > * Other Properties: Default
> > >
> > > Error Log (nifi-app.log):
> > > Class org.xerial.snappy.SnappyNative does not implement the
> > > requested interface org.xerial.snappy.SnappyApi
> > >
> > >
> > > Could you help in order to resolve this situation?
> > >
> > > Thank you in advance,
> > >
> > > --Bilal
> > >
> > >
> > >
> > > -Original Message-
> > > From: Bilal Bektas 
> > > Sent: 3 Ağustos 2021 Salı 14:55
> > > To: dev@nifi.apache.org
> > > Subject: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0)
> > >
> > > Hi Dev Team,
> > >
> > > SNAPPY compression feature of PutParquet Processor works correctly
> > > on NiFi
> > > 1.12.1 but NiFi 1.14.0 gives the following error.
> > >
> > > Could you help in order to resolve this 

RE: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0)

[Bilal Bektas]

Hi Pierre,

I have tested this situation on NiFi 1.15.0-RC3. PutParquet processor 
(Compression Type: SNAPPY) does not work correctly and gives the same error :(

To summarize the situation, this error only happens in NiFi 1.14.0 and 1.15.0 
versions.

Could you help in order to resolve this situation?

Thank you in advance,

--Bilal



-Original Message-
From: Pierre Villard  
Sent: 15 Eylül 2021 Çarşamba 19:00
To: dev@nifi.apache.org
Subject: Re: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0)

OK so we upgraded the version of parquet-avro in [1]. The snappy-java version 
between both is not the same so I *guess* there could be something weird in 
terms of classloader with another dependency conflicting. Checking.

[1]
https://github.com/apache/nifi/commit/5108d7cdd015ef14e98f2acfb61b0213972fe29e#diff-c2fdc63240d48e73764ca171426b65fc143249bd466b9ba71eb76664cceb4410

Le mer. 15 sept. 2021 à 15:53, Bilal Bektas  a écrit :

> Hi Pierre,
>
> Thank you for your return.
>
> In order to do lean testing, the default configuration was used generally:
> * nifi-env.sh file has the default configuration.
> * bootstrap.conf file has the default configuration.
> * nifi.properties file has the default configuration except security 
> configuration.
> * PutParquet Processor has the default configuration except 
> configuration which I mentioned in the previous post. (But SNAPPY 
> compression is not
> working)
> * ConvertAvroToParquet Processor has the default configuration. 
> (SNAPPY compression is working correctly)
> * There is no custom processor in our NiFi environment.
> * There is no custom lib location in Nifi properties.
>
> As I mentioned in my previous email, this error happens only in NiFi
> 1.14.0 version.
>
> Thank you for helping,
>
> --Bilal
>
>
> -Original Message-
> From: Pierre Villard 
> Sent: 15 Eylül 2021 Çarşamba 13:50
> To: dev@nifi.apache.org
> Subject: Re: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi 
> 1.14.0)
>
> Hi,
>
> Is there something you changed on how the Snappy libraries are loaded 
> in the NiFi JVM?
> I'm not familiar with this specifically but you may need more recent 
> versions of the native libraries.
> Also, anything related to a snappy-java lib you added somewhere? /lib?
> custom processor? etc.
>
> I believe it may be related to 
> https://github.com/apache/nifi/pull/4601
> where we're adding snappy-java and maybe there is a version conflict there.
>
> Pierre
>
> Le mer. 15 sept. 2021 à 12:38, Bilal Bektas  a 
> écrit :
>
> > Hi Dev Team,
> >
> > To be able to detect the problem, is there anything you expect from 
> > us or is there anything we can help with
> >
> > Thank you in advance,
> >
> > --Bilal
> >
> >
> > -Original Message-
> > From: Bilal Bektas 
> > Sent: 1 Eylül 2021 Çarşamba 12:43
> > To: dev@nifi.apache.org
> > Subject: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi
> > 1.14.0)
> >
> > Hi Dev Team,
> >
> > I have tested different compression types which is a feature of 
> > PutParquet and ConvertAvroToParquet Processors on different NiFi
> versions.
> >
> > To avoid confusion, i will give a summary information:
> >
> > * Compression types (UNCOMPRESSED, GZIP, SNAPPY) of PutParquet 
> > Processor works correctly on NiFi 1.12.1 and 1.13.2
> > * Compression types (UNCOMPRESSED, GZIP) of PutParquet Processor 
> > works correctly on NiFi 1.14.0; SNAPPY gives an error in the 
> > previous email
> >
> > * Compression types (UNCOMPRESSED, GZIP, SNAPPY) of 
> > ConvertAvroToParquet Processor works correctly on NiFi 1.12.1, 
> > 1.13.2
> and 1.14.0.
> >
> >
> > PutParquet - Properties:
> > * Hadoop Configuration Resources: File locations
> > * Kerberos Credentials Service: Keytab service
> > * Record Reader: AvroReader Service (Embedded Avro Schema)
> > * Overwrite Files: True
> > * Compression Type: SNAPPY
> > * Other Properties: Default
> >
> > Error Log (nifi-app.log):
> > Class org.xerial.snappy.SnappyNative does not implement the 
> > requested interface org.xerial.snappy.SnappyApi
> >
> >
> > Could you help in order to resolve this situation?
> >
> > Thank you in advance,
> >
> > --Bilal
> >
> >
> >
> > -Original Message-
> > From: Bilal Bektas 
> > Sent: 3 Ağustos 2021 Salı 14:55
> > To: dev@nifi.apache.org
> > Subject: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0)
> >
> > Hi Dev Team,
> >
> > SNAPPY compression feature of PutParquet Processor works correctly 
> > on NiFi
> > 1.12.1 but NiFi 1.14.0 gives the following error.
> >
> > Could you help in order to resolve this situation?
> >
> > Thank you in advance,
> >
> > --Bilal
> >
> >
> > PutParquet - Config:
> > Hadoop Configuration Resources
> >
> > File locations
> >
> > Kerberos Credentials Service
> >
> > Keytab service
> >
> > Kerberos Relogin Period
> >
> > 4 Hours
> >
> > Record Reader
> >
> > AvroReader Service (Embedded Avro Schema)
> >
> > Compression Type
> >
> > SNAPPY
> >
> >
> >
> > Error Log (nifi-app.log):
> > 2021-08-03 14:1