Hi Pierre, Thank you for quick response.
You can find JIRA: https://issues.apache.org/jira/browse/NIFI-9380 Thank you in advance, --Bilal -----Original Message----- From: Pierre Villard <pierre.villard...@gmail.com> Sent: 16 Kasım 2021 Salı 12:59 To: dev@nifi.apache.org Subject: Re: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0) Hi Bilal, Can you please file a JIRA? I won't have time to dig into this before some time. Le mar. 16 nov. 2021 à 09:06, Bilal Bektas <bilal.bek...@obase.com> a écrit : > Hi Pierre, > > I have tested this situation on NiFi 1.15.0-RC3. PutParquet processor > (Compression Type: SNAPPY) does not work correctly and gives the same > error :( > > To summarize the situation, this error only happens in NiFi 1.14.0 and > 1.15.0 versions. > > Could you help in order to resolve this situation? > > Thank you in advance, > > --Bilal > > > > -----Original Message----- > From: Pierre Villard <pierre.villard...@gmail.com> > Sent: 15 Eylül 2021 Çarşamba 19:00 > To: dev@nifi.apache.org > Subject: Re: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi > 1.14.0) > > OK so we upgraded the version of parquet-avro in [1]. The snappy-java > version between both is not the same so I *guess* there could be > something weird in terms of classloader with another dependency conflicting. > Checking. > > [1] > > https://github.com/apache/nifi/commit/5108d7cdd015ef14e98f2acfb61b0213 > 972fe29e#diff-c2fdc63240d48e73764ca171426b65fc143249bd466b9ba71eb76664 > cceb4410 > > Le mer. 15 sept. 2021 à 15:53, Bilal Bektas <bilal.bek...@obase.com> a > écrit : > > > Hi Pierre, > > > > Thank you for your return. > > > > In order to do lean testing, the default configuration was used > generally: > > * nifi-env.sh file has the default configuration. > > * bootstrap.conf file has the default configuration. > > * nifi.properties file has the default configuration except security > > configuration. > > * PutParquet Processor has the default configuration except > > configuration which I mentioned in the previous post. (But SNAPPY > > compression is not > > working) > > * ConvertAvroToParquet Processor has the default configuration. > > (SNAPPY compression is working correctly) > > * There is no custom processor in our NiFi environment. > > * There is no custom lib location in Nifi properties. > > > > As I mentioned in my previous email, this error happens only in NiFi > > 1.14.0 version. > > > > Thank you for helping, > > > > --Bilal > > > > > > -----Original Message----- > > From: Pierre Villard <pierre.villard...@gmail.com> > > Sent: 15 Eylül 2021 Çarşamba 13:50 > > To: dev@nifi.apache.org > > Subject: Re: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi > > 1.14.0) > > > > Hi, > > > > Is there something you changed on how the Snappy libraries are > > loaded in the NiFi JVM? > > I'm not familiar with this specifically but you may need more recent > > versions of the native libraries. > > Also, anything related to a snappy-java lib you added somewhere? /lib? > > custom processor? etc. > > > > I believe it may be related to > > https://github.com/apache/nifi/pull/4601 > > where we're adding snappy-java and maybe there is a version conflict > there. > > > > Pierre > > > > Le mer. 15 sept. 2021 à 12:38, Bilal Bektas <bilal.bek...@obase.com> > > a écrit : > > > > > Hi Dev Team, > > > > > > To be able to detect the problem, is there anything you expect > > > from us or is there anything we can help with > > > > > > Thank you in advance, > > > > > > --Bilal > > > > > > > > > -----Original Message----- > > > From: Bilal Bektas <bilal.bek...@obase.com> > > > Sent: 1 Eylül 2021 Çarşamba 12:43 > > > To: dev@nifi.apache.org > > > Subject: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi > > > 1.14.0) > > > > > > Hi Dev Team, > > > > > > I have tested different compression types which is a feature of > > > PutParquet and ConvertAvroToParquet Processors on different NiFi > > versions. > > > > > > To avoid confusion, i will give a summary information: > > > > > > * Compression types (UNCOMPRESSED, GZIP, SNAPPY) of PutParquet > > > Processor works correctly on NiFi 1.12.1 and 1.13.2 > > > * Compression types (UNCOMPRESSED, GZIP) of PutParquet Processor > > > works correctly on NiFi 1.14.0; SNAPPY gives an error in the > > > previous email > > > > > > * Compression types (UNCOMPRESSED, GZIP, SNAPPY) of > > > ConvertAvroToParquet Processor works correctly on NiFi 1.12.1, > > > 1.13.2 > > and 1.14.0. > > > > > > > > > PutParquet - Properties: > > > * Hadoop Configuration Resources: File locations > > > * Kerberos Credentials Service: Keytab service > > > * Record Reader: AvroReader Service (Embedded Avro Schema) > > > * Overwrite Files: True > > > * Compression Type: SNAPPY > > > * Other Properties: Default > > > > > > Error Log (nifi-app.log): > > > Class org.xerial.snappy.SnappyNative does not implement the > > > requested interface org.xerial.snappy.SnappyApi > > > > > > > > > Could you help in order to resolve this situation? > > > > > > Thank you in advance, > > > > > > --Bilal > > > > > > > > > > > > -----Original Message----- > > > From: Bilal Bektas <bilal.bek...@obase.com> > > > Sent: 3 Ağustos 2021 Salı 14:55 > > > To: dev@nifi.apache.org > > > Subject: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0) > > > > > > Hi Dev Team, > > > > > > SNAPPY compression feature of PutParquet Processor works correctly > > > on NiFi > > > 1.12.1 but NiFi 1.14.0 gives the following error. > > > > > > Could you help in order to resolve this situation? > > > > > > Thank you in advance, > > > > > > --Bilal > > > > > > > > > PutParquet - Config: > > > Hadoop Configuration Resources > > > > > > File locations > > > > > > Kerberos Credentials Service > > > > > > Keytab service > > > > > > Kerberos Relogin Period > > > > > > 4 Hours > > > > > > Record Reader > > > > > > AvroReader Service (Embedded Avro Schema) > > > > > > Compression Type > > > > > > SNAPPY > > > > > > > > > > > > Error Log (nifi-app.log): > > > 2021-08-03 14:13:01,955 ERROR [Timer-Driven Process Thread-12] > > > o.a.nifi.processors.parquet.PutParquet > > > PutParquet[id=6caab337-68e8-3834-b64a-1d2cbd93aba8] Failed to > > > write due to > > > java.lang.IncompatibleClassChangeError: Class > > > org.xerial.snappy.SnappyNative does not implement the requested > > > interface > > > org.xerial.snappy.SnappyApi: java.lang.IncompatibleClassChangeError: > > > Class org.xerial.snappy.SnappyNative does not implement the > > > requested interface org.xerial.snappy.SnappyApi > > > java.lang.IncompatibleClassChangeError: Class > > > org.xerial.snappy.SnappyNative does not implement the requested > > > interface org.xerial.snappy.SnappyApi > > > at > org.xerial.snappy.Snappy.maxCompressedLength(Snappy.java:380) > > > at > > > > > org.apache.parquet.hadoop.codec.SnappyCompressor.compress(SnappyComp > > re > > ssor.java:67) > > > at > > > > > org.apache.hadoop.io.compress.CompressorStream.compress(CompressorSt > > re > > am.java:81) > > > at > > > > > org.apache.hadoop.io.compress.CompressorStream.finish(CompressorStre > > am > > .java:92) > > > at > > > > > org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.compress( > > Co > > decFactory.java:167) > > > at > > > > > org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageW > > ri > > ter.writePage(ColumnChunkPageWriteStore.java:168) > > > at > > > > > org.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriter > > V1 > > .java:59) > > > at > > > > > org.apache.parquet.column.impl.ColumnWriterBase.writePage(ColumnWrit > > er > > Base.java:387) > > > at > > > > > org.apache.parquet.column.impl.ColumnWriteStoreBase.flush(ColumnWrit > > eS > > toreBase.java:186) > > > at > > > > > org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteS > > to > > reV1.java:29) > > > at > > > > > org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupT > > oS > > tore(InternalParquetRecordWriter.java:185) > > > at > > > > > org.apache.parquet.hadoop.InternalParquetRecordWriter.close(Internal > > Pa > > rquetRecordWriter.java:124) > > > at > > > org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:319) > > > at > > > > > org.apache.nifi.parquet.hadoop.AvroParquetHDFSRecordWriter.close(Avr > > oP > > arquetHDFSRecordWriter.java:49) > > > at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:534) > > > at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:466) > > > at > > > > > org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$ > > 0( > > AbstractPutHDFSRecord.java:326) > > > at > > > > > org.apache.nifi.controller.repository.StandardProcessSession.read(St > > an > > dardProcessSession.java:2466) > > > at > > > > > org.apache.nifi.controller.repository.StandardProcessSession.read(St > > an > > dardProcessSession.java:2434) > > > at > > > > > org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTri > > gg > > er$1(AbstractPutHDFSRecord.java:303) > > > at java.security.AccessController.doPrivileged(Native Method) > > > at javax.security.auth.Subject.doAs(Subject.java:360) > > > at > > > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInform > > at > > ion.java:1822) > > > at > > > > > org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(Ab > > st > > ractPutHDFSRecord.java:271) > > > at > > > > > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProces > > so > > r.java:27) > > > at > > > > > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardP > > ro > > cessorNode.java:1202) > > > at > > > > > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableT > > as > > k.java:214) > > > at > > > > > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.r > > un > > (TimerDrivenSchedulingAgent.java:103) > > > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > > > at > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > at > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > > > at > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask > > .a > > ccess$301(ScheduledThreadPoolExecutor.java:180) > > > at > > > > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask > > .r > > un(ScheduledThreadPoolExecutor.java:294) > > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor > > .j > > ava:1149) > > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > > java:624) > > > at java.lang.Thread.run(Thread.java:748) > > > > > > obase > > > TEL: +90216 527 30 00 > > > FAX: +90216 527 31 11 > > > [http://www.obase.com/images/signature/home.png]<http://www.obase. > > > co > > > m> [ http://www.obase.com/images/signature/facebook.png] < > > > https://www.facebook.com/obasesocial> [ > > > http://www.obase.com/images/signature/twitter.png] < > > > https://twitter.com/obasesocial> [ > > > http://www.obase.com/images/signature/linkedin.png] < > > > https://tr.linkedin.com/in/obase> > > > [http://www.obase.com/images/signature/obase25tr.png]<http://www.o > > > ba > > > se > > > .com > > > > > > > > > > Bu elektronik posta ve onunla iletilen b?t?n dosyalar sadece > > > g?ndericisi tarafindan almasi amaclanan yetkili gercek ya da t?zel > > > kisinin kullanimi icindir. Eger s?z konusu yetkili alici > > > degilseniz bu elektronik postanin icerigini aciklamaniz, > > > kopyalamaniz, y?nlendirmeniz ve kullanmaniz kesinlikle yasaktir ve > > > bu elektronik > > postayi derhal silmeniz gerekmektedir. > > > OBASE bu mesajin icerdigi bilgilerin do?rulu?u veya eksiksiz > > > oldugu konusunda herhangi bir garanti vermemektedir. Bu nedenle bu > > > bilgilerin ne sekilde olursa olsun iceriginden, iletilmesinden, > > > alinmasindan ve saklanmasindan sorumlu degildir. Bu mesajdaki > > > g?r?sler yalnizca g?nderen kisiye aittir ve OBASE g?r?slerini > yansitmayabilir. > > > > > > Bu e-posta bilinen b?t?n bilgisayar vir?slerine karsi taranmistir. > > > > > > This e-mail and any files transmitted with it are confidential and > > > intended solely for the use of the individual or entity to whom > > > they are addressed. If you are not the intended recipient you are > > > hereby notified that any dissemination, forwarding, copying or use > > > of any of the information is strictly prohibited, and the e-mail > > > should immediately be deleted. OBASE makes no warranty as to the > > > accuracy or completeness of any information contained in this > > > message and hereby excludes any liability of any kind for the > > > information contained therein or for the information transmission, > > > recepxion, storage or use of such in any way whatsoever. The > > > opinions expressed in this message belong to sender alone and may > > > not necessarily reflect the opinions of > > OBASE. > > > > > > This e-mail has been scanned for all known computer viruses. > > > > > >
