Hi, Is there something you changed on how the Snappy libraries are loaded in the NiFi JVM? I'm not familiar with this specifically but you may need more recent versions of the native libraries. Also, anything related to a snappy-java lib you added somewhere? /lib? custom processor? etc.
I believe it may be related to https://github.com/apache/nifi/pull/4601 where we're adding snappy-java and maybe there is a version conflict there. Pierre Le mer. 15 sept. 2021 à 12:38, Bilal Bektas <bilal.bek...@obase.com> a écrit : > Hi Dev Team, > > To be able to detect the problem, is there anything you expect from us or > is there anything we can help with > > Thank you in advance, > > --Bilal > > > -----Original Message----- > From: Bilal Bektas <bilal.bek...@obase.com> > Sent: 1 Eylül 2021 Çarşamba 12:43 > To: dev@nifi.apache.org > Subject: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0) > > Hi Dev Team, > > I have tested different compression types which is a feature of PutParquet > and ConvertAvroToParquet Processors on different NiFi versions. > > To avoid confusion, i will give a summary information: > > * Compression types (UNCOMPRESSED, GZIP, SNAPPY) of PutParquet Processor > works correctly on NiFi 1.12.1 and 1.13.2 > * Compression types (UNCOMPRESSED, GZIP) of PutParquet Processor works > correctly on NiFi 1.14.0; SNAPPY gives an error in the previous email > > * Compression types (UNCOMPRESSED, GZIP, SNAPPY) of ConvertAvroToParquet > Processor works correctly on NiFi 1.12.1, 1.13.2 and 1.14.0. > > > PutParquet - Properties: > * Hadoop Configuration Resources: File locations > * Kerberos Credentials Service: Keytab service > * Record Reader: AvroReader Service (Embedded Avro Schema) > * Overwrite Files: True > * Compression Type: SNAPPY > * Other Properties: Default > > Error Log (nifi-app.log): > Class org.xerial.snappy.SnappyNative does not implement the requested > interface org.xerial.snappy.SnappyApi > > > Could you help in order to resolve this situation? > > Thank you in advance, > > --Bilal > > > > -----Original Message----- > From: Bilal Bektas <bilal.bek...@obase.com> > Sent: 3 Ağustos 2021 Salı 14:55 > To: dev@nifi.apache.org > Subject: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0) > > Hi Dev Team, > > SNAPPY compression feature of PutParquet Processor works correctly on NiFi > 1.12.1 but NiFi 1.14.0 gives the following error. > > Could you help in order to resolve this situation? > > Thank you in advance, > > --Bilal > > > PutParquet - Config: > Hadoop Configuration Resources > > File locations > > Kerberos Credentials Service > > Keytab service > > Kerberos Relogin Period > > 4 Hours > > Record Reader > > AvroReader Service (Embedded Avro Schema) > > Compression Type > > SNAPPY > > > > Error Log (nifi-app.log): > 2021-08-03 14:13:01,955 ERROR [Timer-Driven Process Thread-12] > o.a.nifi.processors.parquet.PutParquet > PutParquet[id=6caab337-68e8-3834-b64a-1d2cbd93aba8] Failed to write due to > java.lang.IncompatibleClassChangeError: Class > org.xerial.snappy.SnappyNative does not implement the requested interface > org.xerial.snappy.SnappyApi: java.lang.IncompatibleClassChangeError: Class > org.xerial.snappy.SnappyNative does not implement the requested interface > org.xerial.snappy.SnappyApi > java.lang.IncompatibleClassChangeError: Class > org.xerial.snappy.SnappyNative does not implement the requested interface > org.xerial.snappy.SnappyApi > at org.xerial.snappy.Snappy.maxCompressedLength(Snappy.java:380) > at > org.apache.parquet.hadoop.codec.SnappyCompressor.compress(SnappyCompressor.java:67) > at > org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:81) > at > org.apache.hadoop.io.compress.CompressorStream.finish(CompressorStream.java:92) > at > org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.compress(CodecFactory.java:167) > at > org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:168) > at > org.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:59) > at > org.apache.parquet.column.impl.ColumnWriterBase.writePage(ColumnWriterBase.java:387) > at > org.apache.parquet.column.impl.ColumnWriteStoreBase.flush(ColumnWriteStoreBase.java:186) > at > org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:29) > at > org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:185) > at > org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:124) > at > org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:319) > at > org.apache.nifi.parquet.hadoop.AvroParquetHDFSRecordWriter.close(AvroParquetHDFSRecordWriter.java:49) > at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:534) > at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:466) > at > org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:326) > at > org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2466) > at > org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2434) > at > org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:303) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:360) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1822) > at > org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:271) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > obase > TEL: +90216 527 30 00 > FAX: +90216 527 31 11 > [http://www.obase.com/images/signature/home.png]<http://www.obase.com> [ > http://www.obase.com/images/signature/facebook.png] < > https://www.facebook.com/obasesocial> [ > http://www.obase.com/images/signature/twitter.png] < > https://twitter.com/obasesocial> [ > http://www.obase.com/images/signature/linkedin.png] < > https://tr.linkedin.com/in/obase> > [http://www.obase.com/images/signature/obase25tr.png]<http://www.obase.com > > > > Bu elektronik posta ve onunla iletilen b?t?n dosyalar sadece g?ndericisi > tarafindan almasi amaclanan yetkili gercek ya da t?zel kisinin kullanimi > icindir. Eger s?z konusu yetkili alici degilseniz bu elektronik postanin > icerigini aciklamaniz, kopyalamaniz, y?nlendirmeniz ve kullanmaniz > kesinlikle yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir. > OBASE bu mesajin icerdigi bilgilerin do?rulu?u veya eksiksiz oldugu > konusunda herhangi bir garanti vermemektedir. Bu nedenle bu bilgilerin ne > sekilde olursa olsun iceriginden, iletilmesinden, alinmasindan ve > saklanmasindan sorumlu degildir. Bu mesajdaki g?r?sler yalnizca g?nderen > kisiye aittir ve OBASE g?r?slerini yansitmayabilir. > > Bu e-posta bilinen b?t?n bilgisayar vir?slerine karsi taranmistir. > > This e-mail and any files transmitted with it are confidential and > intended solely for the use of the individual or entity to whom they are > addressed. If you are not the intended recipient you are hereby notified > that any dissemination, forwarding, copying or use of any of the > information is strictly prohibited, and the e-mail should immediately be > deleted. OBASE makes no warranty as to the accuracy or completeness of any > information contained in this message and hereby excludes any liability of > any kind for the information contained therein or for the information > transmission, recepxion, storage or use of such in any way whatsoever. The > opinions expressed in this message belong to sender alone and may not > necessarily reflect the opinions of OBASE. > > This e-mail has been scanned for all known computer viruses. >
