Hi Dev Team,
To be able to detect the problem, is there anything you expect from us or is
there anything we can help with
Thank you in advance,
--Bilal
-----Original Message-----
From: Bilal Bektas <bilal.bek...@obase.com>
Sent: 1 Eylül 2021 Çarşamba 12:43
To: dev@nifi.apache.org
Subject: [SPAM] RE: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0)
Hi Dev Team,
I have tested different compression types which is a feature of PutParquet and
ConvertAvroToParquet Processors on different NiFi versions.
To avoid confusion, i will give a summary information:
* Compression types (UNCOMPRESSED, GZIP, SNAPPY) of PutParquet Processor works
correctly on NiFi 1.12.1 and 1.13.2
* Compression types (UNCOMPRESSED, GZIP) of PutParquet Processor works
correctly on NiFi 1.14.0; SNAPPY gives an error in the previous email
* Compression types (UNCOMPRESSED, GZIP, SNAPPY) of ConvertAvroToParquet
Processor works correctly on NiFi 1.12.1, 1.13.2 and 1.14.0.
PutParquet - Properties:
* Hadoop Configuration Resources: File locations
* Kerberos Credentials Service: Keytab service
* Record Reader: AvroReader Service (Embedded Avro Schema)
* Overwrite Files: True
* Compression Type: SNAPPY
* Other Properties: Default
Error Log (nifi-app.log):
Class org.xerial.snappy.SnappyNative does not implement the requested interface
org.xerial.snappy.SnappyApi
Could you help in order to resolve this situation?
Thank you in advance,
--Bilal
-----Original Message-----
From: Bilal Bektas <bilal.bek...@obase.com>
Sent: 3 Ağustos 2021 Salı 14:55
To: dev@nifi.apache.org
Subject: PutParquet - Compression Type: SNAPPY (NiFi 1.14.0)
Hi Dev Team,
SNAPPY compression feature of PutParquet Processor works correctly on NiFi
1.12.1 but NiFi 1.14.0 gives the following error.
Could you help in order to resolve this situation?
Thank you in advance,
--Bilal
PutParquet - Config:
Hadoop Configuration Resources
File locations
Kerberos Credentials Service
Keytab service
Kerberos Relogin Period
4 Hours
Record Reader
AvroReader Service (Embedded Avro Schema)
Compression Type
SNAPPY
Error Log (nifi-app.log):
2021-08-03 14:13:01,955 ERROR [Timer-Driven Process Thread-12]
o.a.nifi.processors.parquet.PutParquet
PutParquet[id=6caab337-68e8-3834-b64a-1d2cbd93aba8] Failed to write due to
java.lang.IncompatibleClassChangeError: Class org.xerial.snappy.SnappyNative
does not implement the requested interface org.xerial.snappy.SnappyApi:
java.lang.IncompatibleClassChangeError: Class org.xerial.snappy.SnappyNative
does not implement the requested interface org.xerial.snappy.SnappyApi
java.lang.IncompatibleClassChangeError: Class org.xerial.snappy.SnappyNative
does not implement the requested interface org.xerial.snappy.SnappyApi
at org.xerial.snappy.Snappy.maxCompressedLength(Snappy.java:380)
at
org.apache.parquet.hadoop.codec.SnappyCompressor.compress(SnappyCompressor.java:67)
at
org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:81)
at
org.apache.hadoop.io.compress.CompressorStream.finish(CompressorStream.java:92)
at
org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.compress(CodecFactory.java:167)
at
org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:168)
at
org.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:59)
at
org.apache.parquet.column.impl.ColumnWriterBase.writePage(ColumnWriterBase.java:387)
at
org.apache.parquet.column.impl.ColumnWriteStoreBase.flush(ColumnWriteStoreBase.java:186)
at
org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:29)
at
org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:185)
at
org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:124)
at org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:319)
at
org.apache.nifi.parquet.hadoop.AvroParquetHDFSRecordWriter.close(AvroParquetHDFSRecordWriter.java:49)
at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:534)
at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:466)
at
org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:326)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2466)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2434)
at
org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:303)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1822)
at
org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:271)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1202)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
obase
TEL: +90216 527 30 00
FAX: +90216 527 31 11
[http://www.obase.com/images/signature/home.png]<http://www.obase.com>
[http://www.obase.com/images/signature/facebook.png]
<https://www.facebook.com/obasesocial>
[http://www.obase.com/images/signature/twitter.png]
<https://twitter.com/obasesocial>
[http://www.obase.com/images/signature/linkedin.png]
<https://tr.linkedin.com/in/obase>
[http://www.obase.com/images/signature/obase25tr.png]<http://www.obase.com>
Bu elektronik posta ve onunla iletilen b?t?n dosyalar sadece g?ndericisi
tarafindan almasi amaclanan yetkili gercek ya da t?zel kisinin kullanimi
icindir. Eger s?z konusu yetkili alici degilseniz bu elektronik postanin
icerigini aciklamaniz, kopyalamaniz, y?nlendirmeniz ve kullanmaniz kesinlikle
yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir. OBASE bu
mesajin icerdigi bilgilerin do?rulu?u veya eksiksiz oldugu konusunda herhangi
bir garanti vermemektedir. Bu nedenle bu bilgilerin ne sekilde olursa olsun
iceriginden, iletilmesinden, alinmasindan ve saklanmasindan sorumlu degildir.
Bu mesajdaki g?r?sler yalnizca g?nderen kisiye aittir ve OBASE g?r?slerini
yansitmayabilir.
Bu e-posta bilinen b?t?n bilgisayar vir?slerine karsi taranmistir.
This e-mail and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed. If
you are not the intended recipient you are hereby notified that any
dissemination, forwarding, copying or use of any of the information is strictly
prohibited, and the e-mail should immediately be deleted. OBASE makes no
warranty as to the accuracy or completeness of any information contained in
this message and hereby excludes any liability of any kind for the information
contained therein or for the information transmission, recepxion, storage or
use of such in any way whatsoever. The opinions expressed in this message
belong to sender alone and may not necessarily reflect the opinions of OBASE.
This e-mail has been scanned for all known computer viruses.
