[
https://issues.apache.org/jira/browse/KYLIN-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenwen updated KYLIN-3508:
---------------------------
Environment:
hadoop 2.7.2
hbase 1.2.5
hive 1.2.2
kylin-2.4.0-bin-hbase1x
kafka_2.10-0.10.2.2
centos 7
was:
hadoop 2.7.2
hbase 1.2.5
hive 1.2.2
kylin-2.4.0-bin-hbase1x
kafka_2.11-2.0.0
centos 7
Description:
I have a kafka topic using lz4 compression algorithm, then I created a cube in
kylin to consume this topic, it will report the following error, is my
configuration wrong? I switched to the other algorithm gzip, and snappy works
fine.
Error: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4Exception at
java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357) at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:264) at
org.apache.kafka.common.record.MemoryRecordsBuilder$4.get(MemoryRecordsBuilder.java:82)
at
org.apache.kafka.common.record.MemoryRecordsBuilder$MemoizingConstructorSupplier.get(MemoryRecordsBuilder.java:489)
at
org.apache.kafka.common.record.MemoryRecordsBuilder.wrapForInput(MemoryRecordsBuilder.java:455)
at
org.apache.kafka.common.record.RecordsIterator$DeepRecordsIterator.<init>(RecordsIterator.java:157)
at
org.apache.kafka.common.record.RecordsIterator.makeNext(RecordsIterator.java:81)
at
org.apache.kafka.common.record.RecordsIterator.makeNext(RecordsIterator.java:33)
at
org.apache.kafka.common.utils.AbstractIterator.maybeComputeNext(AbstractIterator.java:79)
at
org.apache.kafka.common.utils.AbstractIterator.hasNext(AbstractIterator.java:45)
at
org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:787)
at
org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:482)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1062)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:996) at
org.apache.kylin.source.kafka.hadoop.KafkaInputRecordReader.nextKeyValue(KafkaInputRecordReader.java:119)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
at
org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Container
killed by the ApplicationMaster. Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
was:
I have a kafka topic using lz4 compression algorithm, then I created a cube in
kylin to consume this topic, it will report the following error, is my
configuration wrong? I switched to the other algorithm gzip, and snappy works
fine.
Error: org.apache.kafka.common.KafkaException: Received exception when fetching
the next record from kylin.log.error-3. If needed, please seek past the record
to continue consumption. at
org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.fetchRecords(Fetcher.java:1201)
at
org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.access$1500(Fetcher.java:1035)
at
org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:544)
at
org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:505)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1259)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1187)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
at
org.apache.kylin.source.kafka.hadoop.KafkaInputRecordReader.nextKeyValue(KafkaInputRecordReader.java:119)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
at
org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
org.apache.kafka.common.KafkaException: java.lang.NoClassDefFoundError:
net/jpountz/lz4/LZ4Exception at
org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:113)
at
org.apache.kafka.common.record.AbstractLegacyRecordBatch$DeepRecordsIterator.<init>(AbstractLegacyRecordBatch.java:330)
at
org.apache.kafka.common.record.AbstractLegacyRecordBatch$DeepRecordsIterator.<init>(AbstractLegacyRecordBatch.java:310)
at
org.apache.kafka.common.record.AbstractLegacyRecordBatch.iterator(AbstractLegacyRecordBatch.java:232)
at
org.apache.kafka.common.record.AbstractLegacyRecordBatch.streamingIterator(AbstractLegacyRecordBatch.java:263)
at
org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.nextFetchedRecord(Fetcher.java:1144)
at
org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.fetchRecords(Fetcher.java:1181)
... 18 more Caused by: java.lang.NoClassDefFoundError:
net/jpountz/lz4/LZ4Exception at
org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:110)
... 24 more Caused by: java.lang.ClassNotFoundException:
net.jpountz.lz4.LZ4Exception at
java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> kylin cube kafka streaming lz4 exception
> ----------------------------------------
>
> Key: KYLIN-3508
> URL: https://issues.apache.org/jira/browse/KYLIN-3508
> Project: Kylin
> Issue Type: Bug
> Components: Streaming
> Affects Versions: v2.4.0
> Environment: hadoop 2.7.2
> hbase 1.2.5
> hive 1.2.2
> kylin-2.4.0-bin-hbase1x
> kafka_2.10-0.10.2.2
> centos 7
> Reporter: chenwen
> Priority: Major
>
> I have a kafka topic using lz4 compression algorithm, then I created a cube
> in kylin to consume this topic, it will report the following error, is my
> configuration wrong? I switched to the other algorithm gzip, and snappy works
> fine.
> Error: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4Exception at
> java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) at
> java.lang.Class.forName0(Native Method) at
> java.lang.Class.forName(Class.java:264) at
> org.apache.kafka.common.record.MemoryRecordsBuilder$4.get(MemoryRecordsBuilder.java:82)
> at
> org.apache.kafka.common.record.MemoryRecordsBuilder$MemoizingConstructorSupplier.get(MemoryRecordsBuilder.java:489)
> at
> org.apache.kafka.common.record.MemoryRecordsBuilder.wrapForInput(MemoryRecordsBuilder.java:455)
> at
> org.apache.kafka.common.record.RecordsIterator$DeepRecordsIterator.<init>(RecordsIterator.java:157)
> at
> org.apache.kafka.common.record.RecordsIterator.makeNext(RecordsIterator.java:81)
> at
> org.apache.kafka.common.record.RecordsIterator.makeNext(RecordsIterator.java:33)
> at
> org.apache.kafka.common.utils.AbstractIterator.maybeComputeNext(AbstractIterator.java:79)
> at
> org.apache.kafka.common.utils.AbstractIterator.hasNext(AbstractIterator.java:45)
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:787)
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:482)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1062)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:996)
> at
> org.apache.kylin.source.kafka.hadoop.KafkaInputRecordReader.nextKeyValue(KafkaInputRecordReader.java:119)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Container
> killed by the ApplicationMaster. Container killed on request. Exit code is
> 143 Container exited with a non-zero exit code 143
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
