[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
[ https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004079#comment-17004079 ] zhaoguoxian commented on KYLIN-2654: I encountered the same problem. Then your yarn environment is cdh and your 'mapreduce.job.map.output.collector.class' param is org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator. you can not use your Custom writable type or comparator,example,kylin`s org.apache.kylin.engine.mr.steps.SelfDefineSortableKey. my solution was by setting the param mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer. btw. my environment was cdh5.5.2 > Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8 > - > > Key: KYLIN-2654 > URL: https://issues.apache.org/jira/browse/KYLIN-2654 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: Shao Feng Shi >Assignee: Dong Li >Priority: Major > > From community user qiumingming: > When I use kylin 2.0 to build cube, I got a error at Extract Fact Table > Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4. > Below is my error info: > {code} > Error: java.io.IOException: Initialization of all the collectors failed. > Error in last collector was :Cannot find serializer for > org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at > org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at > org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: > java.io.IOException: Cannot find serializer for > org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at > org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147) > at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
[ https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045925#comment-16045925 ] liyang commented on KYLIN-2654: --- This is where others can help little. Only you know your env the best. > Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8 > - > > Key: KYLIN-2654 > URL: https://issues.apache.org/jira/browse/KYLIN-2654 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: Shaofeng SHI >Assignee: Dong Li > > From community user qiumingming: > When I use kylin 2.0 to build cube, I got a error at Extract Fact Table > Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4. > Below is my error info: > {code} > Error: java.io.IOException: Initialization of all the collectors failed. > Error in last collector was :Cannot find serializer for > org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at > org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at > org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: > java.io.IOException: Cannot find serializer for > org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at > org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147) > at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
[ https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045924#comment-16045924 ] liyang commented on KYLIN-2654: --- Seems you've compiled Kylin from source. Then it might be the way of compiling, like the Hadoop version you compiled against, the javac version, the target code level etc.. > Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8 > - > > Key: KYLIN-2654 > URL: https://issues.apache.org/jira/browse/KYLIN-2654 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: Shaofeng SHI >Assignee: Dong Li > > From community user qiumingming: > When I use kylin 2.0 to build cube, I got a error at Extract Fact Table > Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4. > Below is my error info: > {code} > Error: java.io.IOException: Initialization of all the collectors failed. > Error in last collector was :Cannot find serializer for > org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at > org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at > org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: > java.io.IOException: Cannot find serializer for > org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at > org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147) > at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
[ https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035029#comment-16035029 ] qiumingming commented on KYLIN-2654: By the way, is there any specific configuration to make SelfDefineSortableKey work, like register serializer for SelfDefineSortableKey explicitly. And Have you modified the pom.xml in your test, like change some versions ? > Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8 > - > > Key: KYLIN-2654 > URL: https://issues.apache.org/jira/browse/KYLIN-2654 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: Shaofeng SHI >Assignee: Dong Li > > From community user qiumingming: > When I use kylin 2.0 to build cube, I got a error at Extract Fact Table > Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4. > Below is my error info: > {code} > Error: java.io.IOException: Initialization of all the collectors failed. > Error in last collector was :Cannot find serializer for > org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at > org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at > org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: > java.io.IOException: Cannot find serializer for > org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at > org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147) > at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
[ https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035017#comment-16035017 ] qiumingming commented on KYLIN-2654: Yeah, I think so. Maybe my environment is so specific. And I solved this problem by backing SelfDefineSortableKey to Text as old version done. I want to know is this will involve any bug ? I am not very sure about that. > Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8 > - > > Key: KYLIN-2654 > URL: https://issues.apache.org/jira/browse/KYLIN-2654 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: Shaofeng SHI >Assignee: Dong Li > > From community user qiumingming: > When I use kylin 2.0 to build cube, I got a error at Extract Fact Table > Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4. > Below is my error info: > {code} > Error: java.io.IOException: Initialization of all the collectors failed. > Error in last collector was :Cannot find serializer for > org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at > org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at > org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: > java.io.IOException: Cannot find serializer for > org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at > org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147) > at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
[ https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034860#comment-16034860 ] liyang commented on KYLIN-2654: --- >From source code, the cause should be {{SelfDefineSortableKey}} is not a >{{Writeable}}. However {{SelfDefineSortableKey}} does implements >{{WritableComparable}}. Doubt this issue is very specific to the environment >under test. > Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8 > - > > Key: KYLIN-2654 > URL: https://issues.apache.org/jira/browse/KYLIN-2654 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: Shaofeng SHI >Assignee: Dong Li > > From community user qiumingming: > When I use kylin 2.0 to build cube, I got a error at Extract Fact Table > Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4. > Below is my error info: > {code} > Error: java.io.IOException: Initialization of all the collectors failed. > Error in last collector was :Cannot find serializer for > org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at > org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at > org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:422) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: > java.io.IOException: Cannot find serializer for > org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at > org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147) > at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)