[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866613#comment-16866613 ] ASF subversion and git services commented on KYLIN-4015: Commit b0ae00315e1fe3d9301581387668fde3ec0efd6f in kylin's branch refs/heads/master-hadoop3.1 from shqmh [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=b0ae003 ] KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files to solve the problem that MR engine run failed because the ".dci" file is not a Sequence file. > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Fix For: v2.6.3 > > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866614#comment-16866614 ] ASF subversion and git services commented on KYLIN-4015: Commit f19f16aad334fd65f9946dd51d369497a15cb9e4 in kylin's branch refs/heads/master-hadoop3.1 from shqmh [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=f19f16a ] KYLIN-4015 change uhc path filter .dci to FactDistinctColumnsReducer.DIMENSION_COL_INFO_FILE_POSTFIX > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Fix For: v2.6.3 > > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862648#comment-16862648 ] ASF subversion and git services commented on KYLIN-4015: Commit fb486745ba7ca024b117ba2cc7ec95f64c41f401 in kylin's branch refs/heads/2.6.x from shqmh [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=fb48674 ] KYLIN-4015 change uhc path filter .dci to FactDistinctColumnsReducer.DIMENSION_COL_INFO_FILE_POSTFIX > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Fix For: v2.6.3 > > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862647#comment-16862647 ] ASF subversion and git services commented on KYLIN-4015: Commit fc3e6e52140b8e724043e7a0134d06aa4c9e34ca in kylin's branch refs/heads/2.6.x from shqmh [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=fc3e6e5 ] KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files to solve the problem that MR engine run failed because the ".dci" file is not a Sequence file. > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Fix For: v2.6.3 > > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862646#comment-16862646 ] ASF subversion and git services commented on KYLIN-4015: Commit f19f16aad334fd65f9946dd51d369497a15cb9e4 in kylin's branch refs/heads/master from shqmh [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=f19f16a ] KYLIN-4015 change uhc path filter .dci to FactDistinctColumnsReducer.DIMENSION_COL_INFO_FILE_POSTFIX > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Fix For: v2.6.3 > > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862644#comment-16862644 ] ASF GitHub Bot commented on KYLIN-4015: --- shaofengshi commented on pull request #661: KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files URL: https://github.com/apache/kylin/pull/661 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Fix For: v2.6.3 > > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862645#comment-16862645 ] ASF subversion and git services commented on KYLIN-4015: Commit b0ae00315e1fe3d9301581387668fde3ec0efd6f in kylin's branch refs/heads/master from shqmh [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=b0ae003 ] KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files to solve the problem that MR engine run failed because the ".dci" file is not a Sequence file. > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Fix For: v2.6.3 > > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850291#comment-16850291 ] Shaofeng SHI commented on KYLIN-4015: - I see, the ".dci" file was generated for keeping each dimension's min/max info, which is introduced in KYLIN-3370, the files should be excluded. > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4015) Kylin build cube error at the "Build UHC Dictionary" step
[ https://issues.apache.org/jira/browse/KYLIN-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848469#comment-16848469 ] ASF GitHub Bot commented on KYLIN-4015: --- zhaojintaozhao commented on pull request #661: KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files URL: https://github.com/apache/kylin/pull/661 KYLIN-4015 "Build UHC Dictionary" step filter ".dci" files to solve the problem that MR engine run failed because the ".dci" file is not a sequence file. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Kylin build cube error at the "Build UHC Dictionary" step > - > > Key: KYLIN-4015 > URL: https://issues.apache.org/jira/browse/KYLIN-4015 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.2 > Environment: Fusion Insight >Reporter: zhao jintao >Assignee: zhao jintao >Priority: Major > Labels: easyfix > Original Estimate: 168h > Remaining Estimate: 168h > > Hi All: > We know, kylin builds dimension dictionary in kylin job client. But if a cube > has uhc dimensions, it will cost much more CPU and memory resources. Kylin > provides the ability to build uhc dictionary using the MR engine to reduce > the resource consumption of the build engine. > But I find that the "Build UHC Dictionary" step build error. This step run > using MR engine. This is the error info from yarn: > org.apache.hadoop.mapred.YarnChild: Exception running child : > java.io.IOException: > hdfs://hacluster/xxx.../xxx/fact_distinct_columns/xxx/FIELD_NAME.dic-r-1 > not a SequenceFile. > at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:) > at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:) > The reason of this problem is that the "Extract Fact Table Distinct " step > output two type of files:".dci" and ".rldict"; but the ".dci" file is not a > sequence file, so the "Build UHC Dictionary" step should filter ".dci" file > when run with MR engine. > I resolve this problem and will summit my code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
