[jira] [Commented] (KYLIN-3980) Cube planner cuboid id error when dimensions too many
[ https://issues.apache.org/jira/browse/KYLIN-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825736#comment-16825736 ] Pan, Julian commented on KYLIN-3980: The fix is change the type of cuboid from Number to String during the API. And will create a pull request to fix it. > Cube planner cuboid id error when dimensions too many > - > > Key: KYLIN-3980 > URL: https://issues.apache.org/jira/browse/KYLIN-3980 > Project: Kylin > Issue Type: Bug >Reporter: Pan, Julian >Assignee: Pan, Julian >Priority: Major > > Cube planner cuboid id error when dimensions beyond too many. > The issue due to cuboid is the Number and Number in javascript max value is > 2^53-1. > [https://www.ecma-international.org/ecma-262/5.1/#sec-8.5] > If the cuboid id is greater than Number.MAX_SAFE_INTEGER(9007199254740991), > then it will get error cuboid in page, which impact optimize job. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3980) Cube planner cuboid id error when dimensions too many
Pan, Julian created KYLIN-3980: -- Summary: Cube planner cuboid id error when dimensions too many Key: KYLIN-3980 URL: https://issues.apache.org/jira/browse/KYLIN-3980 Project: Kylin Issue Type: Bug Reporter: Pan, Julian Assignee: Pan, Julian Cube planner cuboid id error when dimensions beyond too many. The issue due to the Number in javascript max value is 2^53-1. [https://www.ecma-international.org/ecma-262/5.1/#sec-8.5] If the cuboid id is greater than Number.MAX_SAFE_INTEGER(9007199254740991), then it will get error cuboid in page, which impact optimize job. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3980) Cube planner cuboid id error when dimensions too many
[ https://issues.apache.org/jira/browse/KYLIN-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pan, Julian updated KYLIN-3980: --- Description: Cube planner cuboid id error when dimensions beyond too many. The issue due to cuboid is the Number and Number in javascript max value is 2^53-1. [https://www.ecma-international.org/ecma-262/5.1/#sec-8.5] If the cuboid id is greater than Number.MAX_SAFE_INTEGER(9007199254740991), then it will get error cuboid in page, which impact optimize job. was: Cube planner cuboid id error when dimensions beyond too many. The issue due to the Number in javascript max value is 2^53-1. [https://www.ecma-international.org/ecma-262/5.1/#sec-8.5] If the cuboid id is greater than Number.MAX_SAFE_INTEGER(9007199254740991), then it will get error cuboid in page, which impact optimize job. > Cube planner cuboid id error when dimensions too many > - > > Key: KYLIN-3980 > URL: https://issues.apache.org/jira/browse/KYLIN-3980 > Project: Kylin > Issue Type: Bug >Reporter: Pan, Julian >Assignee: Pan, Julian >Priority: Major > > Cube planner cuboid id error when dimensions beyond too many. > The issue due to cuboid is the Number and Number in javascript max value is > 2^53-1. > [https://www.ecma-international.org/ecma-262/5.1/#sec-8.5] > If the cuboid id is greater than Number.MAX_SAFE_INTEGER(9007199254740991), > then it will get error cuboid in page, which impact optimize job. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] coveralls edited a comment on issue #602: KYLIN-3939 add BOM character,slove the bug that it shows Chinese garb…
coveralls edited a comment on issue #602: KYLIN-3939 add BOM character,slove the bug that it shows Chinese garb… URL: https://github.com/apache/kylin/pull/602#issuecomment-482995526 ## Pull Request Test Coverage Report for [Build 4421](https://coveralls.io/builds/23012977) * **0** of **6** **(0.0%)** changed or added relevant lines in **1** file are covered. * **813** unchanged lines in **14** files lost coverage. * Overall coverage increased (+**0.06%**) to **27.815%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [server-base/src/main/java/org/apache/kylin/rest/controller/QueryController.java](https://coveralls.io/builds/23012977/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fcontroller%2FQueryController.java#L87) | 0 | 6 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [source-jdbc/src/main/java/org/apache/kylin/source/jdbc/extensible/JdbcHiveInputBase.java](https://coveralls.io/builds/23012977/source?filename=source-jdbc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fjdbc%2Fextensible%2FJdbcHiveInputBase.java#L115) | 1 | 94.0% | | [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/23012977/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% | | [server-base/src/main/java/org/apache/kylin/rest/controller/JobController.java](https://coveralls.io/builds/23012977/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fcontroller%2FJobController.java#L187) | 6 | 0.0% | | [source-jdbc/src/main/java/org/apache/kylin/source/jdbc/JdbcHiveInputBase.java](https://coveralls.io/builds/23012977/source?filename=source-jdbc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fjdbc%2FJdbcHiveInputBase.java#L187) | 7 | 12.5% | | [server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java](https://coveralls.io/builds/23012977/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fsecurity%2FKylinAuthenticationProvider.java#L118) | 8 | 0.0% | | [stream-source-kafka/src/main/java/org/apache/kylin/stream/source/kafka/TimedJsonStreamParser.java](https://coveralls.io/builds/23012977/source?filename=stream-source-kafka%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fsource%2Fkafka%2FTimedJsonStreamParser.java#L75) | 9 | 86.3% | | [source-hive/src/main/java/org/apache/kylin/source/hive/GarbageCollectionStep.java](https://coveralls.io/builds/23012977/source?filename=source-hive%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fhive%2FGarbageCollectionStep.java#L42) | 20 | 0.0% | | [core-job/src/main/java/org/apache/kylin/job/util/FlatTableSqlQuoteUtils.java](https://coveralls.io/builds/23012977/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Futil%2FFlatTableSqlQuoteUtils.java#L52) | 41 | 27.59% | | [core-metadata/src/main/java/org/apache/kylin/metadata/model/FunctionDesc.java](https://coveralls.io/builds/23012977/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FFunctionDesc.java#L190) | 42 | 28.28% | | [core-job/src/main/java/org/apache/kylin/job/execution/ExecutableManager.java](https://coveralls.io/builds/23012977/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fexecution%2FExecutableManager.java#L379) | 52 | 39.02% | | Totals | [](https://coveralls.io/builds/23012977) | | :-- | --: | | Change from base [Build 4386](https://coveralls.io/builds/22784891): | 0.06% | | Covered Lines: | 22755 | | Relevant Lines: | 81808 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3979) Job Fetcher caught a exception java.lang.IllegalArgumentException: there is no related output for job
[
https://issues.apache.org/jira/browse/KYLIN-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825701#comment-16825701
]
tangyupeng commented on KYLIN-3979:
---
By the way, how to fix this problem.
# backup metadata
# Under backuped metadata folder, find files in execute folder but not in
execute_output folder. These files are the inconsistent Job IDs.
# Remove the job from the metadata through command. And then reload meta data,
problem solved.
{code:java}
metastore.sh remove /execute/jobId{code}
> Job Fetcher caught a exception java.lang.IllegalArgumentException: there is
> no related output for job
> -
>
> Key: KYLIN-3979
> URL: https://issues.apache.org/jira/browse/KYLIN-3979
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.6.1
>Reporter: tangyupeng
>Priority: Minor
>
> After Kylin restart, it failed to load job information through the "Monitor"
> tab. Use command 'metastore.sh clean' seems not working, Kylin still has the
> same problem. The log is attached here:
> {code:java}
> // code placeholder
> 2019-04-24 06:33:46,750 WARN [pool-9-thread-1]
> threadpool.DefaultFetcherRunner:99 : Job Fetcher caught a exception
> java.lang.IllegalArgumentException: there is no related output for job
> id:484b1cb1-54a7-1ded-3cf6-928686557585
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
> at
> org.apache.kylin.job.execution.ExecutableManager.getOutputDigest(ExecutableManager.java:179)
> at
> org.apache.kylin.job.impl.threadpool.DefaultFetcherRunner.run(DefaultFetcherRunner.java:61)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-04-24 06:34:09,210 ERROR [http-bio-7070-exec-10]
> controller.JobController:89 :
> java.lang.NullPointerException
> at org.apache.kylin.rest.service.JobService$22.apply(JobService.java:1113)
> at org.apache.kylin.rest.service.JobService$22.apply(JobService.java:1107)
> at com.google.common.base.Predicates$AndPredicate.apply(Predicates.java:343)
> at com.google.common.collect.Iterators$7.computeNext(Iterators.java:702)
> ...
> at java.lang.Thread.run(Thread.java:748)
> 2019-04-24 06:34:09,212 ERROR [http-bio-7070-exec-10]
> controller.BasicController:63 :
> org.apache.kylin.rest.exception.InternalErrorException:
> java.lang.NullPointerException{code}
> The reason is that Job plan and Job output has consystent data. But "clean"
> command seems not clean up the unconsistent meta data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3979) Job Fetcher caught a exception java.lang.IllegalArgumentException: there is no related output for job
tangyupeng created KYLIN-3979:
-
Summary: Job Fetcher caught a exception
java.lang.IllegalArgumentException: there is no related output for job
Key: KYLIN-3979
URL: https://issues.apache.org/jira/browse/KYLIN-3979
Project: Kylin
Issue Type: Bug
Components: Metadata
Affects Versions: v2.6.1
Reporter: tangyupeng
After Kylin restart, it failed to load job information through the "Monitor"
tab. Use command 'metastore.sh clean' seems not working, Kylin still has the
same problem. The log is attached here:
{code:java}
// code placeholder
2019-04-24 06:33:46,750 WARN [pool-9-thread-1]
threadpool.DefaultFetcherRunner:99 : Job Fetcher caught a exception
java.lang.IllegalArgumentException: there is no related output for job
id:484b1cb1-54a7-1ded-3cf6-928686557585
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
at
org.apache.kylin.job.execution.ExecutableManager.getOutputDigest(ExecutableManager.java:179)
at
org.apache.kylin.job.impl.threadpool.DefaultFetcherRunner.run(DefaultFetcherRunner.java:61)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-04-24 06:34:09,210 ERROR [http-bio-7070-exec-10]
controller.JobController:89 :
java.lang.NullPointerException
at org.apache.kylin.rest.service.JobService$22.apply(JobService.java:1113)
at org.apache.kylin.rest.service.JobService$22.apply(JobService.java:1107)
at com.google.common.base.Predicates$AndPredicate.apply(Predicates.java:343)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:702)
...
at java.lang.Thread.run(Thread.java:748)
2019-04-24 06:34:09,212 ERROR [http-bio-7070-exec-10]
controller.BasicController:63 :
org.apache.kylin.rest.exception.InternalErrorException:
java.lang.NullPointerException{code}
The reason is that Job plan and Job output has consystent data. But "clean"
command seems not clean up the unconsistent meta data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3978) InternalErrorException: null with precise count distinct
Temple Zhou created KYLIN-3978: -- Summary: InternalErrorException: null with precise count distinct Key: KYLIN-3978 URL: https://issues.apache.org/jira/browse/KYLIN-3978 Project: Kylin Issue Type: Bug Reporter: Temple Zhou Assignee: Temple Zhou when kylin.query.segment-cache-enabled=true and sql with precise count distinct, the InternalErrorException: null happens sometimes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3977) Aviod mistaken deleting dicts by storage cleanup while building jobs are running
[ https://issues.apache.org/jira/browse/KYLIN-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825626#comment-16825626 ] PENG Zhengshuai edited comment on KYLIN-3977 at 4/25/19 1:42 AM: - 1.add ResourceStore.delete(respath, timestamp) interface & impl 2.add ResourceStore.updateTimestamp(respath, timestamp) interface & impl 3.update resource timestamp when reusing dict/snapshot 4.Retry avaibility for building process of dict & snapshot 5.refactor metadata cleanup by using delete with timestamp interface was (Author: peng zhengshuai): 1.add ResourceStore.delete(respath, timestamp) interface & impl 2.add ResourceStore.updateTimestamp(respath, timestamp) interface & impl 3.update resource timestamp when reusing dict/snapshot 4.Retry avaibility build process of dict & snapshot 5.refactor metadata cleanup by using delete with timestamp interface > Aviod mistaken deleting dicts by storage cleanup while building jobs are > running > > > Key: KYLIN-3977 > URL: https://issues.apache.org/jira/browse/KYLIN-3977 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Major > > When there are cube building jobs running, storage cleanup may delete some > dictionaries which the cubes are not referenced to. But the in build jobs, > those dictionaries may be reused. > If these reused dictionaries have been deleted by storage cleanup, the build > job will be failed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3977) Aviod mistaken deleting dicts by storage cleanup while building jobs are running
[ https://issues.apache.org/jira/browse/KYLIN-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825626#comment-16825626 ] PENG Zhengshuai commented on KYLIN-3977: 1.add ResourceStore.delete(respath, timestamp) interface & impl 2.add ResourceStore.updateTimestamp(respath, timestamp) interface & impl 3.update resource timestamp when reusing dict/snapshot 4.Retry avaibility build process of dict & snapshot 5.refactor metadata cleanup by using delete with timestamp interface > Aviod mistaken deleting dicts by storage cleanup while building jobs are > running > > > Key: KYLIN-3977 > URL: https://issues.apache.org/jira/browse/KYLIN-3977 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Major > > When there are cube building jobs running, storage cleanup may delete some > dictionaries which the cubes are not referenced to. But the in build jobs, > those dictionaries may be reused. > If these reused dictionaries have been deleted by storage cleanup, the build > job will be failed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] coveralls edited a comment on issue #602: KYLIN-3939 add BOM character,slove the bug that it shows Chinese garb…
coveralls edited a comment on issue #602: KYLIN-3939 add BOM character,slove the bug that it shows Chinese garb… URL: https://github.com/apache/kylin/pull/602#issuecomment-482995526 ## Pull Request Test Coverage Report for [Build 4419](https://coveralls.io/builds/22998866) * **0** of **6** **(0.0%)** changed or added relevant lines in **1** file are covered. * **815** unchanged lines in **15** files lost coverage. * Overall coverage increased (+**0.06%**) to **27.81%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [server-base/src/main/java/org/apache/kylin/rest/controller/QueryController.java](https://coveralls.io/builds/22998866/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fcontroller%2FQueryController.java#L87) | 0 | 6 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [source-jdbc/src/main/java/org/apache/kylin/source/jdbc/extensible/JdbcHiveInputBase.java](https://coveralls.io/builds/22998866/source?filename=source-jdbc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fjdbc%2Fextensible%2FJdbcHiveInputBase.java#L115) | 1 | 94.0% | | [stream-core/src/main/java/org/apache/kylin/stream/core/storage/CheckPointStore.java](https://coveralls.io/builds/22998866/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2FCheckPointStore.java#L98) | 2 | 73.74% | | [core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/22998866/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L194) | 2 | 80.23% | | [server-base/src/main/java/org/apache/kylin/rest/controller/JobController.java](https://coveralls.io/builds/22998866/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fcontroller%2FJobController.java#L187) | 6 | 0.0% | | [source-jdbc/src/main/java/org/apache/kylin/source/jdbc/JdbcHiveInputBase.java](https://coveralls.io/builds/22998866/source?filename=source-jdbc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fjdbc%2FJdbcHiveInputBase.java#L187) | 7 | 12.5% | | [server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java](https://coveralls.io/builds/22998866/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fsecurity%2FKylinAuthenticationProvider.java#L118) | 8 | 0.0% | | [stream-source-kafka/src/main/java/org/apache/kylin/stream/source/kafka/TimedJsonStreamParser.java](https://coveralls.io/builds/22998866/source?filename=stream-source-kafka%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fsource%2Fkafka%2FTimedJsonStreamParser.java#L75) | 9 | 86.3% | | [source-hive/src/main/java/org/apache/kylin/source/hive/GarbageCollectionStep.java](https://coveralls.io/builds/22998866/source?filename=source-hive%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fhive%2FGarbageCollectionStep.java#L42) | 20 | 0.0% | | [core-job/src/main/java/org/apache/kylin/job/util/FlatTableSqlQuoteUtils.java](https://coveralls.io/builds/22998866/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Futil%2FFlatTableSqlQuoteUtils.java#L52) | 41 | 27.59% | | [core-metadata/src/main/java/org/apache/kylin/metadata/model/FunctionDesc.java](https://coveralls.io/builds/22998866/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FFunctionDesc.java#L190) | 42 | 28.28% | | Totals | [](https://coveralls.io/builds/22998866) | | :-- | --: | | Change from base [Build 4386](https://coveralls.io/builds/22784891): | 0.06% | | Covered Lines: | 22751 | | Relevant Lines: | 81808 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] zhaojintaozhao commented on issue #602: KYLIN-3939 add BOM character,slove the bug that it shows Chinese garb…
zhaojintaozhao commented on issue #602: KYLIN-3939 add BOM character,slove the bug that it shows Chinese garb… URL: https://github.com/apache/kylin/pull/602#issuecomment-486286393 > Tested, there will be an empty line at first in the exported csv file Hi Chunen Thanks for your response, I also find the same problem. I add a new commit to resolve the empty line problem. Best regards. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] nichunen commented on issue #602: KYLIN-3939 add BOM character,slove the bug that it shows Chinese garb…
nichunen commented on issue #602: KYLIN-3939 add BOM character,slove the bug that it shows Chinese garb… URL: https://github.com/apache/kylin/pull/602#issuecomment-486201433 Tested, there will be an empty line at first in the exported csv file This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 9:30 AM:
--
h2. *Try-2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/${LOGIN_NMAE}/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
## repackage mapreduce.tar.gz
tar zcf mapreduce.tar.gz hadoop/
## put it in right place
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
sh bin/kylin.sh start{code}
Though it looks so tricky, but this can work .
was (Author: hit_lacus):
h2. *Try-2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
{code}
Though it looks so tricky, but this can work .
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h3. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
> h3. *Error message*
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h3. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| h3. *Error message* !image-2019-04-24-15-38-10-098.png! was: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h3. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| h3. Error message !image-2019-04-24-15-38-10-098.png! > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h3. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > h3. *Error message* > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h3. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| h3. Error message !image-2019-04-24-15-38-10-098.png! was: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h2. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| !image-2019-04-24-15-38-10-098.png! > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h3. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > h3. Error message > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 9:23 AM:
--
h2. *Try-2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
{code}
Though it looks so tricky, but this can work .
was (Author: hit_lacus):
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
{code}
Though it looks so tricky, but this can work .
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 9:22 AM:
--
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
{code}
Though it looks so tricky, but this can work .
was (Author: hit_lacus):
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
{code}
Though it looks so tricky, but this can work without repackage.
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 9:00 AM:
--
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
{code}
Though it looks so tricky, but this can work without repackage.
was (Author: hit_lacus):
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
{code}
In my env, this can work.
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:57 AM:
--
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
{code}
In my env, this can work.
was (Author: hit_lacus):
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in HDFS
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
{code}
In my env, this can work.
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:56 AM:
--
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in HDFS
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
{code}
In my env, this can work.
was (Author: hit_lacus):
h2. *Try2*
{code}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in HDFS
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
{code}
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:53 AM:
--
h2. *Try2*
{code}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in HDFS
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
{code}
was (Author: hit_lacus):
h2. *Try2*
# remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
# remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
# remove
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923 ] XiaoXiang Yu commented on KYLIN-3970: - h2. *Try2* # remove old version kafka lib mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old # remove mapdreduce lib cache rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache # remove hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz . hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori tar zxf mapreduce.tar.gz rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar tar zcf mapreduce.tar.gz hadoop/ hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824897#comment-16824897
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:43 AM:
--
h2. *Try-1*
remove old version kafka-client.jar
{code:java}
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
{code}
We want to let mapper use kylin's classpath first, so we add
*mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
But this cause more conflict, we will got a NoSuchMethodError caused by Guava.
{code:java}
2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24
08:06:04,379 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24
08:06:04,381 FATAL [Thread-13]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[Thread-13,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher;
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411)
at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO
[Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1:
ExitException
{code}
was (Author: hit_lacus):
h2. *Try*
remove old version kafka-client.jar
{code}
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
{code}
add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
We will got a NoSuchMethodError caused by Guava
{code:java}
2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24
08:06:04,379 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24
08:06:04,381 FATAL [Thread-13]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[Thread-13,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher;
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411)
at java.lang.Thread.run(Thread.java:748) 2019-04-2
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824897#comment-16824897
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:26 AM:
--
h2. *Try*
remove old version kafka-client.jar
{code}
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
{code}
add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
We will got a NoSuchMethodError caused by Guava
{code:java}
2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24
08:06:04,379 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24
08:06:04,381 FATAL [Thread-13]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[Thread-13,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher;
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411)
at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO
[Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1:
ExitException
{code}
was (Author: hit_lacus):
h2. *Try*
# remove old version kafka-client.jar
# add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
We will got a NoSuchMethodError caused by Guava
{code}
2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24
08:06:04,379 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24
08:06:04,381 FATAL [Thread-13]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[Thread-13,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher;
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411)
at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO
[Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1:
ExitException
{code}
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https:
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824897#comment-16824897
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:21 AM:
--
h2. *Try*
# remove old version kafka-client.jar
# add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
We will got a NoSuchMethodError caused by Guava
{code}
2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24
08:06:04,379 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24
08:06:04,381 FATAL [Thread-13]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[Thread-13,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher;
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411)
at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO
[Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1:
ExitException
{code}
was (Author: hit_lacus):
h2. *Solution*
# remove old version kafka-client.jar
# add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.1
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824897#comment-16824897 ] XiaoXiang Yu commented on KYLIN-3970: - h2. *Solution* # remove old version kafka-client.jar # add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_ > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Attachment: (was: image-2019-04-23-14-25-51-066.png) > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823735#comment-16823735
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:55 AM:
--
h2. *Problem:* *Kafka jar conflict*
* when click build buttom
{code:java}
Caused by: org.apache.kafka.common.config.ConfigException: Missing required
configuration "partition.assignment.strategy" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
at
org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400)
at
org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48)
at
org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128)
at
org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100)
at
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
at
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394)
... 80 more
{code}
* First step of build segment
{code:java}
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1555653800318_0004Job Diagnostics:Task failed
task_1555653800318_0004_m_00
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
h3.
was (Author: hit_lacus):
*Problem:* *Kafka jar conflict*
{code}
Caused by: org.apache.kafka.common.config.ConfigException: Missing required
configuration "partition.assignment.strategy" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
at
org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400)
at
org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48)
at
org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128)
at
org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100)
at
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
at
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394)
... 80 more
{code}
{code}
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1555653800318_0004Job Diagnostics:Task failed
task_1555653800318_0004_m_00
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
h3.
> NR
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823735#comment-16823735
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:54 AM:
--
*Problem:* *Kafka jar conflict*
{code}
Caused by: org.apache.kafka.common.config.ConfigException: Missing required
configuration "partition.assignment.strategy" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
at
org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400)
at
org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48)
at
org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128)
at
org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100)
at
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
at
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394)
... 80 more
{code}
{code}
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1555653800318_0004Job Diagnostics:Task failed
task_1555653800318_0004_m_00
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
h3.
was (Author: hit_lacus):
*Problem:* *Kafka jar conflict*
Caused by: org.apache.kafka.common.config.ConfigException: Missing required
configuration "partition.assignment.strategy" which has no default value.
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
at
org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413)
at
org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400)
at
org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48)
at
org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128)
at
org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100)
at
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232)
at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
at
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394)
... 80 more
org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job
job_1555653800318_0004Job Diagnostics:Task failed
task_1555653800318_0004_m_00
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0
killedReduces: 0
Failure task Diagnostics:
Error:
org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set;
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
h3.
> NRT Streaming cannot work at HDP3.0
> ---
>
> Key: KYLIN-
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Fix Version/s: (was: v2.6.2) (was: v3.0.0) > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Attachments: image-2019-04-23-14-25-51-066.png, > image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824880#comment-16824880
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:50 AM:
--
h2. *Root Cause*
HDP contains a low version kafka-client jar, both in Kylin process's classpath
and MapReduce's classpath :
"/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar".
This is my classpath of Kylin:
{code}CLASSPATH =
/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/bin/bootstrap.jar:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/bin/tomcat-juli.jar:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/lib/*:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/conf:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/lib/*:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/ext/*::/usr/hdp/3.0.1.0-187/hbase/conf:/usr/local/bin/jdk1.8.0_191/lib/tools.jar:/usr/hdp/3.0.1.0-187/hbase:/usr/hdp/3.0.1.0-187/hbase/lib/accessors-smart-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aopalliance-1.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aopalliance-repackaged-2.5.0-b32.jar:/usr/hdp/3.0.1.0-187/hbase/lib/asm-5.0.4.jar:/usr/hdp/3.0.1.0-187/hbase/lib/atlas-plugin-classloader-1.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/audience-annotations-0.5.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/avro-1.7.7.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aws-java-sdk-bundle-1.11.271.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-beanutils-1.9.3.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-cli-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-codec-1.10.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-collections-3.2.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-compress-1.4.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-configuration2-2.1.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-crypto-1.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-csv-1.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-daemon-1.0.13.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-io-2.5.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-lang-2.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-lang3-3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-logging-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-math3-3.6.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-net-3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-client-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-framework-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-recipes-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/disruptor-3.3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/dnsjava-2.1.7.jar:/usr/hdp/3.0.1.0-187/hbase/lib/ehcache-3.3.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/findbugs-annotations-1.3.9-1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/fst-2.50.jar:/usr/hdp/3.0.1.0-187/hbase/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/gson-2.2.4.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guava-11.0.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guice-4.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guice-servlet-4.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hamcrest-core-1.3.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-backup-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-backup.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-bridge-shim-1.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-client-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-client.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-endpoint-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-endpoint.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-examples-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-examples.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-external-blockcache-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-external-blockcache.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-http-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-http.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-mapreduce-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-18
[jira] [Comment Edited] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823735#comment-16823735 ] XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 7:48 AM: -- *Problem:* *Kafka jar conflict* Caused by: org.apache.kafka.common.config.ConfigException: Missing required configuration "partition.assignment.strategy" which has no default value. at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124) at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48) at org.apache.kafka.clients.consumer.ConsumerConfig.(ConsumerConfig.java:194) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:430) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:413) at org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:400) at org.apache.kylin.source.kafka.util.KafkaClient.getKafkaConsumer(KafkaClient.java:48) at org.apache.kylin.source.kafka.util.KafkaClient.getEarliestOffsets(KafkaClient.java:128) at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartitionBeforeBuild(KafkaSource.java:100) at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:232) at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:394) ... 80 more org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0004Job Diagnostics:Task failed task_1555653800318_0004_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) h3. was (Author: hit_lacus): h3. *First problem:* *Kafka jar conflict* org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0004Job Diagnostics:Task failed task_1555653800318_0004_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: org.apache.kafka.clients.consumer.ConsumerConfig.configNames()Ljava/util/Set; at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:173) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) h3. *Second problem: Hive null value* org.apache.kylin.engine.mr.exception.MapReduceException: no counters for job job_1555653800318_0007Job Diagnostics:Task failed task_1555653800318_0007_m_00 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 Failure task Diagnostics: Error: java.lang.IllegalArgumentException: there is no valid date pattern for:\N at org.apache.kylin.common.util.DateFormat.stringToMillis(DateFormat.java:124) at org.apache.kylin.metadata.datatype.DataTypeOrder$DateTimeOrder.toComparable(DataTypeOrder.java:154) at org.apache.kylin.metadata.datatype.DataTypeOrder$DateTimeOrder.toComparable(DataTypeOrder.java:148) at org.apache.kylin.metadata.datatype.DataTypeOrder.compare(DataTypeOrder.java:95) at org.apache.kylin.metadata.datatype.DataTypeOrder.max(DataTypeOrder.java:90) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper.doMap(FactDistinctColumnsMapper.java:194) at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.m
[jira] [Commented] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824880#comment-16824880 ] XiaoXiang Yu commented on KYLIN-3970: - Root Cause HDP contains a low version kafka-client jar, both in Kylin process's classpath and MapReduce's classpath : "/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar". This is my classpath of Kylin: CLASSPATH = /home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/bin/bootstrap.jar:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/bin/tomcat-juli.jar:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/bin/../tomcat/lib/*:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/conf:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/lib/*:/home/kylin-hdp3/xiaoxiang/apache-kylin-2.6.1-bin-hadoop3/ext/*::/usr/hdp/3.0.1.0-187/hbase/conf:/usr/local/bin/jdk1.8.0_191/lib/tools.jar:/usr/hdp/3.0.1.0-187/hbase:/usr/hdp/3.0.1.0-187/hbase/lib/accessors-smart-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aopalliance-1.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aopalliance-repackaged-2.5.0-b32.jar:/usr/hdp/3.0.1.0-187/hbase/lib/asm-5.0.4.jar:/usr/hdp/3.0.1.0-187/hbase/lib/atlas-plugin-classloader-1.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/audience-annotations-0.5.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/avro-1.7.7.jar:/usr/hdp/3.0.1.0-187/hbase/lib/aws-java-sdk-bundle-1.11.271.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-beanutils-1.9.3.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-cli-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-codec-1.10.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-collections-3.2.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-compress-1.4.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-configuration2-2.1.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-crypto-1.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-csv-1.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-daemon-1.0.13.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-io-2.5.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-lang-2.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-lang3-3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-logging-1.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-math3-3.6.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/commons-net-3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-client-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-framework-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/curator-recipes-4.0.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/disruptor-3.3.6.jar:/usr/hdp/3.0.1.0-187/hbase/lib/dnsjava-2.1.7.jar:/usr/hdp/3.0.1.0-187/hbase/lib/ehcache-3.3.1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/findbugs-annotations-1.3.9-1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/fst-2.50.jar:/usr/hdp/3.0.1.0-187/hbase/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/hdp/3.0.1.0-187/hbase/lib/gson-2.2.4.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guava-11.0.2.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guice-4.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/guice-servlet-4.0.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hamcrest-core-1.3.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-annotations.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-backup-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-backup.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-bridge-shim-1.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-client-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-client.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-common.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-endpoint-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-endpoint.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-examples-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-examples.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-external-blockcache-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-external-blockcache.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop2-compat.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-hadoop-compat.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-http-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-http.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it-2.0.0.3.0.1.0-187-tests.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-it.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-mapreduce-2.0.0.3.0.1.0-187.jar:/usr/hdp/3.0.1.0-187/hbase/lib/hbase-mapreduce-2.0.0.3.0.1.0-187-tests.jar:/usr
[jira] [Comment Edited] (KYLIN-3977) Aviod mistaken deleting dicts by storage cleanup while building jobs are running
[ https://issues.apache.org/jira/browse/KYLIN-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824878#comment-16824878 ] PENG Zhengshuai edited comment on KYLIN-3977 at 4/24/19 7:42 AM: - PR: https://github.com/apache/kylin/pull/619 was (Author: peng zhengshuai): PR: *https://github.com/apache/kylin/pull/619* > Aviod mistaken deleting dicts by storage cleanup while building jobs are > running > > > Key: KYLIN-3977 > URL: https://issues.apache.org/jira/browse/KYLIN-3977 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Major > > When there are cube building jobs running, storage cleanup may delete some > dictionaries which the cubes are not referenced to. But the in build jobs, > those dictionaries may be reused. > If these reused dictionaries have been deleted by storage cleanup, the build > job will be failed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3977) Aviod mistaken deleting dicts by storage cleanup while building jobs are running
[ https://issues.apache.org/jira/browse/KYLIN-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824878#comment-16824878 ] PENG Zhengshuai commented on KYLIN-3977: PR: *https://github.com/apache/kylin/pull/619* > Aviod mistaken deleting dicts by storage cleanup while building jobs are > running > > > Key: KYLIN-3977 > URL: https://issues.apache.org/jira/browse/KYLIN-3977 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Major > > When there are cube building jobs running, storage cleanup may delete some > dictionaries which the cubes are not referenced to. But the in build jobs, > those dictionaries may be reused. > If these reused dictionaries have been deleted by storage cleanup, the build > job will be failed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] asfgit commented on issue #619: Kylin 3977
asfgit commented on issue #619: Kylin 3977 URL: https://github.com/apache/kylin/pull/619#issuecomment-486107144 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] ZhengshuaiPENG opened a new pull request #619: Kylin 3977
ZhengshuaiPENG opened a new pull request #619: Kylin 3977 URL: https://github.com/apache/kylin/pull/619 1. add ResourceStore.delete(respath, timestamp) interface & impl 2. add ResourceStore.updateTimestamp(respath, timestamp) interface & impl 3. update resource timestamp when reusing dict/snapshot 4. Retry avaibility build process of dict & snapshot 5. refactor metadata cleanup by using delete with timestamp interface This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Attachment: image-2019-04-24-15-38-10-098.png > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png, > image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > > !image-2019-04-24-15-35-06-712.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. h2. *Env* |HDFS|3.1.1| |YARN|3.1.1| |MapReduce2|3.1.1| |Hive|3.1.0| |HBase|2.0.0| |ZooKeeper|3.4.6| |HDP|3.0.1.0| !image-2019-04-24-15-38-10-098.png! was: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. !image-2019-04-24-15-35-06-712.png! > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png, > image-2019-04-24-15-38-10-098.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > h2. *Env* > |HDFS|3.1.1| > |YARN|3.1.1| > |MapReduce2|3.1.1| > |Hive|3.1.0| > |HBase|2.0.0| > |ZooKeeper|3.4.6| > |HDP|3.0.1.0| > > > !image-2019-04-24-15-38-10-098.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3970) NRT Streaming cannot work at HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3970: Description: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. !image-2019-04-24-15-35-06-712.png! was: We found that when using [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], NRT streaming cannot work in serval step for jar conflict problem. !image-2019-04-23-14-25-51-066.png! > NRT Streaming cannot work at HDP3.0 > --- > > Key: KYLIN-3970 > URL: https://issues.apache.org/jira/browse/KYLIN-3970 > Project: Kylin > Issue Type: Bug > Components: NRT Streaming >Affects Versions: v2.6.1 >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Major > Labels: HDP, Hadoop3 > Fix For: v3.0.0, v2.6.2 > > Attachments: image-2019-04-23-14-25-51-066.png > > > We found that when using > [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz], > NRT streaming cannot work in serval step for jar conflict problem. > > !image-2019-04-24-15-35-06-712.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
