[jira] [Updated] (KYLIN-4437) Should replace deprecated "mapred.job.name"
[ https://issues.apache.org/jira/browse/KYLIN-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4437: Summary: Should replace deprecated "mapred.job.name" (was: Should replace mapred.job.name, because it's deprecated) > Should replace deprecated "mapred.job.name" > > > Key: KYLIN-4437 > URL: https://issues.apache.org/jira/browse/KYLIN-4437 > Project: Kylin > Issue Type: Improvement >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > > mapred.job.name is deprecated. > see: > [https://hadoop.apache.org/docs/r2.10.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html] > we should replace it to mapreduce.job.name -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4437) Should replace mapred.job.name, because it's deprecated
[ https://issues.apache.org/jira/browse/KYLIN-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4437: Summary: Should replace mapred.job.name, because it's deprecated (was: Should replace mapred.job.name, because it is deprecated) > Should replace mapred.job.name, because it's deprecated > --- > > Key: KYLIN-4437 > URL: https://issues.apache.org/jira/browse/KYLIN-4437 > Project: Kylin > Issue Type: Improvement >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > > mapred.job.name is deprecated. > see: > [https://hadoop.apache.org/docs/r2.10.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html] > we should replace it to mapreduce.job.name -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4437) Should replace mapred.job.name, because it is deprecated
ZhouKang created KYLIN-4437: --- Summary: Should replace mapred.job.name, because it is deprecated Key: KYLIN-4437 URL: https://issues.apache.org/jira/browse/KYLIN-4437 Project: Kylin Issue Type: Improvement Reporter: ZhouKang Assignee: ZhouKang mapred.job.name is deprecated. see: [https://hadoop.apache.org/docs/r2.10.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html] we should replace it to mapreduce.job.name -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4413) Add canary tool to Kylin
ZhouKang created KYLIN-4413: --- Summary: Add canary tool to Kylin Key: KYLIN-4413 URL: https://issues.apache.org/jira/browse/KYLIN-4413 Project: Kylin Issue Type: Improvement Reporter: ZhouKang Canary tool can be used to detect service availability, and also can detect query latency. Many other hadoop system has canary, such as HBase and HDFS. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4398) Display Kylin Version on UI
[ https://issues.apache.org/jira/browse/KYLIN-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17055532#comment-17055532 ] ZhouKang commented on KYLIN-4398: - [~yaho] Sure, the url is ok. I usually do operation in wrong cluster, so I wonder that whether cluster name show in UI will be better. > Display Kylin Version on UI > --- > > Key: KYLIN-4398 > URL: https://issues.apache.org/jira/browse/KYLIN-4398 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Zhong Yanghong >Assignee: Julian Pan >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4412) Show cluster name in job notification email title
ZhouKang created KYLIN-4412: --- Summary: Show cluster name in job notification email title Key: KYLIN-4412 URL: https://issues.apache.org/jira/browse/KYLIN-4412 Project: Kylin Issue Type: Improvement Reporter: ZhouKang Assignee: ZhouKang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4398) Display Kylin Version on UI
[ https://issues.apache.org/jira/browse/KYLIN-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17055506#comment-17055506 ] ZhouKang commented on KYLIN-4398: - maybe more information should be displayed on UI, such as Kylin cluster name, Kylin env(DEV, TST, PROD). It's better if could be config in kylin.properties like kylin.web.help > Display Kylin Version on UI > --- > > Key: KYLIN-4398 > URL: https://issues.apache.org/jira/browse/KYLIN-4398 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Zhong Yanghong >Assignee: Julian Pan >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4400) Use beeline as hive client in system-cube.sh
[ https://issues.apache.org/jira/browse/KYLIN-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang reassigned KYLIN-4400: --- Assignee: ZhouKang > Use beeline as hive client in system-cube.sh > > > Key: KYLIN-4400 > URL: https://issues.apache.org/jira/browse/KYLIN-4400 > Project: Kylin > Issue Type: Improvement >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > > Just like sample.sh > We should use hive-client-mode's client in kylin.properties to submit hive > sql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4401) NPE when bash bin/system-cube.sh setup
ZhouKang created KYLIN-4401:
---
Summary: NPE when bash bin/system-cube.sh setup
Key: KYLIN-4401
URL: https://issues.apache.org/jira/browse/KYLIN-4401
Project: Kylin
Issue Type: Improvement
Reporter: ZhouKang
Assignee: ZhouKang
when use bin/system-cube.sh to setup project KYLIN_SYSTEM cube, there is a NPE
{code:java}
// code placeholder
2020-03-03 22:35:21,206 ERROR [main] cli.CubeSignatureRefresher:105 :
error2020-03-03 22:35:21,206 ERROR [main] cli.CubeSignatureRefresher:105 :
errorjava.lang.NullPointerException at
org.apache.kylin.cube.cli.CubeSignatureRefresher.updateCubeDesc(CubeSignatureRefresher.java:98)
at
org.apache.kylin.cube.cli.CubeSignatureRefresher.update(CubeSignatureRefresher.java:74)
at
org.apache.kylin.cube.cli.CubeSignatureRefresher.main(CubeSignatureRefresher.java:117)Exception
in thread "main" java.lang.NullPointerException at
org.apache.kylin.cube.cli.CubeSignatureRefresher.updateCubeDesc(CubeSignatureRefresher.java:106)
at
org.apache.kylin.cube.cli.CubeSignatureRefresher.update(CubeSignatureRefresher.java:74)
at
org.apache.kylin.cube.cli.CubeSignatureRefresher.main(CubeSignatureRefresher.java:117)
{code}
It happened when refresh cube signature. I think it's not such important, but
will be better if fixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4400) Use beeline as hive client in system-cube.sh
ZhouKang created KYLIN-4400: --- Summary: Use beeline as hive client in system-cube.sh Key: KYLIN-4400 URL: https://issues.apache.org/jira/browse/KYLIN-4400 Project: Kylin Issue Type: Improvement Reporter: ZhouKang Just like sample.sh We should use hive-client-mode's client in kylin.properties to submit hive sql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4231) write conflict while add model by different query server parallelly
[
https://issues.apache.org/jira/browse/KYLIN-4231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17049997#comment-17049997
]
ZhouKang commented on KYLIN-4231:
-
[~shaofengshi]
yes, you are right. I think we should share the http session in all Kylin
instances.
We use Nginx as load balance. By this rule, we found that the user login info
cannot shared by different Kylin instance, which made the query request failed.
Maybe we could add a cache to spring framework.
[~xiacongling] will add this feature later.
> write conflict while add model by different query server parallelly
> ---
>
> Key: KYLIN-4231
> URL: https://issues.apache.org/jira/browse/KYLIN-4231
> Project: Kylin
> Issue Type: New Feature
>Reporter: ZhouKang
>Priority: Major
>
> a kylin cluster have more than 1 query server, all of them are the backend
> server of nginx.
> when our user use RESTful API to create model in the *same* project
> parallelly, there will be a problem.
> the server returns:
> {code:java}
> // code placeholder
> Overwriting conflict /project/learn_kylin.json, expect old TS 1572596034269,
> but it is 1572596042929
> {code}
> BUT, the model '/model_desc/xxx.json' has already in metastore, so the next
> time our user want
> to retry (create model), he will get:
> {code:java}
> // code placeholder
> Overwriting conflict /model_desc/test_945.json, expect old TS 0, but it is
> 1572596193812
> {code}
> I think the problem is that:
> when server A receive PUT model, it will change the project info, but the
> cache update broadcast cannot be processed by server B before server B
> processing another model creation request.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4268) build job failed caused by hadoop utils update
[
https://issues.apache.org/jira/browse/KYLIN-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048140#comment-17048140
]
ZhouKang commented on KYLIN-4268:
-
I think it's worked using solution 1.
So close this issue.
> build job failed caused by hadoop utils update
> --
>
> Key: KYLIN-4268
> URL: https://issues.apache.org/jira/browse/KYLIN-4268
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> When we update hadoop package, such as hbase and hive. The job will fill
> cause by:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.io.FileNotFoundException: File
> file:/home/work/app/kylin-all-2.5.2-2.5.2-HBase2-SNAPSHOT/hbase-2.2.0-3.1.0/lib/hbase-common-2.2.0-3.1.0.jar
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
> at
> org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:406)
> at
> org.apache.spark.deploy.yarn.Client.org\$apache\$spark\$deploy\$yarn\$Client\$\$distribute\
> {code}
>
> The root cause is kylin will generate the full command as soon as the job
> summited. And the parameter '--jars' use file's absolute path.
>
> there will be 2 ways to fix this problem.
> # keep the old package in the same path
> # change the generation of '--jars' to the stage of job execution.
>
> which is the better choose?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (KYLIN-4268) build job failed caused by hadoop utils update
[
https://issues.apache.org/jira/browse/KYLIN-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang closed KYLIN-4268.
---
Resolution: Abandoned
> build job failed caused by hadoop utils update
> --
>
> Key: KYLIN-4268
> URL: https://issues.apache.org/jira/browse/KYLIN-4268
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> When we update hadoop package, such as hbase and hive. The job will fill
> cause by:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.io.FileNotFoundException: File
> file:/home/work/app/kylin-all-2.5.2-2.5.2-HBase2-SNAPSHOT/hbase-2.2.0-3.1.0/lib/hbase-common-2.2.0-3.1.0.jar
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
> at
> org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:406)
> at
> org.apache.spark.deploy.yarn.Client.org\$apache\$spark\$deploy\$yarn\$Client\$\$distribute\
> {code}
>
> The root cause is kylin will generate the full command as soon as the job
> summited. And the parameter '--jars' use file's absolute path.
>
> there will be 2 ways to fix this problem.
> # keep the old package in the same path
> # change the generation of '--jars' to the stage of job execution.
>
> which is the better choose?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4379) Calculate column cardinality cannot use kylin config overwrite cause job failed
[ https://issues.apache.org/jira/browse/KYLIN-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038229#comment-17038229 ] ZhouKang commented on KYLIN-4379: - Hi, [~nichunen] I have updated the details. If necessary, I can upload screenshots. > Calculate column cardinality cannot use kylin config overwrite cause job > failed > --- > > Key: KYLIN-4379 > URL: https://issues.apache.org/jira/browse/KYLIN-4379 > Project: Kylin > Issue Type: Bug >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > > Calculate column cardinality submit job to the default queue, which cause job > failed. > It seems this job cannot use kylin config overwrite. > When you load or reload data from hive to kylin, there will be a checkbox as > "Calculate column cardinality". If this checkbox is selected, Kylin will > submit a MR/Spark job. > When a MR job sumitted to yarn, it will use default queue. But in many > environment, it's forbidden. You should use the queue in > kylin.engine.mr.config-override.mapreduce.job.queuename -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4379) Calculate column cardinality cannot use kylin config overwrite cause job failed
[ https://issues.apache.org/jira/browse/KYLIN-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4379: Description: Calculate column cardinality submit job to the default queue, which cause job failed. It seems this job cannot use kylin config overwrite. When you load or reload data from hive to kylin, there will be a checkbox as "Calculate column cardinality". If this checkbox is selected, Kylin will submit a MR/Spark job. When a MR job sumitted to yarn, it will use default queue. But in many environment, it's forbidden. You should use the queue in kylin.engine.mr.config-override.mapreduce.job.queuename was: Calculate column cardinality submit job to the default queue, which cause job failed. It seems this job cannot use kylin config overwrite. > Calculate column cardinality cannot use kylin config overwrite cause job > failed > --- > > Key: KYLIN-4379 > URL: https://issues.apache.org/jira/browse/KYLIN-4379 > Project: Kylin > Issue Type: Bug >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > > Calculate column cardinality submit job to the default queue, which cause job > failed. > It seems this job cannot use kylin config overwrite. > When you load or reload data from hive to kylin, there will be a checkbox as > "Calculate column cardinality". If this checkbox is selected, Kylin will > submit a MR/Spark job. > When a MR job sumitted to yarn, it will use default queue. But in many > environment, it's forbidden. You should use the queue in > kylin.engine.mr.config-override.mapreduce.job.queuename -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4379) Calculate column cardinality cannot use kylin config overwrite cause job failed
[ https://issues.apache.org/jira/browse/KYLIN-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4379: Description: Calculate column cardinality submit job to the default queue, which cause job failed. It seems this job cannot use kylin config overwrite. was: Calculate column cardinality submit job to the root queue, which cause job failed. It seems this job cannot use kylin config overwrite. > Calculate column cardinality cannot use kylin config overwrite cause job > failed > --- > > Key: KYLIN-4379 > URL: https://issues.apache.org/jira/browse/KYLIN-4379 > Project: Kylin > Issue Type: Bug >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > > Calculate column cardinality submit job to the default queue, which cause job > failed. > It seems this job cannot use kylin config overwrite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4379) Calculate column cardinality cannot use kylin config overwrite cause job failed
ZhouKang created KYLIN-4379: --- Summary: Calculate column cardinality cannot use kylin config overwrite cause job failed Key: KYLIN-4379 URL: https://issues.apache.org/jira/browse/KYLIN-4379 Project: Kylin Issue Type: Bug Reporter: ZhouKang Assignee: ZhouKang Calculate column cardinality submit job to the root queue, which cause job failed. It seems this job cannot use kylin config overwrite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4377) Project's admin cannot operate Hybrid model
ZhouKang created KYLIN-4377: --- Summary: Project's admin cannot operate Hybrid model Key: KYLIN-4377 URL: https://issues.apache.org/jira/browse/KYLIN-4377 Project: Kylin Issue Type: Bug Reporter: ZhouKang Assignee: ZhouKang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4322) Cost–benefit of compression HBase result
[ https://issues.apache.org/jira/browse/KYLIN-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17027168#comment-17027168 ] ZhouKang commented on KYLIN-4322: - Thank you [~shaofengshi] I will do more work in testing and repairment > Cost–benefit of compression HBase result > > > Key: KYLIN-4322 > URL: https://issues.apache.org/jira/browse/KYLIN-4322 > Project: Kylin > Issue Type: Bug >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > Fix For: v3.1.0 > > > kylin.storage.hbase.endpoint-compress-result is TRUE as default. > In our production environment, when the hbase scan result is larger than > 200M, it will take more than 10s to compress data. > We can find this by hbase's log: > ||Size||avg rate||min rate||avg time||max time|| > |<1M|0.12|0.25|0.18ms|0.7s| > |1M ~ 10M|0.39|0.97|0.2s|0.6s| > |10M ~ 100M|0.47|0.81|2s|6.3s| > |>100M|0.95|0.96|15.7s|24.8s| > Notice： > # rate: compressed data size / origin data size > # when the source data size is < 1M, compressed data may larger than the > source data. So the table(Row 1) only calculate then compressed data less > than the source data > # In our environment, 65% compression data (<1M) is larger than source data > When source data is less then 10M, the latency of data transmission is > acceptability. When data is larger then 100M, it will take a long time to > compress data. > > So, I think kylin.storage.hbase.endpoint-compress-result should be FALSE by > default; > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4363) Query failed with "I failed to find one of the right cookies" error
[
https://issues.apache.org/jira/browse/KYLIN-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17027166#comment-17027166
]
ZhouKang commented on KYLIN-4363:
-
I will revert this commit.
In my test cases, the setting seems worked( have note upgrade coprocessor).
Coprocessor's log is
{code:java}
// code placeholder
# kylin.storage.hbase.endpoint-compress-result=TRUE
2020-01-31,10:01:18,005 INFO
[RpcServer.default.RWQ.Fifo.read.handler=315,queue=7,port=24600]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
start query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0 in thread
RpcServer.default.RWQ.Fifo.read.handler=315,queue=7,port=24600
2020-01-31,10:01:18,015 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.gridtable.GTScanRequest: pre aggregating results before
returning
2020-01-31,10:01:18,015 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.gridtable.GTAggregateScanner: setting IGTBypassChecker of child
2020-01-31,10:01:18,468 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.gridtable.GTAggregateScanner: GTAggregateScanner input rows:
399
2020-01-31,10:01:18,639 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.gridtable.GTAggregateScanner: closing aggrCache
2020-01-31,10:01:18,639 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
Total scanned 399 rows and 150930080 bytes
2020-01-31,10:01:23,903 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
Size of final result = 76890905 (82646392 before compressing)
# kylin.storage.hbase.endpoint-compress-result=FALSE
2020-01-31,10:59:54,657 INFO
[RpcServer.default.RWQ.Fifo.read.handler=406,queue=6,port=24600]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
start query 3645d35d-cb78-10c4-c327-a04acc9ff2a4 in thread
RpcServer.default.RWQ.Fifo.read.handler=406,queue=6,port=24600
2020-01-31,10:59:54,666 INFO [Query 3645d35d-cb78-10c4-c327-a04acc9ff2a4-492]
org.apache.kylin.gridtable.GTScanRequest: pre aggregating results before
returning
2020-01-31,10:59:54,666 INFO [Query 3645d35d-cb78-10c4-c327-a04acc9ff2a4-492]
org.apache.kylin.gridtable.GTAggregateScanner: setting IGTBypassChecker of child
2020-01-31,10:59:58,772 INFO [Query 3645d35d-cb78-10c4-c327-a04acc9ff2a4-492]
org.apache.kylin.gridtable.GTAggregateScanner: GTAggregateScanner input rows:
399
2020-01-31,11:00:00,064 INFO [Query 3645d35d-cb78-10c4-c327-a04acc9ff2a4-492]
org.apache.kylin.gridtable.GTAggregateScanner: closing aggrCache
2020-01-31,11:00:00,064 INFO [Query 3645d35d-cb78-10c4-c327-a04acc9ff2a4-492]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
Total scanned 399 rows and 150930080 bytes
2020-01-31,11:00:00,103 INFO [Query 3645d35d-cb78-10c4-c327-a04acc9ff2a4-492]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
Size of final result = 71950057 (71950057 before compressing){code}
I will test it fully and repair it later.
> Query failed with "I failed to find one of the right cookies" error
> ---
>
> Key: KYLIN-4363
> URL: https://issues.apache.org/jira/browse/KYLIN-4363
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v3.1.0
>Reporter: Shao Feng Shi
>Assignee: ZhouKang
>Priority: Major
> Fix For: v3.1.0
>
>
> {code:java}
> // code placeholder
> Caused by: java.lang.RuntimeException: I failed to find one of the right
> cookies. 1701223222Caused by: java.lang.RuntimeException: I failed to find
> one of the right cookies. 1701223222 at
> org.roaringbitmap.buffer.ImmutableRoaringArray.(ImmutableRoaringArray.java:46)
> at
> org.roaringbitmap.buffer.ImmutableRoaringBitmap.(ImmutableRoaringBitmap.java:908)
> at
> org.apache.kylin.measure.bitmap.RoaringBitmapCounter.peekLength(RoaringBitmapCounter.java:141)
> at
> org.apache.kylin.measure.bitmap.BitmapSerializer.peekLength(BitmapSerializer.java:69)
> at
> org.apache.kylin.cube.gridtable.CubeCodeSystem.codeLength(CubeCodeSystem.java:100)
> at org.apache.kylin.gridtable.GTRecord.loadColumns(GTRecord.java:279) at
> org.apache.kylin.storage.gtrecord.PartitionResultIterator.next(PartitionResultIterator.java:56)
> at
> org.apache.kylin.storage.gtrecord.PartitionResultIterator.next(PartitionResultIterator.java:35)
> at com.google.common.collect.Iterators$PeekingImpl.next(Iterators.java:1222)
> at
> org.apache.kylin.storage.gtrecord.SortMergedPartitionResultIterator.next(SortMergedPartitionResultIterator.java:93)
> at
> org.apache.kylin.storage.gtrecord.SortMergedPart
[jira] [Comment Edited] (KYLIN-4363) Query failed with "I failed to find one of the right cookies" error
[
https://issues.apache.org/jira/browse/KYLIN-4363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17027166#comment-17027166
]
ZhouKang edited comment on KYLIN-4363 at 1/31/20 3:39 AM:
--
I will revert this commit.
In my test cases, the setting seems worked( did not upgrade coprocessor).
Coprocessor's log is
{code:java}
// code placeholder
# kylin.storage.hbase.endpoint-compress-result=TRUE
2020-01-31,10:01:18,005 INFO
[RpcServer.default.RWQ.Fifo.read.handler=315,queue=7,port=24600]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
start query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0 in thread
RpcServer.default.RWQ.Fifo.read.handler=315,queue=7,port=24600
2020-01-31,10:01:18,015 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.gridtable.GTScanRequest: pre aggregating results before
returning
2020-01-31,10:01:18,015 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.gridtable.GTAggregateScanner: setting IGTBypassChecker of child
2020-01-31,10:01:18,468 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.gridtable.GTAggregateScanner: GTAggregateScanner input rows:
399
2020-01-31,10:01:18,639 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.gridtable.GTAggregateScanner: closing aggrCache
2020-01-31,10:01:18,639 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
Total scanned 399 rows and 150930080 bytes
2020-01-31,10:01:23,903 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
Size of final result = 76890905 (82646392 before compressing)
# kylin.storage.hbase.endpoint-compress-result=FALSE
2020-01-31,10:59:54,657 INFO
[RpcServer.default.RWQ.Fifo.read.handler=406,queue=6,port=24600]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
start query 3645d35d-cb78-10c4-c327-a04acc9ff2a4 in thread
RpcServer.default.RWQ.Fifo.read.handler=406,queue=6,port=24600
2020-01-31,10:59:54,666 INFO [Query 3645d35d-cb78-10c4-c327-a04acc9ff2a4-492]
org.apache.kylin.gridtable.GTScanRequest: pre aggregating results before
returning
2020-01-31,10:59:54,666 INFO [Query 3645d35d-cb78-10c4-c327-a04acc9ff2a4-492]
org.apache.kylin.gridtable.GTAggregateScanner: setting IGTBypassChecker of child
2020-01-31,10:59:58,772 INFO [Query 3645d35d-cb78-10c4-c327-a04acc9ff2a4-492]
org.apache.kylin.gridtable.GTAggregateScanner: GTAggregateScanner input rows:
399
2020-01-31,11:00:00,064 INFO [Query 3645d35d-cb78-10c4-c327-a04acc9ff2a4-492]
org.apache.kylin.gridtable.GTAggregateScanner: closing aggrCache
2020-01-31,11:00:00,064 INFO [Query 3645d35d-cb78-10c4-c327-a04acc9ff2a4-492]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
Total scanned 399 rows and 150930080 bytes
2020-01-31,11:00:00,103 INFO [Query 3645d35d-cb78-10c4-c327-a04acc9ff2a4-492]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
Size of final result = 71950057 (71950057 before compressing){code}
I will test it fully and repair it later.
was (Author: zhoukangcn):
I will revert this commit.
In my test cases, the setting seems worked( have note upgrade coprocessor).
Coprocessor's log is
{code:java}
// code placeholder
# kylin.storage.hbase.endpoint-compress-result=TRUE
2020-01-31,10:01:18,005 INFO
[RpcServer.default.RWQ.Fifo.read.handler=315,queue=7,port=24600]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
start query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0 in thread
RpcServer.default.RWQ.Fifo.read.handler=315,queue=7,port=24600
2020-01-31,10:01:18,015 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.gridtable.GTScanRequest: pre aggregating results before
returning
2020-01-31,10:01:18,015 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.gridtable.GTAggregateScanner: setting IGTBypassChecker of child
2020-01-31,10:01:18,468 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.gridtable.GTAggregateScanner: GTAggregateScanner input rows:
399
2020-01-31,10:01:18,639 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.gridtable.GTAggregateScanner: closing aggrCache
2020-01-31,10:01:18,639 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
Total scanned 399 rows and 150930080 bytes
2020-01-31,10:01:23,903 INFO [Query 745eb1e9-992e-e4ac-ce15-951ca5b3eae0-401]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
Size of final result = 76890905 (82646392 before compressing)
# kylin.storage.hbase.endpoint-compress-result=FALSE
2020-01-31,10:59:54,657 INFO
[RpcServer.default.RWQ.F
[jira] [Assigned] (KYLIN-4357) NPE in drop/cancel unexist job
[
https://issues.apache.org/jira/browse/KYLIN-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang reassigned KYLIN-4357:
---
Assignee: ZhouKang
> NPE in drop/cancel unexist job
> --
>
> Key: KYLIN-4357
> URL: https://issues.apache.org/jira/browse/KYLIN-4357
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Assignee: ZhouKang
>Priority: Major
>
> REST api
> jobs/\{uuid}/cancel
> jobs/\{uuid}/drop
> if uuid is not exist, NPE happened.
> {code:java}
> // code placeholder
> Caused by: java.lang.NullPointerException
> at
> org.apache.kylin.rest.util.AclEvaluate.getProjectByJob(AclEvaluate.java:51)
> at
> org.apache.kylin.rest.util.AclEvaluate.checkProjectOperationPermission(AclEvaluate.java:107)
> at org.apache.kylin.rest.service.JobService.cancelJob(JobService.java:610)
> at
> org.apache.kylin.rest.controller.JobController.cancel(JobController.java:204)
> ... 75 more\n
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4357) NPE in drop/cancel unexist job
[
https://issues.apache.org/jira/browse/KYLIN-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang updated KYLIN-4357:
Description:
REST api
jobs/\{uuid}/cancel
jobs/\{uuid}/drop
if uuid is not exist, NPE happened.
{code:java}
// code placeholder
Caused by: java.lang.NullPointerException\n\tat
org.apache.kylin.rest.util.AclEvaluate.getProjectByJob(AclEvaluate.java:51)\n\tat
org.apache.kylin.rest.util.AclEvaluate.checkProjectOperationPermission(AclEvaluate.java:107)\n\tat
org.apache.kylin.rest.service.JobService.cancelJob(JobService.java:610)\n\tat
org.apache.kylin.rest.controller.JobController.cancel(JobController.java:204)\n\t...
75 more\n
{code}
was:
REST api
jobs/\{uuid}/cancel
jobs/\{uuid}/drop
if uuid is not exist, NPE happened.
> NPE in drop/cancel unexist job
> --
>
> Key: KYLIN-4357
> URL: https://issues.apache.org/jira/browse/KYLIN-4357
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> REST api
> jobs/\{uuid}/cancel
> jobs/\{uuid}/drop
> if uuid is not exist, NPE happened.
> {code:java}
> // code placeholder
> Caused by: java.lang.NullPointerException\n\tat
> org.apache.kylin.rest.util.AclEvaluate.getProjectByJob(AclEvaluate.java:51)\n\tat
>
> org.apache.kylin.rest.util.AclEvaluate.checkProjectOperationPermission(AclEvaluate.java:107)\n\tat
>
> org.apache.kylin.rest.service.JobService.cancelJob(JobService.java:610)\n\tat
> org.apache.kylin.rest.controller.JobController.cancel(JobController.java:204)\n\t...
> 75 more\n
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4357) NPE in drop/cancel unexist job
[
https://issues.apache.org/jira/browse/KYLIN-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang updated KYLIN-4357:
Description:
REST api
jobs/\{uuid}/cancel
jobs/\{uuid}/drop
if uuid is not exist, NPE happened.
{code:java}
// code placeholder
Caused by: java.lang.NullPointerException
at org.apache.kylin.rest.util.AclEvaluate.getProjectByJob(AclEvaluate.java:51)
at
org.apache.kylin.rest.util.AclEvaluate.checkProjectOperationPermission(AclEvaluate.java:107)
at org.apache.kylin.rest.service.JobService.cancelJob(JobService.java:610)
at
org.apache.kylin.rest.controller.JobController.cancel(JobController.java:204)
... 75 more\n
{code}
was:
REST api
jobs/\{uuid}/cancel
jobs/\{uuid}/drop
if uuid is not exist, NPE happened.
{code:java}
// code placeholder
Caused by: java.lang.NullPointerException\n\tat
org.apache.kylin.rest.util.AclEvaluate.getProjectByJob(AclEvaluate.java:51)
at
org.apache.kylin.rest.util.AclEvaluate.checkProjectOperationPermission(AclEvaluate.java:107)
at org.apache.kylin.rest.service.JobService.cancelJob(JobService.java:610)
at
org.apache.kylin.rest.controller.JobController.cancel(JobController.java:204)
... 75 more\n
{code}
> NPE in drop/cancel unexist job
> --
>
> Key: KYLIN-4357
> URL: https://issues.apache.org/jira/browse/KYLIN-4357
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> REST api
> jobs/\{uuid}/cancel
> jobs/\{uuid}/drop
> if uuid is not exist, NPE happened.
> {code:java}
> // code placeholder
> Caused by: java.lang.NullPointerException
> at
> org.apache.kylin.rest.util.AclEvaluate.getProjectByJob(AclEvaluate.java:51)
> at
> org.apache.kylin.rest.util.AclEvaluate.checkProjectOperationPermission(AclEvaluate.java:107)
> at org.apache.kylin.rest.service.JobService.cancelJob(JobService.java:610)
> at
> org.apache.kylin.rest.controller.JobController.cancel(JobController.java:204)
> ... 75 more\n
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4357) NPE in drop/cancel unexist job
[
https://issues.apache.org/jira/browse/KYLIN-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang updated KYLIN-4357:
Description:
REST api
jobs/\{uuid}/cancel
jobs/\{uuid}/drop
if uuid is not exist, NPE happened.
{code:java}
// code placeholder
Caused by: java.lang.NullPointerException\n\tat
org.apache.kylin.rest.util.AclEvaluate.getProjectByJob(AclEvaluate.java:51)
at
org.apache.kylin.rest.util.AclEvaluate.checkProjectOperationPermission(AclEvaluate.java:107)
at org.apache.kylin.rest.service.JobService.cancelJob(JobService.java:610)
at
org.apache.kylin.rest.controller.JobController.cancel(JobController.java:204)
... 75 more\n
{code}
was:
REST api
jobs/\{uuid}/cancel
jobs/\{uuid}/drop
if uuid is not exist, NPE happened.
{code:java}
// code placeholder
Caused by: java.lang.NullPointerException\n\tat
org.apache.kylin.rest.util.AclEvaluate.getProjectByJob(AclEvaluate.java:51)\n\tat
org.apache.kylin.rest.util.AclEvaluate.checkProjectOperationPermission(AclEvaluate.java:107)\n\tat
org.apache.kylin.rest.service.JobService.cancelJob(JobService.java:610)\n\tat
org.apache.kylin.rest.controller.JobController.cancel(JobController.java:204)\n\t...
75 more\n
{code}
> NPE in drop/cancel unexist job
> --
>
> Key: KYLIN-4357
> URL: https://issues.apache.org/jira/browse/KYLIN-4357
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> REST api
> jobs/\{uuid}/cancel
> jobs/\{uuid}/drop
> if uuid is not exist, NPE happened.
> {code:java}
> // code placeholder
> Caused by: java.lang.NullPointerException\n\tat
> org.apache.kylin.rest.util.AclEvaluate.getProjectByJob(AclEvaluate.java:51)
> at
> org.apache.kylin.rest.util.AclEvaluate.checkProjectOperationPermission(AclEvaluate.java:107)
> at org.apache.kylin.rest.service.JobService.cancelJob(JobService.java:610)
> at
> org.apache.kylin.rest.controller.JobController.cancel(JobController.java:204)
> ... 75 more\n
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4357) NPE in drop/cancel unexist job
ZhouKang created KYLIN-4357:
---
Summary: NPE in drop/cancel unexist job
Key: KYLIN-4357
URL: https://issues.apache.org/jira/browse/KYLIN-4357
Project: Kylin
Issue Type: Bug
Reporter: ZhouKang
REST api
jobs/\{uuid}/cancel
jobs/\{uuid}/drop
if uuid is not exist, NPE happened.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KYLIN-4356) Failed "Hive Column Cardinality calculation for table" jobs cannot be delete
[
https://issues.apache.org/jira/browse/KYLIN-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang reassigned KYLIN-4356:
---
Assignee: ZhouKang
> Failed "Hive Column Cardinality calculation for table" jobs cannot be delete
> -
>
> Key: KYLIN-4356
> URL: https://issues.apache.org/jira/browse/KYLIN-4356
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Assignee: ZhouKang
>Priority: Major
>
> In KylinHealthCheck, we found there are too many failed jobs like this:
> "name" : "Hive Column Cardinality calculation for table 'xxx.xxx'"
> the FAILED job cannot be cleanup by MetastoreCleanupJob,
> and the RESTful API DELETE jobs/\{uuid}/drop cannot work, return exeception
> {code:java}
> // code placeholder
> Caused by: org.apache.kylin.rest.exception.BadRequestException: Illegal job
> type, id: 4257d976-706b-90cb-d82b-1828e0128fd1.
> at
> org.apache.kylin.rest.service.JobService.getSingleJobInstance(JobService.java:501)
> at
> org.apache.kylin.rest.service.JobService.getJobInstance(JobService.java:486)
> at
> org.apache.kylin.rest.controller.JobController.dropJob(JobController.java:264)
> ... 75 more
>
> {code}
> seems there is no api to delete failed job of "Cardinality calculation" .
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4356) Failed "Hive Column Cardinality calculation for table" jobs cannot be delete
ZhouKang created KYLIN-4356:
---
Summary: Failed "Hive Column Cardinality calculation for table"
jobs cannot be delete
Key: KYLIN-4356
URL: https://issues.apache.org/jira/browse/KYLIN-4356
Project: Kylin
Issue Type: Bug
Reporter: ZhouKang
In KylinHealthCheck, we found there are too many failed jobs like this:
"name" : "Hive Column Cardinality calculation for table 'xxx.xxx'"
the FAILED job cannot be cleanup by MetastoreCleanupJob,
and the RESTful API DELETE jobs/\{uuid}/drop cannot work, return exeception
{code:java}
// code placeholder
Caused by: org.apache.kylin.rest.exception.BadRequestException: Illegal job
type, id: 4257d976-706b-90cb-d82b-1828e0128fd1.
at
org.apache.kylin.rest.service.JobService.getSingleJobInstance(JobService.java:501)
at
org.apache.kylin.rest.service.JobService.getJobInstance(JobService.java:486)
at
org.apache.kylin.rest.controller.JobController.dropJob(JobController.java:264)
... 75 more
{code}
seems there is no api to delete failed job of "Cardinality calculation" .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4322) Cost–benefit of compression HBase result
[ https://issues.apache.org/jira/browse/KYLIN-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4322: Description: kylin.storage.hbase.endpoint-compress-result is TRUE as default. In our production environment, when the hbase scan result is larger than 200M, it will take more than 10s to compress data. We can find this by hbase's log: ||Size||avg rate||min rate||avg time||max time|| |<1M|0.12|0.25|0.18ms|0.7s| |1M ~ 10M|0.39|0.97|0.2s|0.6s| |10M ~ 100M|0.47|0.81|2s|6.3s| |>100M|0.95|0.96|15.7s|24.8s| Notice： # rate: compressed data size / origin data size # when the source data size is < 1M, compressed data may larger than the source data. So the table(Row 1) only calculate then compressed data less than the source data # In our environment, 65% compression data (<1M) is larger than source data When source data is less then 10M, the latency of data transmission is acceptability. When data is larger then 100M, it will take a long time to compress data. So, I think kylin.storage.hbase.endpoint-compress-result should be FALSE by default; was: kylin.storage.hbase.endpoint-compress-result is TRUE as default. In our production environment, when the hbase scan result is larger than 200M, it will take more than 10s to compress data. We can find this by hbase's log: ||Size||avg rate||max rate||avg time||max time|| |<1M|0.12|0.25|0.18ms|0.7s| |1M ~ 10M|0.39|0.97|0.2s|0.6s| |10M ~ 100M|0.47|0.81|2s|6.3s| |>100M|0.95|0.96|15.7s|24.8s| Notice： # rate: compressed data size / origin data size # when the source data size is < 1M, compressed data may larger than the source data. So the table(Row 1) only calculate then compressed data less than the source data # In our environment, 65% compression data (<1M) is larger than source data When source data is less then 10M, the latency of data transmission is acceptability. When data is larger then 100M, it will take a long time to compress data. So, I think kylin.storage.hbase.endpoint-compress-result should be FALSE by default; > Cost–benefit of compression HBase result > > > Key: KYLIN-4322 > URL: https://issues.apache.org/jira/browse/KYLIN-4322 > Project: Kylin > Issue Type: Bug >Reporter: ZhouKang >Priority: Major > > kylin.storage.hbase.endpoint-compress-result is TRUE as default. > In our production environment, when the hbase scan result is larger than > 200M, it will take more than 10s to compress data. > We can find this by hbase's log: > ||Size||avg rate||min rate||avg time||max time|| > |<1M|0.12|0.25|0.18ms|0.7s| > |1M ~ 10M|0.39|0.97|0.2s|0.6s| > |10M ~ 100M|0.47|0.81|2s|6.3s| > |>100M|0.95|0.96|15.7s|24.8s| > Notice： > # rate: compressed data size / origin data size > # when the source data size is < 1M, compressed data may larger than the > source data. So the table(Row 1) only calculate then compressed data less > than the source data > # In our environment, 65% compression data (<1M) is larger than source data > When source data is less then 10M, the latency of data transmission is > acceptability. When data is larger then 100M, it will take a long time to > compress data. > > So, I think kylin.storage.hbase.endpoint-compress-result should be FALSE by > default; > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4322) Cost–benefit of compression HBase result
[ https://issues.apache.org/jira/browse/KYLIN-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4322: Description: kylin.storage.hbase.endpoint-compress-result is TRUE as default. In our production environment, when the hbase scan result is larger than 200M, it will take more than 10s to compress data. We can find this by hbase's log: ||Size||avg rate||max rate||avg time||max time|| |<1M|0.12|0.25|0.18ms|0.7s| |1M ~ 10M|0.39|0.97|0.2s|0.6s| |10M ~ 100M|0.47|0.81|2s|6.3s| |>100M|0.95|0.96|15.7s|24.8s| Notice： # rate: compressed data size / origin data size # when the source data size is < 1M, compressed data may larger than the source data. So the table(Row 1) only calculate then compressed data less than the source data # In our environment, 65% compression data (<1M) is larger than source data When source data is less then 10M, the latency of data transmission is acceptability. When data is larger then 100M, it will take a long time to compress data. So, I think kylin.storage.hbase.endpoint-compress-result should be FALSE by default; was: kylin.storage.hbase.endpoint-compress-result is TRUE as default. In our production environment, when the hbase scan result is larger than 200M, it will take more than 10s to compress data. We can find this by hbase's log: ||Size||avg rate||max rate||avg time||max time|| |<1M|0.12|0.25|0.18ms|0.7s| |1M ~ 10M|0.39|0.97|0.2s|0.6s| |10M ~ 100M|0.47|0.81|2s|6.3s| |>100M|0.95|0.96|15.7s|24.8s| rate: compressed data size / origin data size AND please NOTICE that, when the source data size is less than 1M, 65% compression data is larger than source data. When source data is less then 10M, the latency of data transmission is acceptability. When data is larger then 100M, it will take a long time to compress data. So, I think kylin.storage.hbase.endpoint-compress-result should be FALSE by default; > Cost–benefit of compression HBase result > > > Key: KYLIN-4322 > URL: https://issues.apache.org/jira/browse/KYLIN-4322 > Project: Kylin > Issue Type: Bug >Reporter: ZhouKang >Priority: Major > > kylin.storage.hbase.endpoint-compress-result is TRUE as default. > In our production environment, when the hbase scan result is larger than > 200M, it will take more than 10s to compress data. > We can find this by hbase's log: > ||Size||avg rate||max rate||avg time||max time|| > |<1M|0.12|0.25|0.18ms|0.7s| > |1M ~ 10M|0.39|0.97|0.2s|0.6s| > |10M ~ 100M|0.47|0.81|2s|6.3s| > |>100M|0.95|0.96|15.7s|24.8s| > Notice： > # rate: compressed data size / origin data size > # when the source data size is < 1M, compressed data may larger than the > source data. So the table(Row 1) only calculate then compressed data less > than the source data > # In our environment, 65% compression data (<1M) is larger than source data > When source data is less then 10M, the latency of data transmission is > acceptability. When data is larger then 100M, it will take a long time to > compress data. > > So, I think kylin.storage.hbase.endpoint-compress-result should be FALSE by > default; > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4322) Cost–benefit of compression HBase result
[ https://issues.apache.org/jira/browse/KYLIN-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4322: Summary: Cost–benefit of compression HBase result (was: ROA of compression HBase result) > Cost–benefit of compression HBase result > > > Key: KYLIN-4322 > URL: https://issues.apache.org/jira/browse/KYLIN-4322 > Project: Kylin > Issue Type: Bug >Reporter: ZhouKang >Priority: Major > > kylin.storage.hbase.endpoint-compress-result is TRUE as default. > In our production environment, when the hbase scan result is larger than > 200M, it will take more than 10s to compress data. > We can find this by hbase's log: > ||Size||avg rate||max rate||avg time||max time|| > |<1M|0.12|0.25|0.18ms|0.7s| > |1M ~ 10M|0.39|0.97|0.2s|0.6s| > |10M ~ 100M|0.47|0.81|2s|6.3s| > |>100M|0.95|0.96|15.7s|24.8s| > rate: compressed data size / origin data size > AND please NOTICE that, > when the source data size is less than 1M, 65% compression data is larger > than source data. > When source data is less then 10M, the latency of data transmission is > acceptability. When data is larger then 100M, it will take a long time to > compress data. > > So, I think kylin.storage.hbase.endpoint-compress-result should be FALSE by > default; > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4322) ROA of compression HBase result
[ https://issues.apache.org/jira/browse/KYLIN-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4322: Summary: ROA of compression HBase result (was: ROC of compression HBase result) > ROA of compression HBase result > --- > > Key: KYLIN-4322 > URL: https://issues.apache.org/jira/browse/KYLIN-4322 > Project: Kylin > Issue Type: Bug >Reporter: ZhouKang >Priority: Major > > kylin.storage.hbase.endpoint-compress-result is TRUE as default. > In our production environment, when the hbase scan result is larger than > 200M, it will take more than 10s to compress data. > We can find this by hbase's log: > ||Size||avg rate||max rate||avg time||max time|| > |<1M|0.12|0.25|0.18ms|0.7s| > |1M ~ 10M|0.39|0.97|0.2s|0.6s| > |10M ~ 100M|0.47|0.81|2s|6.3s| > |>100M|0.95|0.96|15.7s|24.8s| > rate: compressed data size / origin data size > AND please NOTICE that, > when the source data size is less than 1M, 65% compression data is larger > than source data. > When source data is less then 10M, the latency of data transmission is > acceptability. When data is larger then 100M, it will take a long time to > compress data. > > So, I think kylin.storage.hbase.endpoint-compress-result should be FALSE by > default; > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4322) ROC of compression HBase result
[ https://issues.apache.org/jira/browse/KYLIN-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4322: Description: kylin.storage.hbase.endpoint-compress-result is TRUE as default. In our production environment, when the hbase scan result is larger than 200M, it will take more than 10s to compress data. We can find this by hbase's log: ||Size||avg rate||max rate||avg time||max time|| |<1M|0.12|0.25|0.18ms|0.7s| |1M ~ 10M|0.39|0.97|0.2s|0.6s| |10M ~ 100M|0.47|0.81|2s|6.3s| |>100M|0.95|0.96|15.7s|24.8s| rate: compressed data size / origin data size AND please NOTICE that, when the source data size is less than 1M, 65% compression data is larger than source data. When source data is less then 10M, the latency of data transmission is acceptability. When data is larger then 100M, it will take a long time to compress data. So, I think kylin.storage.hbase.endpoint-compress-result should be FALSE by default; > ROC of compression HBase result > --- > > Key: KYLIN-4322 > URL: https://issues.apache.org/jira/browse/KYLIN-4322 > Project: Kylin > Issue Type: Bug >Reporter: ZhouKang >Priority: Major > > kylin.storage.hbase.endpoint-compress-result is TRUE as default. > In our production environment, when the hbase scan result is larger than > 200M, it will take more than 10s to compress data. > We can find this by hbase's log: > ||Size||avg rate||max rate||avg time||max time|| > |<1M|0.12|0.25|0.18ms|0.7s| > |1M ~ 10M|0.39|0.97|0.2s|0.6s| > |10M ~ 100M|0.47|0.81|2s|6.3s| > |>100M|0.95|0.96|15.7s|24.8s| > rate: compressed data size / origin data size > AND please NOTICE that, > when the source data size is less than 1M, 65% compression data is larger > than source data. > When source data is less then 10M, the latency of data transmission is > acceptability. When data is larger then 100M, it will take a long time to > compress data. > > So, I think kylin.storage.hbase.endpoint-compress-result should be FALSE by > default; > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4322) ROC of compression HBase result
[ https://issues.apache.org/jira/browse/KYLIN-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4322: Summary: ROC of compression HBase result (was: HBase) > ROC of compression HBase result > --- > > Key: KYLIN-4322 > URL: https://issues.apache.org/jira/browse/KYLIN-4322 > Project: Kylin > Issue Type: Bug >Reporter: ZhouKang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4322) HBase
ZhouKang created KYLIN-4322: --- Summary: HBase Key: KYLIN-4322 URL: https://issues.apache.org/jira/browse/KYLIN-4322 Project: Kylin Issue Type: Bug Reporter: ZhouKang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4227) add segment build callback
[
https://issues.apache.org/jira/browse/KYLIN-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999837#comment-16999837
]
ZhouKang edited comment on KYLIN-4227 at 12/19/19 8:58 AM:
---
HTTP callback settings screen shot. see: Attachments
HTTP callback request :
{code:java}
// code placeholder
GET /test?cube=kylin_sales_cube&date_start=132537600&date_end=157515840
HTTP/1.1
Host: {host}:
Connection: Keep-Alive
{code}
was (Author: zhoukangcn):
HTTP callback settings screen shot. see: Attachments
HTTP callback request :
{code:java}
// code placeholder
GET /test?cube=kylin_sales_cube&date_start=132537600&date_end=157515840
HTTP/1.1
Host: {host}:
Connection: Keep-Alive
{code}
> add segment build callback
> --
>
> Key: KYLIN-4227
> URL: https://issues.apache.org/jira/browse/KYLIN-4227
> Project: Kylin
> Issue Type: New Feature
>Reporter: ZhouKang
>Priority: Major
> Attachments: 1.png, 2.png
>
>
> add HTTP callback for cube, some other system could use this feature to add a
> hook when segment building done.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4227) add segment build callback
[
https://issues.apache.org/jira/browse/KYLIN-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999837#comment-16999837
]
ZhouKang edited comment on KYLIN-4227 at 12/19/19 8:30 AM:
---
HTTP callback settings screen shot. see: Attachments
HTTP callback request :
{code:java}
// code placeholder
GET /test?cube=kylin_sales_cube&date_start=132537600&date_end=157515840
HTTP/1.1
Host: {host}:
Connection: Keep-Alive
{code}
was (Author: zhoukangcn):
HTTP callback settings and HTTP request screen shot
!1.png!
> add segment build callback
> --
>
> Key: KYLIN-4227
> URL: https://issues.apache.org/jira/browse/KYLIN-4227
> Project: Kylin
> Issue Type: New Feature
>Reporter: ZhouKang
>Priority: Major
> Attachments: 1.png, 2.png
>
>
> add HTTP callback for cube, some other system could use this feature to add a
> hook when segment building done.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4227) add segment build callback
[ https://issues.apache.org/jira/browse/KYLIN-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999837#comment-16999837 ] ZhouKang commented on KYLIN-4227: - HTTP callback settings and HTTP request screen shot !1.png! > add segment build callback > -- > > Key: KYLIN-4227 > URL: https://issues.apache.org/jira/browse/KYLIN-4227 > Project: Kylin > Issue Type: New Feature >Reporter: ZhouKang >Priority: Major > Attachments: 1.png, 2.png > > > add HTTP callback for cube, some other system could use this feature to add a > hook when segment building done. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4227) add segment build callback
[ https://issues.apache.org/jira/browse/KYLIN-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4227: Attachment: 2.png 1.png > add segment build callback > -- > > Key: KYLIN-4227 > URL: https://issues.apache.org/jira/browse/KYLIN-4227 > Project: Kylin > Issue Type: New Feature >Reporter: ZhouKang >Priority: Major > Attachments: 1.png, 2.png > > > add HTTP callback for cube, some other system could use this feature to add a > hook when segment building done. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4306) rollback when addModelToProject failed.
ZhouKang created KYLIN-4306:
---
Summary: rollback when addModelToProject failed.
Key: KYLIN-4306
URL: https://issues.apache.org/jira/browse/KYLIN-4306
Project: Kylin
Issue Type: Sub-task
Reporter: ZhouKang
write conflict while add model by different query server parallelly, and
addModelToProject failed.
In KylinClusterHealthCheck, there are some error log caused by this bug:
{code:java}
// code placeholder
Error loading DataModelDesc at /model_desc/model_{xxx}.json
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4293) Backport HBASE-22887 to Kylin HFileOutputFormat3
[ https://issues.apache.org/jira/browse/KYLIN-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993121#comment-16993121 ] ZhouKang commented on KYLIN-4293: - +1 > Backport HBASE-22887 to Kylin HFileOutputFormat3 > > > Key: KYLIN-4293 > URL: https://issues.apache.org/jira/browse/KYLIN-4293 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shao Feng Shi >Assignee: langdamao >Priority: Major > Fix For: v2.6.5, v3.1.0 > > > As Kylin forked HBase's HFileOutputFormat2 as HFileOutputFormat3, so this > bugfix need be applied in Kylin: > https://issues.apache.org/jira/browse/HBASE-22887 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4283) Job error because of FileNotFound in Step Garbage Collection
ZhouKang created KYLIN-4283:
---
Summary: Job error because of FileNotFound in Step Garbage
Collection
Key: KYLIN-4283
URL: https://issues.apache.org/jira/browse/KYLIN-4283
Project: Kylin
Issue Type: Bug
Reporter: ZhouKang
Job failed because of FileNotFound.
I think Garbage Collection failed should not make the build job failed
{code:java}
// code placeholder
java.io.FileNotFoundException: File
/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-ff3f6fc3-2211-3593-1ba4-9b5326a05258/cube
does not exist.
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:773)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:831)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:827)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:838)
at
org.apache.kylin.storage.hbase.steps.HDFSPathGarbageCollectionStep.dropHdfsPathOnCluster(HDFSPathGarbageCollectionStep.java:95)
at
org.apache.kylin.storage.hbase.steps.HDFSPathGarbageCollectionStep.doWork(HDFSPathGarbageCollectionStep.java:65)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4272) problems of docker/build_image.sh
ZhouKang created KYLIN-4272:
---
Summary: problems of docker/build_image.sh
Key: KYLIN-4272
URL: https://issues.apache.org/jira/browse/KYLIN-4272
Project: Kylin
Issue Type: Bug
Reporter: ZhouKang
this script can only be executed in sub dir "docker", if your want to execute
as following, some error got:
{code:java}
// code placeholder
bash docker/build_image.sh{code}
And, the source code's dir name must be kylin, you cannot use other name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4178) Job scheduler support safe mode
[ https://issues.apache.org/jira/browse/KYLIN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982364#comment-16982364 ] ZhouKang edited comment on KYLIN-4178 at 11/26/19 11:12 AM: It does have the problem. Have you submited a PR to fix it? or i can fix it later. was (Author: zhoukangcn): It does have the problem. Have you submit a PR to fix it? or i can fix it later. > Job scheduler support safe mode > > > Key: KYLIN-4178 > URL: https://issues.apache.org/jira/browse/KYLIN-4178 > Project: Kylin > Issue Type: Improvement >Reporter: ZhouKang >Priority: Major > Fix For: v3.0.0 > > Attachments: image-2019-11-26-18-42-33-473.png > > > Job scheduler should support safe mode in case of the HBase cluster change. > In xiaomi, we want update the HBase cluster from hbase0.98 to HBase 2.0. The > history data can be migrated previously, but the job has been submitted will > keep running and write data to the old cluster. So we need a method to ensure > that, job create htable in the old cluster will write data to the old > cluster, and the job have not create htable should not be scheduled. > So we need job scheduler safe mode. Open safe mode before changing cluster > config, the running jobs can run continuous, and the new job cannot be > scheduled. > After all running job finished, we can change the cluster config to the new > one, and rest of job can be scheduled again. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4178) Job scheduler support safe mode
[ https://issues.apache.org/jira/browse/KYLIN-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982364#comment-16982364 ] ZhouKang commented on KYLIN-4178: - It does have the problem. Have you submit a PR to fix it? or i can fix it later. > Job scheduler support safe mode > > > Key: KYLIN-4178 > URL: https://issues.apache.org/jira/browse/KYLIN-4178 > Project: Kylin > Issue Type: Improvement >Reporter: ZhouKang >Priority: Major > Fix For: v3.0.0 > > Attachments: image-2019-11-26-18-42-33-473.png > > > Job scheduler should support safe mode in case of the HBase cluster change. > In xiaomi, we want update the HBase cluster from hbase0.98 to HBase 2.0. The > history data can be migrated previously, but the job has been submitted will > keep running and write data to the old cluster. So we need a method to ensure > that, job create htable in the old cluster will write data to the old > cluster, and the job have not create htable should not be scheduled. > So we need job scheduler safe mode. Open safe mode before changing cluster > config, the running jobs can run continuous, and the new job cannot be > scheduled. > After all running job finished, we can change the cluster config to the new > one, and rest of job can be scheduled again. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4268) build job failed caused by hadoop utils update
[
https://issues.apache.org/jira/browse/KYLIN-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang updated KYLIN-4268:
Summary: build job failed caused by hadoop utils update (was: build job
failed caused by java lib update)
> build job failed caused by hadoop utils update
> --
>
> Key: KYLIN-4268
> URL: https://issues.apache.org/jira/browse/KYLIN-4268
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> When we update hadoop package, such as hbase and hive. The job will fill
> cause by:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.io.FileNotFoundException: File
> file:/home/work/app/kylin-all-2.5.2-2.5.2-HBase2-SNAPSHOT/hbase-2.2.0-3.1.0/lib/hbase-common-2.2.0-3.1.0.jar
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
> at
> org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:406)
> at
> org.apache.spark.deploy.yarn.Client.org\$apache\$spark\$deploy\$yarn\$Client\$\$distribute\
> {code}
>
> The root cause is kylin will generate the full command as soon as the job
> summited. And the parameter '--jars' use file's absolute path.
>
> there will be 2 ways to fix this problem.
> # keep the old package in the same path
> # change the generation of '--jars' to the stage of job execution.
>
> which is the better choose?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4268) build job failed caused by java lib update
ZhouKang created KYLIN-4268:
---
Summary: build job failed caused by java lib update
Key: KYLIN-4268
URL: https://issues.apache.org/jira/browse/KYLIN-4268
Project: Kylin
Issue Type: Bug
Reporter: ZhouKang
When we update hadoop package, such as hbase and hive. The job will fill cause
by:
{code:java}
// code placeholder
Exception in thread "main" java.io.FileNotFoundException: File
file:/home/work/app/kylin-all-2.5.2-2.5.2-HBase2-SNAPSHOT/hbase-2.2.0-3.1.0/lib/hbase-common-2.2.0-3.1.0.jar
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
at
org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:406)
at
org.apache.spark.deploy.yarn.Client.org\$apache\$spark\$deploy\$yarn\$Client\$\$distribute\
{code}
The root cause is kylin will generate the full command as soon as the job
summited. And the parameter '--jars' use file's absolute path.
there will be 2 ways to fix this problem.
# keep the old package in the same path
# change the generation of '--jars' to the stage of job execution.
which is the better choose?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4268) build job failed caused by java lib update
[
https://issues.apache.org/jira/browse/KYLIN-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang updated KYLIN-4268:
Description:
When we update hadoop package, such as hbase and hive. The job will fill cause
by:
{code:java}
// code placeholder
Exception in thread "main" java.io.FileNotFoundException: File
file:/home/work/app/kylin-all-2.5.2-2.5.2-HBase2-SNAPSHOT/hbase-2.2.0-3.1.0/lib/hbase-common-2.2.0-3.1.0.jar
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
at
org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:406)
at
org.apache.spark.deploy.yarn.Client.org\$apache\$spark\$deploy\$yarn\$Client\$\$distribute\
{code}
The root cause is kylin will generate the full command as soon as the job
summited. And the parameter '--jars' use file's absolute path.
there will be 2 ways to fix this problem.
# keep the old package in the same path
# change the generation of '--jars' to the stage of job execution.
which is the better choose?
was:
When we update hadoop package, such as hbase and hive. The job will fill cause
by:
{code:java}
// code placeholder
Exception in thread "main" java.io.FileNotFoundException: File
file:/home/work/app/kylin-all-2.5.2-2.5.2-HBase2-SNAPSHOT/hbase-2.2.0-3.1.0/lib/hbase-common-2.2.0-3.1.0.jar
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
at
org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:406)
at
org.apache.spark.deploy.yarn.Client.org\$apache\$spark\$deploy\$yarn\$Client\$\$distribute\
{code}
The root cause is kylin will generate the full command as soon as the job
summited. And the parameter '--jars' use file's absolute path.
there will be 2 ways to fix this problem.
# keep the old package in the same path
# change the generation of '--jars' to the stage of job execution.
which is the better choose?
> build job failed caused by java lib update
> --
>
> Key: KYLIN-4268
> URL: https://issues.apache.org/jira/browse/KYLIN-4268
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> When we update hadoop package, such as hbase and hive. The job will fill
> cause by:
>
> {code:java}
> // code placeholder
> Exception in thread "main" java.io.FileNotFoundException: File
> file:/home/work/app/kylin-all-2.5.2-2.5.2-HBase2-SNAPSHOT/hbase-2.2.0-3.1.0/lib/hbase-common-2.2.0-3.1.0.jar
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
> at
> org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:406)
> at
> org.apache.spark.deploy.yarn.Client.org\$apache\$spark\$deploy\$yarn\$Client\$\$distribute\
> {code}
>
> The root cause is kylin will generate the full command as soon as the job
> summited. And the parameter '--jars' use file's absolute path.
>
> there will be 2 ways to fix this problem.
> # keep the old package in the same path
> # change the generation of '--jars' to the stage of job execution.
>
> which is the better choose?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4267) Add doc for KYLIN-4178
[ https://issues.apache.org/jira/browse/KYLIN-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4267: Issue Type: Task (was: Bug) > Add doc for KYLIN-4178 > -- > > Key: KYLIN-4267 > URL: https://issues.apache.org/jira/browse/KYLIN-4267 > Project: Kylin > Issue Type: Task >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4267) Add doc for KYLIN-4178
ZhouKang created KYLIN-4267: --- Summary: Add doc for KYLIN-4178 Key: KYLIN-4267 URL: https://issues.apache.org/jira/browse/KYLIN-4267 Project: Kylin Issue Type: Bug Reporter: ZhouKang Assignee: ZhouKang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4233) simple time schedule module for kylin job
[ https://issues.apache.org/jira/browse/KYLIN-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang reassigned KYLIN-4233: --- Assignee: ZhouKang > simple time schedule module for kylin job > - > > Key: KYLIN-4233 > URL: https://issues.apache.org/jira/browse/KYLIN-4233 > Project: Kylin > Issue Type: New Feature >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > > add a simple time schedule module, so user can use kylin instead of > third-party schedule system -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4244) ClassNotFoundException while use org.apache.kylin.engine.mr.common.CubeStatsReader in bash
[
https://issues.apache.org/jira/browse/KYLIN-4244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang updated KYLIN-4244:
Description:
use org.apache.kylin.engine.mr.common.CubeStatsReader to print estimated size
for cube
{code:java}
// code placeholder
bash ./kylin.sh org.apache.kylin.engine.mr.common.CubeStatsReader {cube_name}
{code}
get an Exception
{code:java}
// code placeholder
Exception in thread "main" java.lang.NoClassDefFoundError:
com/tdunning/math/stats/TDigest
at
org.apache.kylin.measure.percentile.PercentileSerializer.current(PercentileSerializer.java:62)
at
org.apache.kylin.measure.percentile.PercentileSerializer.getStorageBytesEstimate(PercentileSerializer.java:52)
at
org.apache.kylin.metadata.datatype.DataType.getStorageBytesEstimate(DataType.java:256)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.estimateCuboidStorageSize(CubeStatsReader.java:251)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMapFromRowCount(CubeStatsReader.java:211)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMap(CubeStatsReader.java:170)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.print(CubeStatsReader.java:273)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.main(CubeStatsReader.java:435)
Caused by: java.lang.ClassNotFoundException: com.tdunning.math.stats.TDigest
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 8 more
{code}
was:
use kylin.sh
{code:java}
// code placeholder
bash ./kylin.sh org.apache.kylin.engine.mr.common.CubeStatsReader
caifeng_usage_cube
{code}
get an Exception
{code:java}
// code placeholder
Exception in thread "main" java.lang.NoClassDefFoundError:
com/tdunning/math/stats/TDigest
at
org.apache.kylin.measure.percentile.PercentileSerializer.current(PercentileSerializer.java:62)
at
org.apache.kylin.measure.percentile.PercentileSerializer.getStorageBytesEstimate(PercentileSerializer.java:52)
at
org.apache.kylin.metadata.datatype.DataType.getStorageBytesEstimate(DataType.java:256)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.estimateCuboidStorageSize(CubeStatsReader.java:251)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMapFromRowCount(CubeStatsReader.java:211)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMap(CubeStatsReader.java:170)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.print(CubeStatsReader.java:273)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.main(CubeStatsReader.java:435)
Caused by: java.lang.ClassNotFoundException: com.tdunning.math.stats.TDigest
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 8 more
{code}
> ClassNotFoundException while use
> org.apache.kylin.engine.mr.common.CubeStatsReader in bash
> --
>
> Key: KYLIN-4244
> URL: https://issues.apache.org/jira/browse/KYLIN-4244
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> use org.apache.kylin.engine.mr.common.CubeStatsReader to print estimated size
> for cube
>
> {code:java}
> // code placeholder
> bash ./kylin.sh org.apache.kylin.engine.mr.common.CubeStatsReader {cube_name}
> {code}
> get an Exception
> {code:java}
> // code placeholder
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/tdunning/math/stats/TDigest
> at
> org.apache.kylin.measure.percentile.PercentileSerializer.current(PercentileSerializer.java:62)
> at
> org.apache.kylin.measure.percentile.PercentileSerializer.getStorageBytesEstimate(PercentileSerializer.java:52)
> at
> org.apache.kylin.metadata.datatype.DataType.getStorageBytesEstimate(DataType.java:256)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.estimateCuboidStorageSize(CubeStatsReader.java:251)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMapFromRowCount(CubeStatsReader.java:211)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMap(CubeStatsReader.java:170)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.print(CubeStatsReader.java:273)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.main(CubeStatsReader.java:435)
> Caused by: java.lang.ClassNotFoundE
[jira] [Created] (KYLIN-4244) ClassNotFoundException while use org.apache.kylin.engine.mr.common.CubeStatsReader in bash
ZhouKang created KYLIN-4244:
---
Summary: ClassNotFoundException while use
org.apache.kylin.engine.mr.common.CubeStatsReader in bash
Key: KYLIN-4244
URL: https://issues.apache.org/jira/browse/KYLIN-4244
Project: Kylin
Issue Type: Bug
Reporter: ZhouKang
use kylin.sh
{code:java}
// code placeholder
bash ./kylin.sh org.apache.kylin.engine.mr.common.CubeStatsReader
caifeng_usage_cube
{code}
get an Exception
{code:java}
// code placeholder
Exception in thread "main" java.lang.NoClassDefFoundError:
com/tdunning/math/stats/TDigest
at
org.apache.kylin.measure.percentile.PercentileSerializer.current(PercentileSerializer.java:62)
at
org.apache.kylin.measure.percentile.PercentileSerializer.getStorageBytesEstimate(PercentileSerializer.java:52)
at
org.apache.kylin.metadata.datatype.DataType.getStorageBytesEstimate(DataType.java:256)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.estimateCuboidStorageSize(CubeStatsReader.java:251)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMapFromRowCount(CubeStatsReader.java:211)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMap(CubeStatsReader.java:170)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.print(CubeStatsReader.java:273)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.main(CubeStatsReader.java:435)
Caused by: java.lang.ClassNotFoundException: com.tdunning.math.stats.TDigest
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 8 more
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (KYLIN-4241) Dynamically reload user's security configuration with properties or yaml config
[ https://issues.apache.org/jira/browse/KYLIN-4241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4241: Comment: was deleted (was: I think KYLIN-4212 is a better way) > Dynamically reload user's security configuration with properties or yaml > config > --- > > Key: KYLIN-4241 > URL: https://issues.apache.org/jira/browse/KYLIN-4241 > Project: Kylin > Issue Type: Improvement > Components: Security >Affects Versions: v2.6.0 > Environment: testing >Reporter: xiang zhang >Priority: Major > Fix For: Future > > Attachments: kylin-4241-usage-instructions.pdf > > > In the default kylin security authentication configuration, the user needs to > configure the user and password in kylinSecurity.xml. The ability to > dynamically load user permissions, authentication passwords, etc., increases > convenience and scalability. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4241) Dynamically reload user's security configuration with properties or yaml config
[ https://issues.apache.org/jira/browse/KYLIN-4241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16968959#comment-16968959 ] ZhouKang commented on KYLIN-4241: - I think KYLIN-4212 is a better way > Dynamically reload user's security configuration with properties or yaml > config > --- > > Key: KYLIN-4241 > URL: https://issues.apache.org/jira/browse/KYLIN-4241 > Project: Kylin > Issue Type: Improvement > Components: Security >Affects Versions: v2.6.0 > Environment: testing >Reporter: xiang zhang >Priority: Major > Fix For: Future > > > In the default kylin security authentication configuration, the user needs to > configure the user and password in kylinSecurity.xml. The ability to > dynamically load user permissions, authentication passwords, etc., increases > convenience and scalability. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4185) CubeStatsReader estimate wrong cube size
[ https://issues.apache.org/jira/browse/KYLIN-4185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16968205#comment-16968205 ] ZhouKang commented on KYLIN-4185: - In our production environment, the estimated size and real size is as follows： cube_sample, segment: 2019110200_2019110300 estimate size: 1736731MB HTable size: 190701MB cube_sample, segment: 2019100100_2019100800 (this is a merged segemtn) estimate size: 49251MB HTable size: 1153314MB > CubeStatsReader estimate wrong cube size > > > Key: KYLIN-4185 > URL: https://issues.apache.org/jira/browse/KYLIN-4185 > Project: Kylin > Issue Type: Improvement >Reporter: ZhouKang >Priority: Major > > CubeStatsReader estimate wrong cube size, which cause a lot of problems. > when the estimated size is much larger than the real size, the spark > application's executor number is small, and cube build step will take a long > time. sometime the step will failed due to the large dataset. > When the estimated size is much smaller than the real size. the cuboid file > in HDFS is small, and there are much of cuboid file. > > In our production environment, both the two situation happened. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4232) add sql function date_from_now
[ https://issues.apache.org/jira/browse/KYLIN-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4232: Description: SELECT "DATE", count(1) from table WHERE "DATE" = 'date_from_now(-1, '-MM-dd') group by "DATE" (was: SELECT \"DATE\", count(1) from table WHERE \"DATE\" = ''date_from_now(-1, '-MM-dd') group by \"DATE\") > add sql function date_from_now > --- > > Key: KYLIN-4232 > URL: https://issues.apache.org/jira/browse/KYLIN-4232 > Project: Kylin > Issue Type: New Feature >Reporter: ZhouKang >Priority: Major > > SELECT "DATE", count(1) from table WHERE "DATE" = 'date_from_now(-1, > '-MM-dd') group by "DATE" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4232) add sql function date_from_now
[ https://issues.apache.org/jira/browse/KYLIN-4232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4232: Description: SELECT "DATE", count(1) from table WHERE "DATE" = date_from_now(-1, '-MM-dd') group by "DATE" (was: SELECT "DATE", count(1) from table WHERE "DATE" = 'date_from_now(-1, '-MM-dd') group by "DATE") > add sql function date_from_now > --- > > Key: KYLIN-4232 > URL: https://issues.apache.org/jira/browse/KYLIN-4232 > Project: Kylin > Issue Type: New Feature >Reporter: ZhouKang >Priority: Major > > SELECT "DATE", count(1) from table WHERE "DATE" = date_from_now(-1, > '-MM-dd') group by "DATE" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4234) add a project extractor for Kylin admin
ZhouKang created KYLIN-4234: --- Summary: add a project extractor for Kylin admin Key: KYLIN-4234 URL: https://issues.apache.org/jira/browse/KYLIN-4234 Project: Kylin Issue Type: New Feature Reporter: ZhouKang the data in Kylin metastore is not relational, so when there are many projects or models in the system, it's hard to get all information. sometimes we need feature such as: * project's email list * cube list and status (last modify time) Suggestions are welcomed, thanks~ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4233) simple time schedule module for kylin job
ZhouKang created KYLIN-4233: --- Summary: simple time schedule module for kylin job Key: KYLIN-4233 URL: https://issues.apache.org/jira/browse/KYLIN-4233 Project: Kylin Issue Type: New Feature Reporter: ZhouKang add a simple time schedule module, so user can use kylin instead of third-party schedule system -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4232) add sql function date_from_now
ZhouKang created KYLIN-4232: --- Summary: add sql function date_from_now Key: KYLIN-4232 URL: https://issues.apache.org/jira/browse/KYLIN-4232 Project: Kylin Issue Type: New Feature Reporter: ZhouKang SELECT \"DATE\", count(1) from table WHERE \"DATE\" = ''date_from_now(-1, '-MM-dd') group by \"DATE\" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4231) write conflict while add model by different query server parallelly
ZhouKang created KYLIN-4231:
---
Summary: write conflict while add model by different query server
parallelly
Key: KYLIN-4231
URL: https://issues.apache.org/jira/browse/KYLIN-4231
Project: Kylin
Issue Type: New Feature
Reporter: ZhouKang
a kylin cluster have more than 1 query server, all of them are the backend
server of nginx.
when our user use RESTful API to create model in the *same* project parallelly,
there will be a problem.
the server returns:
{code:java}
// code placeholder
Overwriting conflict /project/learn_kylin.json, expect old TS 1572596034269,
but it is 1572596042929
{code}
BUT, the model '/model_desc/xxx.json' has already in metastore, so the next
time our user want
to retry (create model), he will get:
{code:java}
// code placeholder
Overwriting conflict /model_desc/test_945.json, expect old TS 0, but it is
1572596193812
{code}
I think the problem is that:
when server A receive PUT model, it will change the project info, but the cache
update broadcast cannot be processed by server B before server B processing
another model creation request.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4227) add segment build callback
ZhouKang created KYLIN-4227: --- Summary: add segment build callback Key: KYLIN-4227 URL: https://issues.apache.org/jira/browse/KYLIN-4227 Project: Kylin Issue Type: New Feature Reporter: ZhouKang add HTTP callback for cube, some other system could use this feature to add a hook when segment building done. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4226) Skip current unavailable tables when updating hbase coprocessor
ZhouKang created KYLIN-4226:
---
Summary: Skip current unavailable tables when updating hbase
coprocessor
Key: KYLIN-4226
URL: https://issues.apache.org/jira/browse/KYLIN-4226
Project: Kylin
Issue Type: Improvement
Reporter: ZhouKang
when update hbase coprocessor for htable, there are exceptions cause by
unavailable tables.
{code:java}
// code placeholder
2019-08-22 09:12:20,445 INFO [main] util.DeployCoprocessorCLI:181 : Commit
Information: 2945cb67bae6242f275a038e7bf872a7a9797464;
Exception in thread "main" org.apache.hadoop.hbase.TableNotFoundException:
kylin_zjyprc_bigdata_staging:KYLIN_UCHB8D2MVP
at
org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:567)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getDescriptor(HBaseAdmin.java:365)
at
org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.filterByGitCommit(DeployCoprocessorCLI.java:183)
at
org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.main(DeployCoprocessorCLI.java:131)
2019-08-22 09:12:21,207 INFO [close-hbase-conn] hbase.HBaseConnection:137 :
Closing HBase connections...
2019-08-22 09:12:21,208 INFO [close-hbase-conn]
client.ConnectionImplementation:1889 : Closing master protocol: MasterService
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4225) unclosed hive session cause too many temp file
[
https://issues.apache.org/jira/browse/KYLIN-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang updated KYLIN-4225:
Description:
in org.apache.kylin.metrics.lib.impl.hive.HiveProducer, when need add a new
partition to metrics data, a SessionState will be started , and unclosed after
use.
which cause many temp dir in HDFS (\{hadoop.tmp.dir}/tmp/hive/xxx).
I think the session should be stoped manually
was:
in org.apache.kylin.metrics.lib.impl.hive.HiveProducer, when need add a new
partition to metrics data, a SessionState will be started , and unclosed after
use.
which cause many temp dir in HDFS (\{ROOT}/tmp/hive/xxx).
I think the session should be stoped manually
> unclosed hive session cause too many temp file
> --
>
> Key: KYLIN-4225
> URL: https://issues.apache.org/jira/browse/KYLIN-4225
> Project: Kylin
> Issue Type: Improvement
>Reporter: ZhouKang
>Priority: Major
>
> in org.apache.kylin.metrics.lib.impl.hive.HiveProducer, when need add a new
> partition to metrics data, a SessionState will be started , and unclosed
> after use.
> which cause many temp dir in HDFS (\{hadoop.tmp.dir}/tmp/hive/xxx).
> I think the session should be stoped manually
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4225) unclosed hive session cause too many temp file
ZhouKang created KYLIN-4225:
---
Summary: unclosed hive session cause too many temp file
Key: KYLIN-4225
URL: https://issues.apache.org/jira/browse/KYLIN-4225
Project: Kylin
Issue Type: Improvement
Reporter: ZhouKang
in org.apache.kylin.metrics.lib.impl.hive.HiveProducer, when need add a new
partition to metrics data, a SessionState will be started , and unclosed after
use.
which cause many temp dir in HDFS (\{ROOT}/tmp/hive/xxx).
I think the session should be stoped manually
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4139) Compatible old user security xml config when user upgrate new kylin version
[ https://issues.apache.org/jira/browse/KYLIN-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961967#comment-16961967 ] ZhouKang commented on KYLIN-4139: - I notice that, in KYLIN-3531 (https://issues.apache.org/jira/browse/KYLIN-3531) save uppercase username in metadata, and then in KYLIN-4103 (https://issues.apache.org/jira/browse/KYLIN-4103) make user name in granting operation of project is case insensitive but in this issue, username in metadata is original. what is the standard rule for username? > Compatible old user security xml config when user upgrate new kylin version > > > Key: KYLIN-4139 > URL: https://issues.apache.org/jira/browse/KYLIN-4139 > Project: Kylin > Issue Type: Improvement >Reporter: luguosheng >Assignee: luguosheng >Priority: Major > Fix For: v3.0.0-beta > > > > Forward compatible > # when user config 'spring.profiles.active=testing', sync old version user > metadata with user config in kylinSecurity.xml > Other improvement > # Remove the logic of auto trans new user name to uppercase -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4139) Compatible old user security xml config when user upgrate new kylin version
[ https://issues.apache.org/jira/browse/KYLIN-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961967#comment-16961967 ] ZhouKang edited comment on KYLIN-4139 at 10/29/19 12:38 PM: I notice that, in KYLIN-3531 save uppercase username in metadata, and then in KYLIN-4103 make user name in granting operation of project is case insensitive but in this issue, username in metadata is original. what is the standard rule for username? was (Author: zhoukangcn): I notice that, in KYLIN-3531 (https://issues.apache.org/jira/browse/KYLIN-3531) save uppercase username in metadata, and then in KYLIN-4103 (https://issues.apache.org/jira/browse/KYLIN-4103) make user name in granting operation of project is case insensitive but in this issue, username in metadata is original. what is the standard rule for username? > Compatible old user security xml config when user upgrate new kylin version > > > Key: KYLIN-4139 > URL: https://issues.apache.org/jira/browse/KYLIN-4139 > Project: Kylin > Issue Type: Improvement >Reporter: luguosheng >Assignee: luguosheng >Priority: Major > Fix For: v3.0.0-beta > > > > Forward compatible > # when user config 'spring.profiles.active=testing', sync old version user > metadata with user config in kylinSecurity.xml > Other improvement > # Remove the logic of auto trans new user name to uppercase -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4207) CubeMetadataValidator exception
[
https://issues.apache.org/jira/browse/KYLIN-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16960801#comment-16960801
]
ZhouKang commented on KYLIN-4207:
-
see: [https://github.com/apache/kylin/pull/900]
> CubeMetadataValidator exception
> ---
>
> Key: KYLIN-4207
> URL: https://issues.apache.org/jira/browse/KYLIN-4207
> Project: Kylin
> Issue Type: Improvement
>Reporter: ZhouKang
>Priority: Major
>
> {code:java}
> // code placeholder
> 2019-10-28 11:19:08,583 ERROR [http-nio-7070-exec-9]
> validation.CubeMetadataValidator:56 : Construct cube metadata validator rule:
> org.apache.kylin.cube.model.validation.rule.CubeInfoRule failed. Ignore this
> rule
> java.lang.UnsupportedOperationException
> at java.util.AbstractList.add(AbstractList.java:148)
> at java.util.AbstractList.add(AbstractList.java:108)
> at
> org.apache.kylin.cube.model.validation.CubeMetadataValidator.(CubeMetadataValidator.java:54)
> {code}
> should not use Arrays.asList() in CubeMetadataValidator.java:49, which will
> cause UnsupportedOperationExcepiton
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4207) CubeMetadataValidator exception
[
https://issues.apache.org/jira/browse/KYLIN-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang updated KYLIN-4207:
Description:
{code:java}
// code placeholder
2019-10-28 11:19:08,583 ERROR [http-nio-7070-exec-9]
validation.CubeMetadataValidator:56 : Construct cube metadata validator rule:
org.apache.kylin.cube.model.validation.rule.CubeInfoRule failed. Ignore this
rule
java.lang.UnsupportedOperationException
at java.util.AbstractList.add(AbstractList.java:148)
at java.util.AbstractList.add(AbstractList.java:108)
at
org.apache.kylin.cube.model.validation.CubeMetadataValidator.(CubeMetadataValidator.java:54)
{code}
should not use Arrays.asList() in CubeMetadataValidator.java:49, which will
cause UnsupportedOperationExcepiton
was:
{code:java}
// code placeholder
2019-10-28 11:19:08,583 ERROR [http-nio-7070-exec-9]
validation.CubeMetadataValidator:56 : Construct cube metadata validator rule:
org.apache.kylin.cube.model.validation.rule.CubeInfoRule failed. Ignore this
rule
java.lang.UnsupportedOperationException
at java.util.AbstractList.add(AbstractList.java:148)
at java.util.AbstractList.add(AbstractList.java:108)
at
org.apache.kylin.cube.model.validation.CubeMetadataValidator.(CubeMetadataValidator.java:54)
{code}
should not use Arrays.asList() in CubeMetadataValidator.java:49, which will
cause UnsupportedOperationExcepiton
it is
> CubeMetadataValidator exception
> ---
>
> Key: KYLIN-4207
> URL: https://issues.apache.org/jira/browse/KYLIN-4207
> Project: Kylin
> Issue Type: Improvement
>Reporter: ZhouKang
>Priority: Major
>
> {code:java}
> // code placeholder
> 2019-10-28 11:19:08,583 ERROR [http-nio-7070-exec-9]
> validation.CubeMetadataValidator:56 : Construct cube metadata validator rule:
> org.apache.kylin.cube.model.validation.rule.CubeInfoRule failed. Ignore this
> rule
> java.lang.UnsupportedOperationException
> at java.util.AbstractList.add(AbstractList.java:148)
> at java.util.AbstractList.add(AbstractList.java:108)
> at
> org.apache.kylin.cube.model.validation.CubeMetadataValidator.(CubeMetadataValidator.java:54)
> {code}
> should not use Arrays.asList() in CubeMetadataValidator.java:49, which will
> cause UnsupportedOperationExcepiton
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4185) CubeStatsReader estimate wrong cube size
ZhouKang created KYLIN-4185: --- Summary: CubeStatsReader estimate wrong cube size Key: KYLIN-4185 URL: https://issues.apache.org/jira/browse/KYLIN-4185 Project: Kylin Issue Type: Improvement Reporter: ZhouKang CubeStatsReader estimate wrong cube size, which cause a lot of problems. when the estimated size is much larger than the real size, the spark application's executor number is small, and cube build step will take a long time. sometime the step will failed due to the large dataset. When the estimated size is much smaller than the real size. the cuboid file in HDFS is small, and there are much of cuboid file. In our production environment, both the two situation happened. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4174) job statistics info is incorrect
[
https://issues.apache.org/jira/browse/KYLIN-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939145#comment-16939145
]
ZhouKang commented on KYLIN-4174:
-
[~wangrupeng]
sorry, miss description in issue. you should choose a project first.
> job statistics info is incorrect
>
>
> Key: KYLIN-4174
> URL: https://issues.apache.org/jira/browse/KYLIN-4174
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Assignee: luguosheng
>Priority: Major
> Attachments: image-2019-09-27-12-27-50-336.png,
> image-2019-09-27-12-27-50-393.png, image3.png
>
>
> Kylin: master branch
> The job statistics info is incorrect. The number behind type is not equal to
> the size of job list.
> Step 1: login kylin webui, open the Monitor
> Step 2: refresh the web page
>
> You can see, the query params of RESTful API is wrong, projectName is "_null"
> {code:java}
> // code placeholder
> curl
> 'http://zjy-bigdata-prc-kylin04.bj:7070/kylin/api/jobs/overview?jobSearchMode=ALL&projectName=_null&timeFilter=1'
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4178) Job scheduler support safe mode
ZhouKang created KYLIN-4178: --- Summary: Job scheduler support safe mode Key: KYLIN-4178 URL: https://issues.apache.org/jira/browse/KYLIN-4178 Project: Kylin Issue Type: Improvement Reporter: ZhouKang Job scheduler should support safe mode in case of the HBase cluster change. In xiaomi, we want update the HBase cluster from hbase0.98 to HBase 2.0. The history data can be migrated previously, but the job has been submitted will keep running and write data to the old cluster. So we need a method to ensure that, job create htable in the old cluster will write data to the old cluster, and the job have not create htable should not be scheduled. So we need job scheduler safe mode. Open safe mode before changing cluster config, the running jobs can run continuous, and the new job cannot be scheduled. After all running job finished, we can change the cluster config to the new one, and rest of job can be scheduled again. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4174) job statistics info is incorrect
ZhouKang created KYLIN-4174:
---
Summary: job statistics info is incorrect
Key: KYLIN-4174
URL: https://issues.apache.org/jira/browse/KYLIN-4174
Project: Kylin
Issue Type: Bug
Reporter: ZhouKang
Kylin: master branch
The job statistics info is incorrect. The number behind type is not equal to
the size of job list.
Step 1: login kylin webui, open the Monitor
Step 2: refresh the web page
You can see, the query params of RESTful API is wrong, projectName is "_null"
{code:java}
// code placeholder
curl
'http://zjy-bigdata-prc-kylin04.bj:7070/kylin/api/jobs/overview?jobSearchMode=ALL&projectName=_null&timeFilter=1'
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-3973) InvalidProtocolBufferException: Protocol message was too large. May be malicious.
[
https://issues.apache.org/jira/browse/KYLIN-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933326#comment-16933326
]
ZhouKang commented on KYLIN-3973:
-
glad to hear that
> InvalidProtocolBufferException: Protocol message was too large. May be
> malicious.
> --
>
> Key: KYLIN-3973
> URL: https://issues.apache.org/jira/browse/KYLIN-3973
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.1
>Reporter: Grzegorz Kołakowski
>Priority: Major
>
> For many queries I receive the following exception.
> {noformat}
> 2019-04-23 11:33:15,576 WARN [kylin-coproc--pool6-t17]
> client.SyncCoprocessorRpcChannel:54 : Call failed on IOException
> com.google.protobuf.InvalidProtocolBufferException: Protocol message was too
> large. May be malicious. Use CodedInputStream.setSizeLimit() to increase
> the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
> at
> com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse.(CubeVisitProtos.java:2307)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse.(CubeVisitProtos.java:2271)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$1.parsePartialFrom(CubeVisitProtos.java:2380)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$1.parsePartialFrom(CubeVisitProtos.java:2375)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$Builder.mergeFrom(CubeVisitProtos.java:5101)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$Builder.mergeFrom(CubeVisitProtos.java:4949)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:337)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:267)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:210)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:904)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:267)
> at
> org.apache.hadoop.hbase.ipc.CoprocessorRpcUtils.getResponse(CoprocessorRpcUtils.java:141)
> at
> org.apache.hadoop.hbase.client.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:94)
> at
> org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callMethod(SyncCoprocessorRpcChannel.java:52)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:5616)
> at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:246)
> at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:242)
> at org.apache.hadoop.hbase.client.HTable$12.call(HTable.java:1012)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I use lz4 compression algorithm in HBase.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-3973) InvalidProtocolBufferException: Protocol message was too large. May be malicious.
[
https://issues.apache.org/jira/browse/KYLIN-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932395#comment-16932395
]
ZhouKang commented on KYLIN-3973:
-
Try to replace other package contain CubeVisitProtos. I'm not sure which file
it used. you can use java --verbose:class to findout.
Otherwise you can try to replace the whole environment
> InvalidProtocolBufferException: Protocol message was too large. May be
> malicious.
> --
>
> Key: KYLIN-3973
> URL: https://issues.apache.org/jira/browse/KYLIN-3973
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.1
>Reporter: Grzegorz Kołakowski
>Priority: Major
>
> For many queries I receive the following exception.
> {noformat}
> 2019-04-23 11:33:15,576 WARN [kylin-coproc--pool6-t17]
> client.SyncCoprocessorRpcChannel:54 : Call failed on IOException
> com.google.protobuf.InvalidProtocolBufferException: Protocol message was too
> large. May be malicious. Use CodedInputStream.setSizeLimit() to increase
> the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
> at
> com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse.(CubeVisitProtos.java:2307)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse.(CubeVisitProtos.java:2271)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$1.parsePartialFrom(CubeVisitProtos.java:2380)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$1.parsePartialFrom(CubeVisitProtos.java:2375)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$Builder.mergeFrom(CubeVisitProtos.java:5101)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$Builder.mergeFrom(CubeVisitProtos.java:4949)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:337)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:267)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:210)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:904)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:267)
> at
> org.apache.hadoop.hbase.ipc.CoprocessorRpcUtils.getResponse(CoprocessorRpcUtils.java:141)
> at
> org.apache.hadoop.hbase.client.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:94)
> at
> org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callMethod(SyncCoprocessorRpcChannel.java:52)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:5616)
> at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:246)
> at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:242)
> at org.apache.hadoop.hbase.client.HTable$12.call(HTable.java:1012)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I use lz4 compression algorithm in HBase.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4169) too many logs while DataModelManager init, cause the first RESTful API hang for a long time
[
https://issues.apache.org/jira/browse/KYLIN-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932289#comment-16932289
]
ZhouKang edited comment on KYLIN-4169 at 9/18/19 10:39 AM:
---
I think the problem is not caused by un-existed models.
The stack is here:
{code:java}
// code placeholder
at
org.apache.kylin.metadata.model.DataModelManager.getModels(DataModelManager.java:190)
at
org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:92)
at
org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:87)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAt(CachedCrudAssist.java:167)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadQuietlyAt(CachedCrudAssist.java:144)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAll(CachedCrudAssist.java:127)
at
org.apache.kylin.metadata.model.DataModelManager.init(DataModelManager.java:100)
at
org.apache.kylin.metadata.model.DataModelManager.(DataModelManager.java:80)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.kylin.metadata.model.DataModelManager.newInstance(DataModelManager.java:61)
{code}
and the code: CachedCrudAssist.java:125-127
{code:java}
// code placeholder
List paths = store.collectResourceRecursively(resRootPath,
resPathSuffix);
for (String path : paths) {
reloadQuietlyAt(path);
}
{code}
everytime load a single meta record, initEntityAfterReload will scan the whole
project. At this moment, the remain models (which is valid actually) have not
been loaded to the cache.
If there are 100 models in a project, the logs count will be 1+2+3+...+99
was (Author: zhoukangcn):
I think the problem is not caused by un-existed models.
The stack is here:
{code:java}
// code placeholder
at
org.apache.kylin.metadata.model.DataModelManager.getModels(DataModelManager.java:190)
at
org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:92)
at
org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:87)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAt(CachedCrudAssist.java:167)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadQuietlyAt(CachedCrudAssist.java:144)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAll(CachedCrudAssist.java:127)
at
org.apache.kylin.metadata.model.DataModelManager.init(DataModelManager.java:100)
at
org.apache.kylin.metadata.model.DataModelManager.(DataModelManager.java:80)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.kylin.metadata.model.DataModelManager.newInstance(DataModelManager.java:61)
{code}
and the code: CachedCrudAssist.java:125-127
{code:java}
// code placeholder
List paths = store.collectResourceRecursively(resRootPath,
resPathSuffix);
for (String path : paths) {
reloadQuietlyAt(path);
}
{code}
everytime load a single meta record, initEntityAfterReload will scan the whole
project. At this moment, the remain models (which is valid actually) have not
been loaded to the cache.
> too many logs while DataModelManager init, cause the first RESTful API hang
> for a long time
> ---
>
> Key: KYLIN-4169
> URL: https://issues.apache.org/jira/browse/KYLIN-4169
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> Too many logs while DataModelManager init. It is will happend after startup
> or reload metastore.
> In our production enviroment，there are 120,000+ lines.
> {code:java}
> // code placeholder
> grep 'is missing or unloaded yet' kylin.log.8|wc -l
> 121861
> {code}
>
> which cause the first sql query's time up to 40s.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4169) too many logs while DataModelManager init, cause the first RESTful API hang for a long time
[
https://issues.apache.org/jira/browse/KYLIN-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932289#comment-16932289
]
ZhouKang edited comment on KYLIN-4169 at 9/18/19 10:39 AM:
---
I think the problem is not caused by un-existed models.
The stack is here:
{code:java}
// code placeholder
at
org.apache.kylin.metadata.model.DataModelManager.getModels(DataModelManager.java:190)
at
org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:92)
at
org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:87)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAt(CachedCrudAssist.java:167)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadQuietlyAt(CachedCrudAssist.java:144)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAll(CachedCrudAssist.java:127)
at
org.apache.kylin.metadata.model.DataModelManager.init(DataModelManager.java:100)
at
org.apache.kylin.metadata.model.DataModelManager.(DataModelManager.java:80)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.kylin.metadata.model.DataModelManager.newInstance(DataModelManager.java:61)
{code}
and the code: CachedCrudAssist.java:125-127
{code:java}
// code placeholder
List paths = store.collectResourceRecursively(resRootPath,
resPathSuffix);
for (String path : paths) {
reloadQuietlyAt(path);
}
{code}
everytime load a single meta record, initEntityAfterReload will scan the whole
project. At this moment, the remain models (which is valid actually) have not
been loaded to the cache.
If there are 100 models in a project, the logs count will be 99+98+97+...+1
was (Author: zhoukangcn):
I think the problem is not caused by un-existed models.
The stack is here:
{code:java}
// code placeholder
at
org.apache.kylin.metadata.model.DataModelManager.getModels(DataModelManager.java:190)
at
org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:92)
at
org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:87)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAt(CachedCrudAssist.java:167)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadQuietlyAt(CachedCrudAssist.java:144)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAll(CachedCrudAssist.java:127)
at
org.apache.kylin.metadata.model.DataModelManager.init(DataModelManager.java:100)
at
org.apache.kylin.metadata.model.DataModelManager.(DataModelManager.java:80)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.kylin.metadata.model.DataModelManager.newInstance(DataModelManager.java:61)
{code}
and the code: CachedCrudAssist.java:125-127
{code:java}
// code placeholder
List paths = store.collectResourceRecursively(resRootPath,
resPathSuffix);
for (String path : paths) {
reloadQuietlyAt(path);
}
{code}
everytime load a single meta record, initEntityAfterReload will scan the whole
project. At this moment, the remain models (which is valid actually) have not
been loaded to the cache.
If there are 100 models in a project, the logs count will be 1+2+3+...+99
> too many logs while DataModelManager init, cause the first RESTful API hang
> for a long time
> ---
>
> Key: KYLIN-4169
> URL: https://issues.apache.org/jira/browse/KYLIN-4169
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> Too many logs while DataModelManager init. It is will happend after startup
> or reload metastore.
> In our production enviroment，there are 120,000+ lines.
> {code:java}
> // code placeholder
> grep 'is missing or unloaded yet' kylin.log.8|wc -l
> 121861
> {code}
>
> which cause the first sql query's time up to 40s.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4169) too many logs while DataModelManager init, cause the first RESTful API hang for a long time
[
https://issues.apache.org/jira/browse/KYLIN-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932289#comment-16932289
]
ZhouKang commented on KYLIN-4169:
-
I think the problem is not caused by un-existed models.
The stack is here:
{code:java}
// code placeholder
at
org.apache.kylin.metadata.model.DataModelManager.getModels(DataModelManager.java:190)
at
org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:92)
at
org.apache.kylin.metadata.model.DataModelManager$1.initEntityAfterReload(DataModelManager.java:87)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAt(CachedCrudAssist.java:167)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadQuietlyAt(CachedCrudAssist.java:144)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAll(CachedCrudAssist.java:127)
at
org.apache.kylin.metadata.model.DataModelManager.init(DataModelManager.java:100)
at
org.apache.kylin.metadata.model.DataModelManager.(DataModelManager.java:80)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.kylin.metadata.model.DataModelManager.newInstance(DataModelManager.java:61)
{code}
and the code: CachedCrudAssist.java:125-127
{code:java}
// code placeholder
List paths = store.collectResourceRecursively(resRootPath,
resPathSuffix);
for (String path : paths) {
reloadQuietlyAt(path);
}
{code}
everytime load a single meta record, initEntityAfterReload will scan the whole
project. At this moment, the remain models (which is valid actually) have not
been loaded to the cache.
> too many logs while DataModelManager init, cause the first RESTful API hang
> for a long time
> ---
>
> Key: KYLIN-4169
> URL: https://issues.apache.org/jira/browse/KYLIN-4169
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> Too many logs while DataModelManager init. It is will happend after startup
> or reload metastore.
> In our production enviroment，there are 120,000+ lines.
> {code:java}
> // code placeholder
> grep 'is missing or unloaded yet' kylin.log.8|wc -l
> 121861
> {code}
>
> which cause the first sql query's time up to 40s.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-3973) InvalidProtocolBufferException: Protocol message was too large. May be malicious.
[
https://issues.apache.org/jira/browse/KYLIN-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932075#comment-16932075
]
ZhouKang commented on KYLIN-3973:
-
[~shaoxiaowei] If you are using HBase>2.0.0， you can fix it by
[https://github.com/apache/kylin/pull/837]
This a quick fix.
> InvalidProtocolBufferException: Protocol message was too large. May be
> malicious.
> --
>
> Key: KYLIN-3973
> URL: https://issues.apache.org/jira/browse/KYLIN-3973
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.1
>Reporter: Grzegorz Kołakowski
>Priority: Major
>
> For many queries I receive the following exception.
> {noformat}
> 2019-04-23 11:33:15,576 WARN [kylin-coproc--pool6-t17]
> client.SyncCoprocessorRpcChannel:54 : Call failed on IOException
> com.google.protobuf.InvalidProtocolBufferException: Protocol message was too
> large. May be malicious. Use CodedInputStream.setSizeLimit() to increase
> the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
> at
> com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse.(CubeVisitProtos.java:2307)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse.(CubeVisitProtos.java:2271)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$1.parsePartialFrom(CubeVisitProtos.java:2380)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$1.parsePartialFrom(CubeVisitProtos.java:2375)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$Builder.mergeFrom(CubeVisitProtos.java:5101)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$Builder.mergeFrom(CubeVisitProtos.java:4949)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:337)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:267)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:210)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:904)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:267)
> at
> org.apache.hadoop.hbase.ipc.CoprocessorRpcUtils.getResponse(CoprocessorRpcUtils.java:141)
> at
> org.apache.hadoop.hbase.client.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:94)
> at
> org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callMethod(SyncCoprocessorRpcChannel.java:52)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:5616)
> at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:246)
> at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:242)
> at org.apache.hadoop.hbase.client.HTable$12.call(HTable.java:1012)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I use lz4 compression algorithm in HBase.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4169) too many logs while DataModelManager init, cause the first RESTful API hang for a long time
ZhouKang created KYLIN-4169:
---
Summary: too many logs while DataModelManager init, cause the
first RESTful API hang for a long time
Key: KYLIN-4169
URL: https://issues.apache.org/jira/browse/KYLIN-4169
Project: Kylin
Issue Type: Bug
Reporter: ZhouKang
Too many logs while DataModelManager init. It is will happend after startup or
reload metastore.
In our production enviroment，there are 120,000+ lines.
{code:java}
// code placeholder
grep 'is missing or unloaded yet' kylin.log.8|wc -l
121861
{code}
which cause the first sql query's time up to 40s.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4161) exception in update metrics when the response is null
[
https://issues.apache.org/jira/browse/KYLIN-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925296#comment-16925296
]
ZhouKang commented on KYLIN-4161:
-
Maybe this bug is fixed in Kylin:master.
I found that, QueryService use buildSqlResponse()
> exception in update metrics when the response is null
> --
>
> Key: KYLIN-4161
> URL: https://issues.apache.org/jira/browse/KYLIN-4161
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Priority: Major
>
> when something error in sql query, the response‘s cube and results is null,
> which cause an exception.
> {code:java}
> // code placeholder
> service.QueryService:432 : Write metric error.java.lang.NullPointerException
> at
> org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetricsToReservoir(QueryMetricsFacade.java:158)
> at
> org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetrics(QueryMetricsFacade.java:77)
> {code}
> {code:java}
> // code placeholder
> service.QueryService:432 : Write metric error.java.lang.NullPointerException
> at
> org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetricsToReservoir(QueryMetricsFacade.java:89)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (KYLIN-4161) exception in update metrics when the response is null
ZhouKang created KYLIN-4161:
---
Summary: exception in update metrics when the response is null
Key: KYLIN-4161
URL: https://issues.apache.org/jira/browse/KYLIN-4161
Project: Kylin
Issue Type: Bug
Reporter: ZhouKang
when something error in sql query, the response‘s cube and results is null,
which cause an exception.
{code:java}
// code placeholder
service.QueryService:432 : Write metric error.java.lang.NullPointerException
at
org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetricsToReservoir(QueryMetricsFacade.java:158)
at
org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetrics(QueryMetricsFacade.java:77)
{code}
{code:java}
// code placeholder
service.QueryService:432 : Write metric error.java.lang.NullPointerException
at
org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetricsToReservoir(QueryMetricsFacade.java:89)
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Issue Comment Deleted] (KYLIN-3973) InvalidProtocolBufferException: Protocol message was too large. May be malicious.
[
https://issues.apache.org/jira/browse/KYLIN-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang updated KYLIN-3973:
Comment: was deleted
(was: I think CodedInputStream's sizelimit should be seted in CubeVisitProtos. )
> InvalidProtocolBufferException: Protocol message was too large. May be
> malicious.
> --
>
> Key: KYLIN-3973
> URL: https://issues.apache.org/jira/browse/KYLIN-3973
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.1
>Reporter: Grzegorz Kołakowski
>Priority: Major
>
> For many queries I receive the following exception.
> {noformat}
> 2019-04-23 11:33:15,576 WARN [kylin-coproc--pool6-t17]
> client.SyncCoprocessorRpcChannel:54 : Call failed on IOException
> com.google.protobuf.InvalidProtocolBufferException: Protocol message was too
> large. May be malicious. Use CodedInputStream.setSizeLimit() to increase
> the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
> at
> com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse.(CubeVisitProtos.java:2307)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse.(CubeVisitProtos.java:2271)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$1.parsePartialFrom(CubeVisitProtos.java:2380)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$1.parsePartialFrom(CubeVisitProtos.java:2375)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$Builder.mergeFrom(CubeVisitProtos.java:5101)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$Builder.mergeFrom(CubeVisitProtos.java:4949)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:337)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:267)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:210)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:904)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:267)
> at
> org.apache.hadoop.hbase.ipc.CoprocessorRpcUtils.getResponse(CoprocessorRpcUtils.java:141)
> at
> org.apache.hadoop.hbase.client.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:94)
> at
> org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callMethod(SyncCoprocessorRpcChannel.java:52)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:5616)
> at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:246)
> at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:242)
> at org.apache.hadoop.hbase.client.HTable$12.call(HTable.java:1012)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I use lz4 compression algorithm in HBase.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KYLIN-3973) InvalidProtocolBufferException: Protocol message was too large. May be malicious.
[
https://issues.apache.org/jira/browse/KYLIN-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921879#comment-16921879
]
ZhouKang commented on KYLIN-3973:
-
I think CodedInputStream's sizelimit should be seted in CubeVisitProtos.
> InvalidProtocolBufferException: Protocol message was too large. May be
> malicious.
> --
>
> Key: KYLIN-3973
> URL: https://issues.apache.org/jira/browse/KYLIN-3973
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.1
>Reporter: Grzegorz Kołakowski
>Priority: Major
>
> For many queries I receive the following exception.
> {noformat}
> 2019-04-23 11:33:15,576 WARN [kylin-coproc--pool6-t17]
> client.SyncCoprocessorRpcChannel:54 : Call failed on IOException
> com.google.protobuf.InvalidProtocolBufferException: Protocol message was too
> large. May be malicious. Use CodedInputStream.setSizeLimit() to increase
> the size limit.
> at
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
> at
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
> at
> com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
> at
> com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse.(CubeVisitProtos.java:2307)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse.(CubeVisitProtos.java:2271)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$1.parsePartialFrom(CubeVisitProtos.java:2380)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$1.parsePartialFrom(CubeVisitProtos.java:2375)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$Builder.mergeFrom(CubeVisitProtos.java:5101)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitResponse$Builder.mergeFrom(CubeVisitProtos.java:4949)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:337)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:267)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:210)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:904)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:267)
> at
> org.apache.hadoop.hbase.ipc.CoprocessorRpcUtils.getResponse(CoprocessorRpcUtils.java:141)
> at
> org.apache.hadoop.hbase.client.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:94)
> at
> org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callMethod(SyncCoprocessorRpcChannel.java:52)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:5616)
> at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:246)
> at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:242)
> at org.apache.hadoop.hbase.client.HTable$12.call(HTable.java:1012)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I use lz4 compression algorithm in HBase.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KYLIN-4143) truncate spark executable job output
[ https://issues.apache.org/jira/browse/KYLIN-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4143: Attachment: KYLIN-4143.master.001.patch > truncate spark executable job output > - > > Key: KYLIN-4143 > URL: https://issues.apache.org/jira/browse/KYLIN-4143 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: ZhouKang >Priority: Major > Attachments: KYLIN-4143.master.001.patch > > > > truncate spark job's output when the job exec ret is not equal 0, which made > the execute output content too large. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KYLIN-4143) truncate spark executable job output
[ https://issues.apache.org/jira/browse/KYLIN-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhouKang updated KYLIN-4143: Description: truncate spark job's output when the job exec ret is not equal 0, which made the execute output content too large. was: truncate spark job's output when the job exec ret is not equal 0, which made the execute output is too large. > truncate spark executable job output > - > > Key: KYLIN-4143 > URL: https://issues.apache.org/jira/browse/KYLIN-4143 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: ZhouKang >Priority: Major > > > truncate spark job's output when the job exec ret is not equal 0, which made > the execute output content too large. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KYLIN-4143) truncate spark executable job output
[
https://issues.apache.org/jira/browse/KYLIN-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang updated KYLIN-4143:
Description:
truncate spark job's output when the job exec ret is not equal 0, which made
the execute output is too large.
was:
Kylin: 2.5.2
job engine: spark
After long time running, job executor will record plenty of content in job
output file, which make ExecutableManager get output fail. And the job will be
marked as "error".
{code:java}
// code placeholder
2019-08-15 11:58:34,442 ERROR [Scheduler 1278051386 Job
db9b01d0-fd8c-d168-8cfa-2e11d807972c-128] spark.SparkExecutable:359 : error run
spark job:
java.lang.RuntimeException:
org.apache.kylin.job.exception.PersistentException:
com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in
VALUE_STRING
at [Source: (DataInputStream); line: 5, column: 10728969]
at [Source: (DataInputStream); line: 5, column: 15] (through reference chain:
org.apache.kylin.job.dao.ExecutableOutputPO["content"])
at
org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:164)
at
org.apache.kylin.job.execution.AbstractExecutable.getOutput(AbstractExecutable.java:414)
at
org.apache.kylin.job.execution.AbstractExecutable.isDiscarded(AbstractExecutable.java:525)
at
org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:304)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kylin.job.exception.PersistentException:
com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in
VALUE_STRING
at [Source: (DataInputStream); line: 5, column: 10728969]
at [Source: (DataInputStream); line: 5, column: 15] (through reference chain:
org.apache.kylin.job.dao.ExecutableOutputPO["content"])
at org.apache.kylin.job.dao.ExecutableDao.getJobOutput(ExecutableDao.java:356)
at
org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:159)
... 10 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected
end-of-input in VALUE_STRING
at [Source: (DataInputStream); line: 5, column: 10728969]
at [Source: (DataInputStream); line: 5, column: 15] (through reference chain:
org.apache.kylin.job.dao.ExecutableOutputPO["content"])
at
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:391)
at
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:351)
at
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1704)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:290)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058)
at org.apache.kylin.common.util.JsonUtil.readValue(JsonUtil.java:69)
at
org.apache.kylin.common.persistence.JsonSerializer.deserialize(JsonSerializer.java:46)
at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:182)
{code}
> truncate spark executable job output
> -
>
> Key: KYLIN-4143
> URL: https://issues.apache.org/jira/browse/KYLIN-4143
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.5.2
>Reporter: ZhouKang
>Priority: Major
>
>
> truncate spark job's output when the job exec ret is not equal 0, which made
> the execute output is too large.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KYLIN-4143) truncate spark executable job output
[
https://issues.apache.org/jira/browse/KYLIN-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ZhouKang updated KYLIN-4143:
Summary: truncate spark executable job output (was: Get job status fail
when too much content in job output)
> truncate spark executable job output
> -
>
> Key: KYLIN-4143
> URL: https://issues.apache.org/jira/browse/KYLIN-4143
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.5.2
>Reporter: ZhouKang
>Priority: Major
>
> Kylin: 2.5.2
> job engine: spark
>
> After long time running, job executor will record plenty of content in job
> output file, which make ExecutableManager get output fail. And the job will
> be marked as "error".
> {code:java}
> // code placeholder
> 2019-08-15 11:58:34,442 ERROR [Scheduler 1278051386 Job
> db9b01d0-fd8c-d168-8cfa-2e11d807972c-128] spark.SparkExecutable:359 : error
> run spark job:
> java.lang.RuntimeException:
> org.apache.kylin.job.exception.PersistentException:
> com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input
> in VALUE_STRING
> at [Source: (DataInputStream); line: 5, column: 10728969]
> at [Source: (DataInputStream); line: 5, column: 15] (through reference
> chain: org.apache.kylin.job.dao.ExecutableOutputPO["content"])
> at
> org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:164)
> at
> org.apache.kylin.job.execution.AbstractExecutable.getOutput(AbstractExecutable.java:414)
> at
> org.apache.kylin.job.execution.AbstractExecutable.isDiscarded(AbstractExecutable.java:525)
> at
> org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:304)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kylin.job.exception.PersistentException:
> com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input
> in VALUE_STRING
> at [Source: (DataInputStream); line: 5, column: 10728969]
> at [Source: (DataInputStream); line: 5, column: 15] (through reference
> chain: org.apache.kylin.job.dao.ExecutableOutputPO["content"])
> at
> org.apache.kylin.job.dao.ExecutableDao.getJobOutput(ExecutableDao.java:356)
> at
> org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:159)
>
> ... 10 more
> Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected
> end-of-input in VALUE_STRING
> at [Source: (DataInputStream); line: 5, column: 10728969]
> at [Source: (DataInputStream); line: 5, column: 15] (through reference
> chain: org.apache.kylin.job.dao.ExecutableOutputPO["content"])
> at
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:391)
> at
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:351)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1704)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:290)
> at
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
> at
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
> at
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058)
> at org.apache.kylin.common.util.JsonUtil.readValue(JsonUtil.java:69)
> at
> org.apache.kylin.common.persistence.JsonSerializer.deserialize(JsonSerializer.java:46)
> at
> org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:182)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (KYLIN-4144) Cannot update user's info because of case-sensitive process
ZhouKang created KYLIN-4144:
---
Summary: Cannot update user's info because of case-sensitive
process
Key: KYLIN-4144
URL: https://issues.apache.org/jira/browse/KYLIN-4144
Project: Kylin
Issue Type: Bug
Reporter: ZhouKang
Fix For: v2.5.3
In https://issues.apache.org/jira/browse/KYLIN-3531 , uppercase of usernames
will be saved in HBase, which make our case faild.
when users was added useing kylin:v2.4.0 (maybe older), the rawkey in hbase is
lowercase, when update user's info in Kylin:v2.5.3, there is en error.
{code:java}
// code placeholder
Overwriting conflict /user/, expect old TS xxx, but it is 0, the expected
new TS: {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (KYLIN-4143) Get job status fail when too much content in job output
ZhouKang created KYLIN-4143:
---
Summary: Get job status fail when too much content in job output
Key: KYLIN-4143
URL: https://issues.apache.org/jira/browse/KYLIN-4143
Project: Kylin
Issue Type: Bug
Affects Versions: v2.5.2
Reporter: ZhouKang
Kylin: 2.5.2
job engine: spark
After long time running, job executor will record plenty of content in job
output file, which make ExecutableManager get output fail. And the job will be
marked as "error".
{code:java}
// code placeholder
2019-08-15 11:58:34,442 ERROR [Scheduler 1278051386 Job
db9b01d0-fd8c-d168-8cfa-2e11d807972c-128] spark.SparkExecutable:359 : error run
spark job:
java.lang.RuntimeException:
org.apache.kylin.job.exception.PersistentException:
com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in
VALUE_STRING
at [Source: (DataInputStream); line: 5, column: 10728969]
at [Source: (DataInputStream); line: 5, column: 15] (through reference chain:
org.apache.kylin.job.dao.ExecutableOutputPO["content"])
at
org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:164)
at
org.apache.kylin.job.execution.AbstractExecutable.getOutput(AbstractExecutable.java:414)
at
org.apache.kylin.job.execution.AbstractExecutable.isDiscarded(AbstractExecutable.java:525)
at
org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:304)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kylin.job.exception.PersistentException:
com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in
VALUE_STRING
at [Source: (DataInputStream); line: 5, column: 10728969]
at [Source: (DataInputStream); line: 5, column: 15] (through reference chain:
org.apache.kylin.job.dao.ExecutableOutputPO["content"])
at org.apache.kylin.job.dao.ExecutableDao.getJobOutput(ExecutableDao.java:356)
at
org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:159)
... 10 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected
end-of-input in VALUE_STRING
at [Source: (DataInputStream); line: 5, column: 10728969]
at [Source: (DataInputStream); line: 5, column: 15] (through reference chain:
org.apache.kylin.job.dao.ExecutableOutputPO["content"])
at
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:391)
at
com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:351)
at
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1704)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:290)
at
com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058)
at org.apache.kylin.common.util.JsonUtil.readValue(JsonUtil.java:69)
at
org.apache.kylin.common.persistence.JsonSerializer.deserialize(JsonSerializer.java:46)
at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:182)
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
