[jira] [Created] (KYLIN-4819) build cube failed when `kylin.metadata.hbase-client-retries-number` great than 1
Guangxu Cheng created KYLIN-4819:
Summary: build cube failed when
`kylin.metadata.hbase-client-retries-number` great than 1
Key: KYLIN-4819
URL: https://issues.apache.org/jira/browse/KYLIN-4819
Project: Kylin
Issue Type: Bug
Components: Job Engine
Affects Versions: v3.1.1
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
{code:bash}
2020-11-11 07:31:49,187 TRACE [Scheduler 2133794029 Job
70c242ce-6756-f77a-4b79-6b75c6ecd884-22265] hbase.HBaseResourceStore:334 :
Update row /execute_output/70c242ce-6756-f77a-4b79-6b75c6ecd884-10 from oldTs:
1605051060239, to newTs: 1605051080210, operation result: false
2020-11-11 07:31:49,196 ERROR [Scheduler 2133794029 Job
70c242ce-6756-f77a-4b79-6b75c6ecd884-22265] common.MapReduceExecutable:212 :
error execute MapReduceExecutable\{id=70c242ce-6756-f77a-4b79-6b75c6ecd884-10,
name=Build N-Dimension Cuboid : level 5, state=RUNNING}
org.apache.kylin.common.persistence.WriteConflictException: Overwriting
conflict /execute_output/70c242ce-6756-f77a-4b79-6b75c6ecd884-10, expect old TS
1605051060239, but it is 1605051080210
at
org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:337)
at
org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:443)
at
org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:440)
at
org.apache.kylin.common.persistence.ExponentialBackoffRetry.doWithRetry(ExponentialBackoffRetry.java:52)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceWithRetry(ResourceStore.java:440)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:428)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:422)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:402)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:381)
at
org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:252)
at
org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:426)
at
org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:570)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:177)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
at
org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
When the HBase cluster has performance problems or regions move, kylin may fail
to access HBase. However, many exceptions can be recovered by retrying.
Therefore, I suggest setting the default value of the number of retries to 3
[KYLIN-4711|https://issues.apache.org/jira/browse/KYLIN-4711]
However, after retrying is enabled, the exception writeconflictexception will
appear in some scenarios, which is caused by the checkAndPut operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4756) user/group page has duplicate information between adjacent page numbers
Guangxu Cheng created KYLIN-4756: Summary: user/group page has duplicate information between adjacent page numbers Key: KYLIN-4756 URL: https://issues.apache.org/jira/browse/KYLIN-4756 Project: Kylin Issue Type: Bug Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-09-10-20-28-22-270.png Click the first page to display the first to the 15th information, and click the second to display the information from the 16th to the 45th. With the increase of page number, the number of items displayed on a single page is more, which is repeated with the previous page. !image-2020-09-10-20-28-22-270.png|width=546,height=350! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4755) Error while compiling generated Java code when using Kylin UDF in "case when"
Guangxu Cheng created KYLIN-4755:
Summary: Error while compiling generated Java code when using
Kylin UDF in "case when"
Key: KYLIN-4755
URL: https://issues.apache.org/jira/browse/KYLIN-4755
Project: Kylin
Issue Type: Bug
Components: Query Engine
Affects Versions: v3.1.0
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
SQL:
{code:sql}
select case when TRANS_ID > 0 then concat('a','b')
else version()
end
from KYLIN_SALES
{code}
Error message:
{panel}
Error while compiling generated Java code:
org.apache.calcite.DataContext root;
public org.apache.calcite.linq4j.Enumerable bind(final
org.apache.calcite.DataContext root0) {
root = root0;
final org.apache.calcite.linq4j.Enumerable _inputEnumerable =
((org.apache.kylin.query.schema.OLAPTable)
root.getRootSchema().getSubSchema("DEFAULT").getTable("KYLIN_SALES")).executeOLAPQuery(root,
0);
final org.apache.calcite.linq4j.AbstractEnumerable child = new
org.apache.calcite.linq4j.AbstractEnumerable(){
public org.apache.calcite.linq4j.Enumerator enumerator() {
return new org.apache.calcite.linq4j.Enumerator(){
public final org.apache.calcite.linq4j.Enumerator inputEnumerator =
_inputEnumerable.enumerator();
public void reset() {
inputEnumerator.reset();
}
public boolean moveNext() {
return inputEnumerator.moveNext();
}
public void close() {
inputEnumerator.close();
}
public Object current() {
final Long inp0_ = (Long) ((Object[]) inputEnumerator.current())[0];
{color:red}return inp0_ != null && inp0_.longValue() > 0L ?
org.apache.kylin.query.udf.ConcatUDF.eval("a", "b") :
org.apache.kylin.query.udf.VersionUDF.eval();{color}
}
};
}
};
return child.take(5);
}
public Class getElementType() {
return java.lang.String.class;
}
at org.apache.calcite.avatica.Helper.wrap(Helper.java:37)
at
org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:108)
at
org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1278)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:331)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:230)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:796)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:655)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:618)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:221)
at
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:603)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
... 83 more
{color:red}Caused by: org.codehaus.commons.compiler.CompileException: Line 24,
Column 104: Instance method "eval" cannot be invoked in static context {color}
{panel}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [VOTE] Release apache-kylin-4.0.0-alpha (RC1)
+1 (non-binding) - Signature: ok - Checksum : ok - Built from source (1.8.0_201): ok - Unit tests pass (1.8.0_201): ok - Start kylin server: ok - Kylin Web UI:ok -- Best Regards, Guangxu Li Yang 于2020年9月10日周四 上午9:03写道： > +1 binding > > Really happy to see 4.0 is finally coming!! > > On Wed, Sep 9, 2020 at 4:50 PM Chao wrote: > > > +1 > > mvn test passed. > > > > Yaqian Zhang 于2020年9月9日周三 下午4:37写道： > > > > > +1 > > > mvn clean test passed > > > > > > > 在 2020年9月9日，11:50，Rupeng Wang 写道： > > > > > > > > +1 (non-binding) > > > > mvn test passed on my machine; > > > > happy path passed on HDP2.4 && CDH 5.7. > > > > > > > > --- > > > > Best wishes, > > > > Rupeng Wang > > > > > > > > > > > > 在 2020/9/9 11:32，“Xiaoxiang Yu” x...@apache.org> > > > 写入: > > > >+1 > > > >mvn test passed on my machine; > > > >happy path passed on CDH 5.7. > > > > > > > >-- > > > >Best wishes to you ! > > > >From ：Xiaoxiang Yu > > > > > > > >在 2020-09-09 08:21:57，"恩爸" <441586...@qq.com> 写道： > > > >> +1 from my side. (non-binding) > > > >> > > > >> Best regards, > > > >> Zhichao Zhang > > > >> --原始邮件-- > > > >> 发件人: > > > "dev" > > > < > > > n...@apache.org; > > > >> 发送时间:2020年9月8日(星期二) 晚上11:49 > > > >> 收件人:"dev" > > >> > > > >> 主题:[VOTE] Release apache-kylin-4.0.0-alpha (RC1) > > > >> > > > >> > > > >> > > > >> Hi all, > > > >> > > > >> I have created a build for Apache Kylin 4.0.0-alpha, release > candidate > > > >> 1. Please note, this release is built on kylin-on-parquet-v2 branch. > > > >> > > > >> > > > >> Changes highlights: > > > >> [KYLIN-4213] - The new build engine with Spark-SQL > > > >> [KYLIN-4450] - Add the feature that adjusting spark driver memory > > > adaptively > > > >> [KYLIN-4458] - FilePruner prune shards > > > >> [KYLIN-4475] - Support intersect count for Kylin on Parquet > > > >> [KYLIN-4462] - Support Count Distinct,TopN and Percentile by kylin > on > > > Parquet > > > >> [KYLIN-4713] - Support use diff spark schedule pool for diff query > > > >> [KYLIN-4468] - Support Percentile by kylin on Parquet > > > >> [KYLIN-4662] - Migrate from third-party Spark to offical Apache > Spark > > > >> [KYLIN-4701] - Upgrade front-end from HBase Storage to Parquet > Storage > > > >> [KYLIN-4644] - New tool to clean up intermediate files for > Kylin > > > 4.0 > > > >> [KYLIN-4744] - Add tracking URL for build spark job on yarn > > > >> > > > >> Thanks to everyone who has contributed to this release. > > > >> > > > >> Here are the release > > > >> notes: > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121version=12348093 > > > >> > > > >> > > > >> The commit to being voted > > > >> upon: > > > > > > https://github.com/apache/kylin/commit/a285f9a5b84affc36c5466ce5a1b2fcdb4348b37 > > > >> > > > >> Its hash is a285f9a5b84affc36c5466ce5a1b2fcdb4348b37. > > > >> > > > >> The artifacts to be voted on, including the source package and two > > > >> pre-compiled binary packages are located > > > >> here: > > > > > > https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-4.0.0-alpha-rc1/ > > > >> > > > >> The hash of the artifacts are as follows: > > > >> apache-kylin-4.0.0-alpha-source-release.zip.sha256 > > > >> f98da070a9839251c8cf3806c274c1191bb11d0b251288b4d2586e034f6ac291 > > > >> apache-kylin-4.0.0-alpha-bin-hadoop2.tar.gz.sha256 > > > >> 8075af2608b62177f04bc5a528194c555959775ed69ae80e5ccaf9a37ec1bf74 > > > >> apache-kylin-4.0.0-alpha-bin-cdh57.tar.gz.sha256 > > > >> f98da070a9839251c8cf3806c274c1191bb11d0b251288b4d2586e034f6ac291 > > > >> > > > >> > > > >> A staged Maven repository is available for review > > > >> at: > > > > https://repository.apache.org/content/repositories/orgapachekylin-1081/ > > > >> > > > >> Release artifacts are signed with the following > > > >> key:https://people.apache.org/keys/committer/nic.asc > > > >> > > > >> > > > >> Please vote on releasing this package as Apache Kylin 4.0.0-alpha. > > > >> > > > >> > > > >> The vote is open for the next 72 hours and passes if a majority of > at > > > >> least three +1 binding votes are cast. > > > >> > > > >> > > > >> [ ] +1 Release this package as Apache Kylin 4.0.0-alpha > > > >> [ ] 0 I don't feel strongly about it, but I'm okay with the > > > release > > > >> [ ] -1 Do not release this package because... > > > >> > > > >> > > > >> Here is my vote: > > > >> > > > >> +1 (binding) > > > >> > > > >> - > > > >> > > > >> Best regards, > > > >> Ni Chunen / George > > > > > > > > > > > > >
[jira] [Created] (KYLIN-4717) Failed to build webapp
Guangxu Cheng created KYLIN-4717: Summary: Failed to build webapp Key: KYLIN-4717 URL: https://issues.apache.org/jira/browse/KYLIN-4717 Project: Kylin Issue Type: Improvement Components: Web Affects Versions: v3.1.0 Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-08-25-11-40-49-948.png !image-2020-08-25-11-40-49-948.png|width=568,height=213! OS Platform: macos 10.15.5 (19F101) node version: 14.8.0 npm version: 6.14.7 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4716) Optimize the project page
Guangxu Cheng created KYLIN-4716: Summary: Optimize the project page Key: KYLIN-4716 URL: https://issues.apache.org/jira/browse/KYLIN-4716 Project: Kylin Issue Type: Improvement Components: Web Affects Versions: v3.1.0 Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-08-25-11-26-34-031.png # Now only click the project name field to display the details, which is very inconvenient. After this patch, click any field in this row to display details. # Remove the # from #cubes field !image-2020-08-25-11-26-34-031.png|width=1236,height=76! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4714) Failed to revoke role access of the project
Guangxu Cheng created KYLIN-4714: Summary: Failed to revoke role access of the project Key: KYLIN-4714 URL: https://issues.apache.org/jira/browse/KYLIN-4714 Project: Kylin Issue Type: Improvement Components: Security Affects Versions: v3.1.0 Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-08-24-17-42-21-218.png When I continuously grant and revoke permissions for multiple users, I often can't get the desired results. For examples, there is a Project A, 1. grant user 'A' 2. grant user 'B' 3. revoke user 'A' 4. revoke user 'B' Now, when I revoke user 'B', there will throw an exception. !image-2020-08-24-17-42-21-218.png|width=309,height=200! The reason for this problem is that after revoke or grant, the sequence number of the returned result is incorrect. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4711) Change default value to 3 for kylin.metadata.hbase-client-retries-number
Guangxu Cheng created KYLIN-4711:
Summary: Change default value to 3 for
kylin.metadata.hbase-client-retries-number
Key: KYLIN-4711
URL: https://issues.apache.org/jira/browse/KYLIN-4711
Project: Kylin
Issue Type: Improvement
Affects Versions: v3.1.0
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
```shell
java.lang.RuntimeException:
org.apache.kylin.job.exception.PersistentException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Thu Aug 20 21:06:01 GMT+08:00 2020, RpcRetryingCaller
{globalStartTime=1597928761253, pause=1000, retries=1}
, org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region
kylin_production_metadata,/execute_output/3adc92f2-edcd-2705-5a9c-ad0afe4a0808-01,1594348337103.48b9e5e9c3c7891750236fcec84b38d5.
is not online on xxx.xxx.xxx.xxx,16031,1558009276096
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3033)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1110)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2064)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33857)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2189)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
on xxx.xxx.xxx.xxx,16031,1558009276096
at
org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:174)
at
org.apache.kylin.job.execution.AbstractExecutable.getOutput(AbstractExecutable.java:450)
at
org.apache.kylin.job.execution.AbstractExecutable.isDiscarded(AbstractExecutable.java:561)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:165)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
at
org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
Recently, our build job failed occasionally. After analysis, it was found that
the reason for the failure was due to abnormal access to the MetaStore. We use
HBase as MetaStore.
When accessing HBase, the client will cache the region information of the
table in the client. When the region was moved, client will not actively update
the information in the cache. So the client will receive a
NotServingRegionException, the client will update the cache information when
retrying. But the number of retries in kylin is 1, which means that the clinet
will not try again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Re: [VOTE] Release apache-kylin-3.1.0 (RC1)
+1 (non-binding) - Signature: ok - Checksum : ok - Built from source (1.8.0_201): ok - Unit tests pass (1.8.0_201): ok - Start kylin server: ok - Kylin Web UI:ok -- Best Regards, Guangxu Chao Long 于2020年6月30日周二 上午11:43写道： > +1 > mvn test passed > > On Mon, Jun 29, 2020 at 1:09 PM Ma Gang wrote: > > > +1 (binding)mvn test pass, my test env is JDK8, mac. > > > > > > > > > > -- > > > > Thanks, > > Gang Ma > > > > > > > > > > > > 在 2020-06-29 10:56:50，"Yaqian Zhang" 写道： > > >+1 > > >Mvn clean test passed > > > > > >> 在 2020年6月29日，10:43，ShaoFeng Shi 写道： > > >> > > >> +1 (binding) > > >> > > >> I checked: > > >> > > >> - Compile successfully with JDK 8 on MacOS; > > >> - "mvn test" success; > > >> - The sha 256 hash is correct; > > >> - The gpg signature is by n...@apache.org; > > >> - The NOTICE and LICENSE files; > > >> > > >> > > >> > > >> Best regards, > > >> > > >> Shaofeng Shi 史少锋 > > >> Apache Kylin PMC > > >> Email: shaofeng...@apache.org > > >> > > >> Apache Kylin FAQ: > https://kylin.apache.org/docs/gettingstarted/faq.html > > >> Join Kylin user mail group: user-subscr...@kylin.apache.org > > >> Join Kylin dev mail group: dev-subscr...@kylin.apache.org > > >> > > >> > > >> > > >> > > >> Rupeng Wang 于2020年6月29日周一 上午9:51写道： > > >> > > >>> +1 > > >>> > > >>> mvn test passed > > >>> Binary package test passed on CDH6.0 > > >>> > > >>> > > >>> --- > > >>> Best wishes, > > >>> Rupeng Wang > > >>> > > >>> > > >>> > > >>> 在 2020/6/29 09:33，“Zhong, Yanghong” 写入: > > >>> > > >>>+1 (binding) > > >>> > > >>>mvn test passed. > > >>> > > >>>Best regards, > > >>>Yanghong Zhong > > >>> > > >>>On 2020/6/28, 11:13 PM, "恩爸" <441586...@qq.com> wrote: > > >>> > > >>>External Email > > >>> > > >>> > > >>>+1 from my side. > > >>>Tested passed on CDH5 and HDP3. > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>>Best regards, > > >>>Zhichao Zhang > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>>--原始邮件-- > > >>>发件人:"George Ni" > >>>发送时间:2020年6月28日(星期天) 晚上7:51 > > >>>收件人:"dev" > >>> > > >>>主题:[VOTE] Release apache-kylin-3.1.0 (RC1) > > >>> > > >>> > > >>> > > >>>Hi all, > > >>> > > >>>I have created a build for Apache Kylin 3.1.0, release > > candidate 1. > > >>> > > >>>Changes highlights: > > >>>[KYLIN-4342] - Build Global Dict by MR/Hive New Version > > >>>[KYLIN-3758] - Flink Cube Build Engine > > >>>[KYLIN-3832] - Kylin pushdown to support PostgreSQL > > >>>[KYLIN-4445] - Provide a Kylin on Kubernetes solution > > >>>[KYLIN-4485] - Create a self-service interface for cube > > migration > > >>>[KYLIN-4425] - Refactor Diagnosis Tool > > >>> > > >>>Thanks to everyone who has contributed to this release. > > >>>Here are the release notes: > > >>> > > >>> > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2FReleaseNote.jspa%3FprojectId%3D12316121%26amp%3Bversion%3D12346593data=02%7C01%7Cyangzhong%40ebay.com%7C3237b3771e5541ab35a408d81b75c6f3%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637289539944798530sdata=ncIL3k82VXOGSPTCoo1HXD0Srmb9ti%2Bcvd%2FJbJQaqOo%3Dreserved=0 > > >>> > > >>> > > >>>The commit to being voted upon: > > >>> > > >>> > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkylin%2Fcommit%2F2e056748bf61ef89d421735bb7be41f4266d7b94data=02%7C01%7Cyangzhong%40ebay.com%7C3237b3771e5541ab35a408d81b75c6f3%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637289539944798530sdata=lCJYAjlP1zM7RzkwO2gI1BRqWHWyagn9jaHfAUdg0hw%3Dreserved=0 > > >>> > > >>>Its hash is 2e056748bf61ef89d421735bb7be41f4266d7b94. > > >>> > > >>>The artifacts to be voted on, including the source package and > > four > > >>>pre-compiled binary packages are located here: > > >>> > > >>> > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fkylin%2Fapache-kylin-3.1.0-rc1%2Fdata=02%7C01%7Cyangzhong%40ebay.com%7C3237b3771e5541ab35a408d81b75c6f3%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637289539944798530sdata=%2FkLpG5SuRmROChmme2Q5l76AqEhHQd1xh%2FkfSbG7rr4%3Dreserved=0 > > >>> > > >>>The hash of the artifacts are as follows: > > >>>apache-kylin-3.1.0-source-release.zip.sha256 > > >>> > 1efa8a60003ea8adaa39de7c020be2bd70ed045de736995153707c1ca08353e1 > > >>>apache-kylin-3.1.0-bin-hbase1x.tar.gz.sha256 > > >>> > 78921a27acba781adc41729aa65316833d2aec5d32047d6ffc195bfaf156cd5d > > >>>apache-kylin-3.1.0-bin-cdh57.tar.gz.sha256 > > >>> > bdc5a790880f1b6e1d894454e3512c54515f3b52d55f4d118363514c5007ba89 > > >>>apache-kylin-3.1.0-bin-hadoop3.tar.gz.sha256 > > >>> >
[jira] [Created] (KYLIN-4572) Add "build/VERSION" to .gitignore
Guangxu Cheng created KYLIN-4572: Summary: Add "build/VERSION" to .gitignore Key: KYLIN-4572 URL: https://issues.apache.org/jira/browse/KYLIN-4572 Project: Kylin Issue Type: Improvement Reporter: Guangxu Cheng Assignee: Guangxu Cheng A temporary file build/VERSION will be generated when executing build/script/package.sh -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4567) Improve TopN merge performance in MR engine
Guangxu Cheng created KYLIN-4567:
Summary: Improve TopN merge performance in MR engine
Key: KYLIN-4567
URL: https://issues.apache.org/jira/browse/KYLIN-4567
Project: Kylin
Issue Type: Improvement
Components: Measure - TopN
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
We have a cube that needs to calculate the TOPN of 13 columns. The number of
data source is only 500k. But, the cubing job always fail when building base
cuboid.
we found that the map task always killed by the ApplicationMaster due to time
out
{noformat}
ERROR-[-10001]-[MR]:[Mr Task
Timeout]:[AttemptID:attempt_1591996262448_229922_m_00_1 Timed out after
3600 secs!] ERROR-[-10015]-[MR]:[Container Exit Accidentally]:[Container killed
by the ApplicationMaster. Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143 ]
{noformat}
the stack information as below:
{noformat}
"SpillThread" #35 daemon prio=5 os_prio=0 tid=0x7f9a89771800 nid=0x133a2
runnable [0x7f9a56e3f000]
java.lang.Thread.State: RUNNABLE
at java.util.LinkedList.toArray(LinkedList.java:1052)
at java.util.List.sort(List.java:477)
at java.util.Collections.sort(Collections.java:175)
at
org.apache.kylin.measure.topn.TopNCounter.sortAndRetain(TopNCounter.java:96)
at org.apache.kylin.measure.topn.TopNCounter.merge(TopNCounter.java:183)
at
org.apache.kylin.measure.topn.TopNAggregator.aggregate(TopNAggregator.java:44)
at
org.apache.kylin.measure.topn.TopNAggregator.aggregate(TopNAggregator.java:27)
at
org.apache.kylin.measure.MeasureAggregators.aggregate(MeasureAggregators.java:83)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.doReduce(CuboidReducer.java:108)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.doReduce(CuboidReducer.java:44)
at org.apache.kylin.engine.mr.KylinReducer.reduce(KylinReducer.java:77)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at
org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1645)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:884)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1540)
{noformat}
>From the stack information, we found that sorting takes a lot of time. After
>merge another counter into this counter, need to re-sort this counter. Maybe
>we can reduce the frequency of sorting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4561) overall job number statistics in monitor page is incorrect after change the job status
Guangxu Cheng created KYLIN-4561: Summary: overall job number statistics in monitor page is incorrect after change the job status Key: KYLIN-4561 URL: https://issues.apache.org/jira/browse/KYLIN-4561 Project: Kylin Issue Type: Bug Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-06-11-15-40-18-370.png, image-2020-06-11-15-41-36-058.png !image-2020-06-11-15-41-36-058.png|width=587,height=45! When the job status is changed, the statistics in monitor page will not change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4560) support to re-run/delete cardinality and lookup snapshot job
Guangxu Cheng created KYLIN-4560: Summary: support to re-run/delete cardinality and lookup snapshot job Key: KYLIN-4560 URL: https://issues.apache.org/jira/browse/KYLIN-4560 Project: Kylin Issue Type: Sub-task Reporter: Guangxu Cheng Assignee: Guangxu Cheng -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4559) show cardinality and lookup snapshot job on job page
Guangxu Cheng created KYLIN-4559: Summary: show cardinality and lookup snapshot job on job page Key: KYLIN-4559 URL: https://issues.apache.org/jira/browse/KYLIN-4559 Project: Kylin Issue Type: Sub-task Reporter: Guangxu Cheng Assignee: Guangxu Cheng -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4557) Refactor JobService to improve code readability
Guangxu Cheng created KYLIN-4557: Summary: Refactor JobService to improve code readability Key: KYLIN-4557 URL: https://issues.apache.org/jira/browse/KYLIN-4557 Project: Kylin Issue Type: Sub-task Reporter: Guangxu Cheng Assignee: Guangxu Cheng -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4556) improve job service
Guangxu Cheng created KYLIN-4556: Summary: improve job service Key: KYLIN-4556 URL: https://issues.apache.org/jira/browse/KYLIN-4556 Project: Kylin Issue Type: Task Reporter: Guangxu Cheng Assignee: Guangxu Cheng At present, only cubing and checkpoint jobs can be obtained through job API,for cardinality and lookup snapshot job, we can't get them which is very inconvenient. when the cardinality task fails, we can only find the reason why the task failed from the system log, and we cannot rerun or delete these failed tasks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4539) Spark build failed due to com.codahale.metrics.json.MetricsModule loaded from kylin job jar
Guangxu Cheng created KYLIN-4539:
Summary: Spark build failed due to
com.codahale.metrics.json.MetricsModule loaded from kylin job jar
Key: KYLIN-4539
URL: https://issues.apache.org/jira/browse/KYLIN-4539
Project: Kylin
Issue Type: Bug
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
{code:shell}
[Loaded com.fasterxml.jackson.databind.Module from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/jackson-databind-2.9.4.jar]
[Loaded org.apache.kylin.job.shaded.com.fasterxml.jackson.core.Versioned from
file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
[Loaded org.apache.kylin.job.shaded.com.fasterxml.jackson.databind.Module from
file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
[Loaded com.codahale.metrics.json.MetricsModule from
file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
[Loaded java.lang.VerifyError from
/data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
scala.util.control.NonFatal$ from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/scala-library-2.11.8.jar]
[Loaded org.apache.spark.SparkUserAppException from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/spark-core_2.11-3.12.0.jar]
[Loaded org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4$$anonfun$run$11
from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/spark-yarn_2.11-3.12.0.jar]
[Loaded org.apache.log4j.spi.ThrowableInformation from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
[Loaded org.apache.log4j.spi.ThrowableRenderer from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
[Loaded org.apache.log4j.DefaultThrowableRenderer from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
[Loaded java.lang.Throwable$PrintStreamOrWriter from
/data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
java.lang.Throwable$WrappedPrintWriter from
/data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
java.io.LineNumberReader from
/data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] User class threw
exception: java.lang.VerifyError: Bad type on operand stack Exception Details:
Location:
org/apache/spark/metrics/sink/MetricsServlet.(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V
@116: invokevirtual Reason: Type 'com/codahale/metrics/json/MetricsModule'
(current frame, stack[2]) is not assignable to
'com/fasterxml/jackson/databind/Module' Current Frame: bci: @116 flags: \{ }
locals: \{ 'org/apache/spark/metrics/sink/MetricsServlet',
'java/util/Properties', 'com/codahale/metrics/MetricRegistry',
'org/apache/spark/SecurityManager' } stack: \{
'org/apache/spark/metrics/sink/MetricsServlet',
'com/fasterxml/jackson/databind/ObjectMapper',
'com/codahale/metrics/json/MetricsModule' } Bytecode: 0x000: 2a2b b500 2a2a
2cb5 002f 2a2d b500 5c2a 0x010: b700 7e2a 1280 b500 322a 1282 b500 342a
0x020: 03b5 0037 2a2b 2ab6 0084 b600 8ab5 0039 0x030: 2ab2 008f 2b2a
b600 91b6 008a b600 95bb 0x040: 0014 592a b700 96b6 009c bb00 1659 2ab7
0x050: 009d b600 a1b8 00a7 b500 3b2a bb00 7159 0x060: b700 a8bb 00aa
59b2 00b0 b200 b32a b600 0x070: b5b7 00b8 b600 bcb5 003e b1
java.lang.VerifyError: Bad type on operand stack Exception Details: Location:
org/apache/spark/metrics/sink/MetricsServlet.(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V
@116: invokevirtual Reason: Type 'com/codahale/metrics/json/MetricsModule'
(current frame, stack[2]) is not assignable to
'com/fasterxml/jackson/databind/Module' Current Frame: bci: @116 flags: \{ }
locals: \{ 'org/apache/spark/metrics/sink/MetricsServlet',
'java/util/Properties', 'com/codahale/metrics/MetricRegistry',
'org/apache/spark/SecurityManager' } stack: \{
'org/apache/spark/metrics/sink/MetricsServlet',
'com/fasterxml/jackson/databind/ObjectMapper',
'com/codahale/metrics/json/MetricsModule' } Bytecode: 0x000: 2a2b b500 2a2a
2cb5 002f 2a2d b500 5c2a 0x010: b700 7e2a 1280 b500 322a 1282 b500 342a
0x020: 03b5 0037 2a2b 2ab6 0084 b600 8ab5 0039 0x030: 2ab2 008f 2b2a
b600 91b6 008a b600 95bb 0x040: 0014 592a b700 96b6 009c bb00 1659 2ab7
0x050: 009d b600 a1b8 00a7 b500 3b2a bb00 7159 0x060: b700 a8bb 00aa
59b2 00b0 b200 b32a b600 0x070: b5b7 00b8 b600 bcb5 003e b1 at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:348) at
org.apache.spark.util.Utils$.classForName(Utils.scala:238) at
org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:198)
at
org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:194)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99
[jira] [Created] (KYLIN-4537) Give a friendly tips to the user when getting task list fails
Guangxu Cheng created KYLIN-4537: Summary: Give a friendly tips to the user when getting task list fails Key: KYLIN-4537 URL: https://issues.apache.org/jira/browse/KYLIN-4537 Project: Kylin Issue Type: Improvement Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-05-27-21-47-34-128.png, image-2020-05-27-21-48-57-223.png On the monitoring page, when the task list acquisition fails, it will show "failed to load jobs", but we don't know the detail reason for the failure, whether it is due to permission problems or the system crash? Before: !image-2020-05-27-21-48-57-223.png|width=491,height=314! After: !image-2020-05-27-21-47-34-128.png|width=441,height=310! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4527) Beautify the drop-down list of the cube on query page
Guangxu Cheng created KYLIN-4527: Summary: Beautify the drop-down list of the cube on query page Key: KYLIN-4527 URL: https://issues.apache.org/jira/browse/KYLIN-4527 Project: Kylin Issue Type: Improvement Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-05-27-12-05-49-425.png !image-2020-05-27-12-05-49-425.png|width=336,height=168! The drop-down list of cube is very compact, which is not convenient to select cube -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4526) Enhance get the hive table rows
Guangxu Cheng created KYLIN-4526: Summary: Enhance get the hive table rows Key: KYLIN-4526 URL: https://issues.apache.org/jira/browse/KYLIN-4526 Project: Kylin Issue Type: Task Reporter: Guangxu Cheng Assignee: Guangxu Cheng In kylin-4315, we get the rows of the hive table from metadata, but when we turn off hive's statistics feature(`hive.stats.autogather=false`), we can't get the correct rows of hive table from metadata -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4511) Support project admin user to calculate column cardinality by web
Guangxu Cheng created KYLIN-4511: Summary: Support project admin user to calculate column cardinality by web Key: KYLIN-4511 URL: https://issues.apache.org/jira/browse/KYLIN-4511 Project: Kylin Issue Type: Improvement Components: Web Reporter: Guangxu Cheng Assignee: Guangxu Cheng Now only the administrator of kylin can recalculate the cardinality of the hive table through the admin web, but the project administrator cannot. Add an entry to calculate cardinality on the table detail web -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4510) Automatically refresh the page after reload table
Guangxu Cheng created KYLIN-4510: Summary: Automatically refresh the page after reload table Key: KYLIN-4510 URL: https://issues.apache.org/jira/browse/KYLIN-4510 Project: Kylin Issue Type: Improvement Components: Web Reporter: Guangxu Cheng Assignee: Guangxu Cheng Now after reload table, we need to manually refresh the page to get the latest metadata information, which is unfriendly -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4509) get hive table rows from metadata when using beeline
Guangxu Cheng created KYLIN-4509: Summary: get hive table rows from metadata when using beeline Key: KYLIN-4509 URL: https://issues.apache.org/jira/browse/KYLIN-4509 Project: Kylin Issue Type: Improvement Affects Versions: v3.0.1, v2.6.5 Reporter: Guangxu Cheng Assignee: Guangxu Cheng Now by running SQL "select count(*) from table" to get the number of the hive table records, the performance is relatively poor and will consume a lot of resources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4490) Fix minor bug which add measure on frontend
Guangxu Cheng created KYLIN-4490: Summary: Fix minor bug which add measure on frontend Key: KYLIN-4490 URL: https://issues.apache.org/jira/browse/KYLIN-4490 Project: Kylin Issue Type: Bug Components: Web Affects Versions: v3.0.1, v2.6.5, v4.0.0-beta Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-05-11-14-52-30-494.png !image-2020-05-11-14-52-30-494.png|width=1227,height=194! -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [ANNOUNCE] Please welcome Chunen Ni to the Apache Kylin PMC
Congratulations, chunen!!! JiaTao Tao 于2019年12月2日周一 上午10:58写道： > Congratulations! > > -- > > Regards! > > Aron Tao > > > ShaoFeng Shi 于2019年12月1日周日 上午10:47写道： > > > On behalf of the Apache Kylin PMC, I am pleased to announce that Chunen > Ni > > has accepted our invitation to become a PMC member on the Kylin project. > We > > appreciate Chunen stepping up to take more responsibility in the Kylin > > project. > > > > Please join me in welcoming Chunen to the Kylin PMC! > > > > Best regards, > > > > Shaofeng Shi 史少锋 > > Apache Kylin PMC > > Email: shaofeng...@apache.org > > > > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html > > Join Kylin user mail group: user-subscr...@kylin.apache.org > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org > > > > > > >
Re: New committer: Temple Zhou
Congratulations! codingfor...@126.com 于2019年11月16日周六 下午2:28写道： > Congratulations! > > > > 在 2019年11月16日，13:57，nichunen 写道： > > > > Congratulations! > >
[jira] [Created] (KYLIN-4235) Failed to load table metadata from JDBC data source
Guangxu Cheng created KYLIN-4235: Summary: Failed to load table metadata from JDBC data source Key: KYLIN-4235 URL: https://issues.apache.org/jira/browse/KYLIN-4235 Project: Kylin Issue Type: Bug Affects Versions: v2.6.4, v3.0.0-alpha2 Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2019-11-01-23-45-56-319.png After adding the following configurations to the project, the table metadata still cannot be loaded !image-2019-11-01-23-45-56-319.png|width=480,height=373! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4145) compile failed due to incompatible version between scala and scala-maven-plugin
Guangxu Cheng created KYLIN-4145:
Summary: compile failed due to incompatible version between scala
and scala-maven-plugin
Key: KYLIN-4145
URL: https://issues.apache.org/jira/browse/KYLIN-4145
Project: Kylin
Issue Type: Bug
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
The version of scala-maven-plugin is not specified, and the latest
version(4.1.1) will be used when compiling the project, which is incompatible
with the version of scala(2.11.0).
So, it would be better to specify the version number of scala-maven-plugin.
{code:xml}
[ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:4.1.1:compile (scala-compile-first) on
project kylin-engine-spark: Execution scala-compile-first of goal
net.alchim31.maven:scala-maven-plugin:4.1.1:compile failed: An API
incompatibility was encountered while executing
net.alchim31.maven:scala-maven-plugin:4.1.1:compile:
java.lang.NoSuchMethodError:
xsbt.ZincCompiler.classPath()Lscala/tools/nsc/util/ClassFileLookup;
[ERROR] -
[ERROR] realm =plugin>net.alchim31.maven:scala-maven-plugin:4.1.1
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] =
file:/Users/cheng/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.1.1/scala-maven-plugin-4.1.1.jar
[ERROR] urls[1] =
file:/Users/cheng/.m2/repository/org/apache/maven/maven-builder-support/3.3.9/maven-builder-support-3.3.9.jar
[ERROR] urls[2] =
file:/Users/cheng/.m2/repository/com/google/guava/guava/18.0/guava-18.0.jar
[ERROR] urls[3] =
file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.21/plexus-interpolation-1.21.jar
[ERROR] urls[4] =
file:/Users/cheng/.m2/repository/javax/enterprise/cdi-api/1.0/cdi-api-1.0.jar
[ERROR] urls[5] =
file:/Users/cheng/.m2/repository/org/eclipse/sisu/org.eclipse.sisu.inject/0.3.2/org.eclipse.sisu.inject-0.3.2.jar
[ERROR] urls[6] =
file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-component-annotations/1.6/plexus-component-annotations-1.6.jar
[ERROR] urls[7] =
file:/Users/cheng/.m2/repository/org/apache/maven/reporting/maven-reporting-api/3.0/maven-reporting-api-3.0.jar
[ERROR] urls[8] =
file:/Users/cheng/.m2/repository/org/eclipse/aether/aether-util/1.0.2.v20150114/aether-util-1.0.2.v20150114.jar
[ERROR] urls[9] =
file:/Users/cheng/.m2/repository/com/google/inject/guice/4.0/guice-4.0-no_aop.jar
[ERROR] urls[10] =
file:/Users/cheng/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar
[ERROR] urls[11] =
file:/Users/cheng/.m2/repository/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.jar
[ERROR] urls[12] =
file:/Users/cheng/.m2/repository/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.jar
[ERROR] urls[13] =
file:/Users/cheng/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
[ERROR] urls[14] =
file:/Users/cheng/.m2/repository/org/apache/maven/shared/maven-dependency-tree/3.0.1/maven-dependency-tree-3.0.1.jar
[ERROR] urls[15] =
file:/Users/cheng/.m2/repository/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar
[ERROR] urls[16] =
file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-utils/3.2.1/plexus-utils-3.2.1.jar
[ERROR] urls[17] =
file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-archiver/4.1.0/plexus-archiver-4.1.0.jar
[ERROR] urls[18] =
file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-io/3.1.1/plexus-io-3.1.1.jar
[ERROR] urls[19] =
file:/Users/cheng/.m2/repository/org/apache/commons/commons-compress/1.18/commons-compress-1.18.jar
[ERROR] urls[20] =
file:/Users/cheng/.m2/repository/org/iq80/snappy/snappy/0.4/snappy-0.4.jar
[ERROR] urls[21] =
file:/Users/cheng/.m2/repository/org/tukaani/xz/1.8/xz-1.8.jar
[ERROR] urls[22] =
file:/Users/cheng/.m2/repository/backport-util-concurrent/backport-util-concurrent/3.1/backport-util-concurrent-3.1.jar
[ERROR] urls[23] =
file:/Users/cheng/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar
[ERROR] urls[24] =
file:/Users/cheng/.m2/repository/org/apache/maven/maven-archiver/3.4.0/maven-archiver-3.4.0.jar
[ERROR] urls[25] =
file:/Users/cheng/.m2/repository/org/apache/maven/shared/maven-shared-utils/3.2.1/maven-shared-utils-3.2.1.jar
[ERROR] urls[26] =
file:/Users/cheng/.m2/repository/commons-io/commons-io/2.5/commons-io-2.5.jar
[ERROR] urls[27] =
file:/Users/cheng/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.9/doxia-sink-api-1.9.jar
[ERROR] urls[28] =
file:/Users/cheng/.m2/repository/org/apache/maven/doxia/doxia-logging-api/1.9/doxia-logging-api-1.9.jar
[ERROR] urls[29] =
file:/Users/cheng/.m2/repository/org/apache/maven/shared/maven-invoker/3.0.1/maven-invoker-3.0.1.jar
[ERROR] urls[30] =
file:/Users/cheng/.m2/repository/org/scala-sbt/zinc_2.12/1.3.0-M8/zinc_2.12-1.3.0-M8.jar
[ERROR] urls[31] =
file:/Users/cheng/.m2/repository/org/sc
[jira] [Created] (KYLIN-3936) MR/Spark task will still run after the job is stopped.
Guangxu Cheng created KYLIN-3936: Summary: MR/Spark task will still run after the job is stopped. Key: KYLIN-3936 URL: https://issues.apache.org/jira/browse/KYLIN-3936 Project: Kylin Issue Type: Bug Components: Job Engine Reporter: Guangxu Cheng Assignee: Guangxu Cheng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3895) Failed to register new MBean when "kylin.server.query-metrics-enabled" set true
Guangxu Cheng created KYLIN-3895:
Summary: Failed to register new MBean when
"kylin.server.query-metrics-enabled" set true
Key: KYLIN-3895
URL: https://issues.apache.org/jira/browse/KYLIN-3895
Project: Kylin
Issue Type: Bug
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
{code}
2019-03-20 10:17:25,753 WARN [Query 46cd99cc-8eb2-8370-d24c-6c10f18da9e0-54]
util.MBeans:94 : Error creating MBean object name: Ha
doop:service=Kylin,name=KYLIN_SYSTEM,sub=CUBE[name=KYLIN_HIVE_METRICS_JOB_QA]
org.apache.hadoop.metrics2.MetricsException:
javax.management.MalformedObjectNameException: Invalid character '=' in value
part of property
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:122)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:102)
at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:92)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55)
{code}
The subname can't contain '='
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3888) TableNotDisabledException when running "Convert Lookup Table to HFile"
Guangxu Cheng created KYLIN-3888:
Summary: TableNotDisabledException when running "Convert Lookup
Table to HFile"
Key: KYLIN-3888
URL: https://issues.apache.org/jira/browse/KYLIN-3888
Project: Kylin
Issue Type: Bug
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
{code}
org.apache.hadoop.hbase.TableNotDisabledException:
KYLIN_LOOKUP_DEFAULT.F_ADVERTISER_DIM_KYLIN_TEST_9WCVUDIXI5
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.convertResult(HBaseAdmin.java:4756)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4714)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4647)
at
org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:904)
at
org.apache.kylin.storage.hbase.lookup.LookupTableToHFileJob.removeSnapshotIfExist(LookupTableToHFileJob.java:170)
at
org.apache.kylin.storage.hbase.lookup.LookupTableToHFileJob.run(LookupTableToHFileJob.java:104)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:131)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotDisabledException):
KYLIN_LOOKUP_DEFAULT.F_ADVERTISER_DIM_KYLIN_TEST_9WCVUDIXI5
at
org.apache.hadoop.hbase.master.HMaster.checkTableModifiable(HMaster.java:2266)
at
org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.prepareDelete(DeleteTableProcedure.java:253)
at
org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:102)
at
org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:59)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:498)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1147)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:942)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:895)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:77)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:497)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3874) "Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS clusters
Guangxu Cheng created KYLIN-3874:
Summary: "Convert Cuboid Data to HFile" failed when HBase and MR
use different HDFS clusters
Key: KYLIN-3874
URL: https://issues.apache.org/jira/browse/KYLIN-3874
Project: Kylin
Issue Type: Bug
Components: Job Engine
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
{panel:title=exception}
2019-03-14 16:33:45,113 INFO [main] org.apache.hadoop.service.AbstractService:
Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED;
cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.io.IOException: Couldn't create proxy provider class
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
...
Caused by: java.lang.RuntimeException: Could not find any configured addresses
for URI
hdfs://{color:#ff}hbase-ns-test{color}/kylin/kylin_metadata/kylin-cd4db32e-42b6-44f2-7fa9-ef6c46334249/derived_meta_global/hfile
at
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.(ConfiguredFailoverProxyProvider.java:116)
... 31 more
{panel}
{color:#ff}hbase-ns-test{color} is the nameservice of HDFS which is used by
HBase.
In our kylin environment, HBase and MR/HIVE use different HDFS clusters.
{code:java|title=CubeHFileJob.java|borderStyle=solid}
// use current hbase configuration
Configuration configuration = new
Configuration(HBaseConnection.getCurrentHBaseConfiguration());
merge(configuration, getConf());
{code}
"getConf()" contains only the HDFS configuration of MR, and "configuration"
contains the HDFS configuration both HBase and MR. See:
[HBaseConnection#addHBaseClusterNNHAConfiguration()|https://github.com/apache/kylin/blob/d23d2bddab3263d39059492b91f71d8091198057/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseConnection.java#L205]
So "merge(configuration, getConf())" will lead to HBASE's HDFS configuration to
be covered. Mainly parameter "dfs.nameservices"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
Guangxu Cheng created KYLIN-3866: Summary: Whether to set mapreduce.application.classpath is determined by the user Key: KYLIN-3866 URL: https://issues.apache.org/jira/browse/KYLIN-3866 Project: Kylin Issue Type: Bug Reporter: Guangxu Cheng In the kylin environment, if there is no configuration parameter "mapreduce.application.classpath", then kylin will use the command "mapred classpath" to get the local classpath and set it to "mapreduce.application.classpath", However, it will cause the following two problems: 1. The local classpath may not be the same as the MapReduce cluster. 2. There is no need to configure the parameter "mapreduce.application.classpath" on the client side in some cluster. So, it may be a better choice for users to decide whether they need to configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
