[jira] [Created] (KYLIN-4819) build cube failed when `kylin.metadata.hbase-client-retries-number` great than 1
Guangxu Cheng created KYLIN-4819:
Summary: build cube failed when
`kylin.metadata.hbase-client-retries-number` great than 1
Key: KYLIN-4819
URL: https://issues.apache.org/jira/browse/KYLIN-4819
Project: Kylin
Issue Type: Bug
Components: Job Engine
Affects Versions: v3.1.1
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
{code:bash}
2020-11-11 07:31:49,187 TRACE [Scheduler 2133794029 Job
70c242ce-6756-f77a-4b79-6b75c6ecd884-22265] hbase.HBaseResourceStore:334 :
Update row /execute_output/70c242ce-6756-f77a-4b79-6b75c6ecd884-10 from oldTs:
1605051060239, to newTs: 1605051080210, operation result: false
2020-11-11 07:31:49,196 ERROR [Scheduler 2133794029 Job
70c242ce-6756-f77a-4b79-6b75c6ecd884-22265] common.MapReduceExecutable:212 :
error execute MapReduceExecutable\{id=70c242ce-6756-f77a-4b79-6b75c6ecd884-10,
name=Build N-Dimension Cuboid : level 5, state=RUNNING}
org.apache.kylin.common.persistence.WriteConflictException: Overwriting
conflict /execute_output/70c242ce-6756-f77a-4b79-6b75c6ecd884-10, expect old TS
1605051060239, but it is 1605051080210
at
org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:337)
at
org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:443)
at
org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:440)
at
org.apache.kylin.common.persistence.ExponentialBackoffRetry.doWithRetry(ExponentialBackoffRetry.java:52)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceWithRetry(ResourceStore.java:440)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:428)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:422)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:402)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:381)
at
org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:252)
at
org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:426)
at
org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:570)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:177)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
at
org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
When the HBase cluster has performance problems or regions move, kylin may fail
to access HBase. However, many exceptions can be recovered by retrying.
Therefore, I suggest setting the default value of the number of retries to 3
[KYLIN-4711|https://issues.apache.org/jira/browse/KYLIN-4711]
However, after retrying is enabled, the exception writeconflictexception will
appear in some scenarios, which is caused by the checkAndPut operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4666) Improve TopNCounter's merge performance
[ https://issues.apache.org/jira/browse/KYLIN-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205366#comment-17205366 ] Guangxu Cheng edited comment on KYLIN-4666 at 10/1/20, 8:21 AM: duplicate of [KYLIN-4567|https://issues.apache.org/jira/browse/KYLIN-4567] ? was (Author: andrewcheng): duplicate of [KYLIN-4567|https://github.com/apache/kylin/pull/1259] ? > Improve TopNCounter's merge performance > --- > > Key: KYLIN-4666 > URL: https://issues.apache.org/jira/browse/KYLIN-4666 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > > Currently, we need to do sort for very merge operation, which will cost much > time for thousands of merges. It's better to leverage a bit more buffer to > reduce the chance of sort -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4666) Improve TopNCounter's merge performance
[ https://issues.apache.org/jira/browse/KYLIN-4666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205366#comment-17205366 ] Guangxu Cheng commented on KYLIN-4666: -- duplicate of [KYLIN-4567|https://github.com/apache/kylin/pull/1259] ? > Improve TopNCounter's merge performance > --- > > Key: KYLIN-4666 > URL: https://issues.apache.org/jira/browse/KYLIN-4666 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > > Currently, we need to do sort for very merge operation, which will cost much > time for thousands of merges. It's better to leverage a bit more buffer to > reduce the chance of sort -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4756) user/group page has duplicate information between adjacent page numbers
Guangxu Cheng created KYLIN-4756: Summary: user/group page has duplicate information between adjacent page numbers Key: KYLIN-4756 URL: https://issues.apache.org/jira/browse/KYLIN-4756 Project: Kylin Issue Type: Bug Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-09-10-20-28-22-270.png Click the first page to display the first to the 15th information, and click the second to display the information from the 16th to the 45th. With the increase of page number, the number of items displayed on a single page is more, which is repeated with the previous page. !image-2020-09-10-20-28-22-270.png|width=546,height=350! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4755) Error while compiling generated Java code when using Kylin UDF in "case when"
Guangxu Cheng created KYLIN-4755:
Summary: Error while compiling generated Java code when using
Kylin UDF in "case when"
Key: KYLIN-4755
URL: https://issues.apache.org/jira/browse/KYLIN-4755
Project: Kylin
Issue Type: Bug
Components: Query Engine
Affects Versions: v3.1.0
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
SQL:
{code:sql}
select case when TRANS_ID > 0 then concat('a','b')
else version()
end
from KYLIN_SALES
{code}
Error message:
{panel}
Error while compiling generated Java code:
org.apache.calcite.DataContext root;
public org.apache.calcite.linq4j.Enumerable bind(final
org.apache.calcite.DataContext root0) {
root = root0;
final org.apache.calcite.linq4j.Enumerable _inputEnumerable =
((org.apache.kylin.query.schema.OLAPTable)
root.getRootSchema().getSubSchema("DEFAULT").getTable("KYLIN_SALES")).executeOLAPQuery(root,
0);
final org.apache.calcite.linq4j.AbstractEnumerable child = new
org.apache.calcite.linq4j.AbstractEnumerable(){
public org.apache.calcite.linq4j.Enumerator enumerator() {
return new org.apache.calcite.linq4j.Enumerator(){
public final org.apache.calcite.linq4j.Enumerator inputEnumerator =
_inputEnumerable.enumerator();
public void reset() {
inputEnumerator.reset();
}
public boolean moveNext() {
return inputEnumerator.moveNext();
}
public void close() {
inputEnumerator.close();
}
public Object current() {
final Long inp0_ = (Long) ((Object[]) inputEnumerator.current())[0];
{color:red}return inp0_ != null && inp0_.longValue() > 0L ?
org.apache.kylin.query.udf.ConcatUDF.eval("a", "b") :
org.apache.kylin.query.udf.VersionUDF.eval();{color}
}
};
}
};
return child.take(5);
}
public Class getElementType() {
return java.lang.String.class;
}
at org.apache.calcite.avatica.Helper.wrap(Helper.java:37)
at
org.apache.calcite.adapter.enumerable.EnumerableInterpretable.toBindable(EnumerableInterpretable.java:108)
at
org.apache.calcite.prepare.CalcitePrepareImpl$CalcitePreparingStmt.implement(CalcitePrepareImpl.java:1278)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:331)
at org.apache.calcite.prepare.Prepare.prepareSql(Prepare.java:230)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepare2_(CalcitePrepareImpl.java:796)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepare_(CalcitePrepareImpl.java:655)
at
org.apache.calcite.prepare.CalcitePrepareImpl.prepareSql(CalcitePrepareImpl.java:618)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.parseQuery(CalciteConnectionImpl.java:221)
at
org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:603)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
... 83 more
{color:red}Caused by: org.codehaus.commons.compiler.CompileException: Line 24,
Column 104: Instance method "eval" cannot be invoked in static context {color}
{panel}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4717) Failed to build webapp
[ https://issues.apache.org/jira/browse/KYLIN-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-4717: - Description: !image-2020-08-25-11-40-49-948.png|width=803,height=301! OS Platform: macos 10.15.5 (19F101) node version: 14.8.0 npm version: 6.14.7 was: !image-2020-08-25-11-40-49-948.png|width=568,height=213! OS Platform: macos 10.15.5 (19F101) node version: 14.8.0 npm version: 6.14.7 > Failed to build webapp > -- > > Key: KYLIN-4717 > URL: https://issues.apache.org/jira/browse/KYLIN-4717 > Project: Kylin > Issue Type: Improvement > Components: Web >Affects Versions: v3.1.0 >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: image-2020-08-25-11-40-49-948.png > > > !image-2020-08-25-11-40-49-948.png|width=803,height=301! > OS Platform: macos 10.15.5 (19F101) > node version: 14.8.0 > npm version: 6.14.7 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4717) Failed to build webapp
Guangxu Cheng created KYLIN-4717: Summary: Failed to build webapp Key: KYLIN-4717 URL: https://issues.apache.org/jira/browse/KYLIN-4717 Project: Kylin Issue Type: Improvement Components: Web Affects Versions: v3.1.0 Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-08-25-11-40-49-948.png !image-2020-08-25-11-40-49-948.png|width=568,height=213! OS Platform: macos 10.15.5 (19F101) node version: 14.8.0 npm version: 6.14.7 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4716) Optimize the project page
Guangxu Cheng created KYLIN-4716: Summary: Optimize the project page Key: KYLIN-4716 URL: https://issues.apache.org/jira/browse/KYLIN-4716 Project: Kylin Issue Type: Improvement Components: Web Affects Versions: v3.1.0 Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-08-25-11-26-34-031.png # Now only click the project name field to display the details, which is very inconvenient. After this patch, click any field in this row to display details. # Remove the # from #cubes field !image-2020-08-25-11-26-34-031.png|width=1236,height=76! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4714) Failed to revoke role access of the project
Guangxu Cheng created KYLIN-4714: Summary: Failed to revoke role access of the project Key: KYLIN-4714 URL: https://issues.apache.org/jira/browse/KYLIN-4714 Project: Kylin Issue Type: Improvement Components: Security Affects Versions: v3.1.0 Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-08-24-17-42-21-218.png When I continuously grant and revoke permissions for multiple users, I often can't get the desired results. For examples, there is a Project A, 1. grant user 'A' 2. grant user 'B' 3. revoke user 'A' 4. revoke user 'B' Now, when I revoke user 'B', there will throw an exception. !image-2020-08-24-17-42-21-218.png|width=309,height=200! The reason for this problem is that after revoke or grant, the sequence number of the returned result is incorrect. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4711) Change default value to 3 for kylin.metadata.hbase-client-retries-number
Guangxu Cheng created KYLIN-4711:
Summary: Change default value to 3 for
kylin.metadata.hbase-client-retries-number
Key: KYLIN-4711
URL: https://issues.apache.org/jira/browse/KYLIN-4711
Project: Kylin
Issue Type: Improvement
Affects Versions: v3.1.0
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
```shell
java.lang.RuntimeException:
org.apache.kylin.job.exception.PersistentException:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Thu Aug 20 21:06:01 GMT+08:00 2020, RpcRetryingCaller
{globalStartTime=1597928761253, pause=1000, retries=1}
, org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region
kylin_production_metadata,/execute_output/3adc92f2-edcd-2705-5a9c-ad0afe4a0808-01,1594348337103.48b9e5e9c3c7891750236fcec84b38d5.
is not online on xxx.xxx.xxx.xxx,16031,1558009276096
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3033)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1110)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2064)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33857)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2189)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)
on xxx.xxx.xxx.xxx,16031,1558009276096
at
org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:174)
at
org.apache.kylin.job.execution.AbstractExecutable.getOutput(AbstractExecutable.java:450)
at
org.apache.kylin.job.execution.AbstractExecutable.isDiscarded(AbstractExecutable.java:561)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:165)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:191)
at
org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
Recently, our build job failed occasionally. After analysis, it was found that
the reason for the failure was due to abnormal access to the MetaStore. We use
HBase as MetaStore.
When accessing HBase, the client will cache the region information of the
table in the client. When the region was moved, client will not actively update
the information in the cache. So the client will receive a
NotServingRegionException, the client will update the cache information when
retrying. But the number of retries in kylin is 1, which means that the clinet
will not try again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4481) Project-level ACL lookups not working for non-admin SAML-federated users
[
https://issues.apache.org/jira/browse/KYLIN-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171863#comment-17171863
]
Guangxu Cheng commented on KYLIN-4481:
--
[~yaho] Sorry for the trouble. As you said it is more reasonable to use a
username without @ validdomain.com for ACL.
+1 for revert the commit.
> Project-level ACL lookups not working for non-admin SAML-federated users
>
>
> Key: KYLIN-4481
> URL: https://issues.apache.org/jira/browse/KYLIN-4481
> Project: Kylin
> Issue Type: Bug
> Components: Security
>Affects Versions: v2.6.5, v3.0.1
>Reporter: Rafael Felix Correa
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v3.1.0, v3.0.2, v2.6.6
>
>
> Steps to reproduce:
> * setup kylin with SAML as described in
> [http://kylin.apache.org/docs/howto/howto_ldap_and_sso.html].
> kylin.properties:
> {code:java}
> kylin.security.profile=saml
> kylin.security.acl.admin-role=Kylin_Admins
> kylin.security.ldap.connection-server=ldap://openldap:389
> kylin.security.ldap.connection-username=cn=admin,dc=example,dc=org
> # set kylin.security.ldap.connection-password appropriately
> kylin.security.ldap.user-search-base=ou=people,dc=example,dc=org
> kylin.security.ldap.user-search-pattern=(uid={0})
> kylin.security.ldap.user-group-search-base=ou=groups,dc=example,dc=org
> kylin.security.saml.context-context-path=/kylin
> kylin.security.saml.context-scheme=https
> kylin.security.saml.context-server-name=kylin.validdomain.com
> kylin.security.saml.context-server-port=443
> kylin.security.saml.metadata-entity-base-url=https://kylin.validdomain.com/kylin{code}
> * on the LDAP server, make sure you have the following objects in place:
> {code:java}
> # example.user, people, example.org
> dn: uid=example.user,ou=people,dc=example,dc=org
> objectClass: top
> objectClass: account
> objectClass: posixAccount
> objectClass: shadowAccount
> gidNumber: 1
> uidNumber: 5000
> cn: Does not matter
> homeDirectory: /home/doesntmatter
> uid: example.user{code}
> *
> {code:java}
> # Kylin_Users, groups, example.org
> dn: cn=Kylin_Users,ou=groups,dc=example,dc=org
> objectClass: top
> objectClass: groupOfNames
> cn: Kylin_Users
> member: uid=example.user,ou=people,dc=example,dc=org{code}
> * as an ADMIN, create a sample project in kylin and grant QUERY, MANAGEMENT
> or OPERATION access to example.user.
> * now, try logging into kylin.validdomain.com's Web UI as
> [example.u...@validdomain.com.|mailto:example.u...@validdomain.com.]
> Expected result:
> * example.user is logged in, able to select the project from the dropdown
> box at the top left corner and navigate through its properties.
> Actual result:
> * example.user is logged in, but no projects are listed in the dropdown box.
> As if he/she had no permissions in any project.
>
> With LDAP-pure installations (no SAML), this configuration works as expected.
>
> Worth noting:
> [https://github.com/apache/kylin/blob/kylin-3.0.1/server-base/src/main/java/org/apache/kylin/rest/security/SAMLUserDetailsService.java#L40-L54]
> splits the user in the '@' char for performing LDAP lookups. However, by
> editing kylin_metadata manually and appending the @validdomain.com to the
> corresponding object under /acls, the lookup works as it should and the
> non-admin user gets to access the sample project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KYLIN-4551) Provide interfaces to transfer cube/model/project ownership
[ https://issues.apache.org/jira/browse/KYLIN-4551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng reassigned KYLIN-4551: Assignee: Guangxu Cheng (was: Julian Pan) Summary: Provide interfaces to transfer cube/model/project ownership (was: Provide an interface to transfer cube ownership) > Provide interfaces to transfer cube/model/project ownership > --- > > Key: KYLIN-4551 > URL: https://issues.apache.org/jira/browse/KYLIN-4551 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Zhong Yanghong >Assignee: Guangxu Cheng >Priority: Major > > In many companies, employee variation is very common. If cube owners leave > office, it's necessary to transfer the cube ownership to others. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4551) Provide an interface to transfer cube ownership
[ https://issues.apache.org/jira/browse/KYLIN-4551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142670#comment-17142670 ] Guangxu Cheng commented on KYLIN-4551: -- [~julianpan] We have implemented a similar interface，Do you mind if I take this ticket? > Provide an interface to transfer cube ownership > --- > > Key: KYLIN-4551 > URL: https://issues.apache.org/jira/browse/KYLIN-4551 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Zhong Yanghong >Assignee: Julian Pan >Priority: Major > > In many companies, employee variation is very common. If cube owners leave > office, it's necessary to transfer the cube ownership to others. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4578) Throws TableNotFoundException in step 'Convert Cuboid Data to HFile' when the value of property 'kylin.storage.hbase.table-name-prefix' or 'kylin.storage.hbase.namespac
[ https://issues.apache.org/jira/browse/KYLIN-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng reassigned KYLIN-4578: Assignee: Guangxu Cheng > Throws TableNotFoundException in step 'Convert Cuboid Data to HFile' when the > value of property 'kylin.storage.hbase.table-name-prefix' or > 'kylin.storage.hbase.namespace' is lowercase. > - > > Key: KYLIN-4578 > URL: https://issues.apache.org/jira/browse/KYLIN-4578 > Project: Kylin > Issue Type: Bug >Reporter: Zhichao Zhang >Assignee: Guangxu Cheng >Priority: Minor > > If the value of property 'kylin.storage.hbase.table-name-prefix' or > 'kylin.storage.hbase.namespace' is lowercase, it will throw > TableNotFoundException in step 'Convert Cuboid Data to HFile' when build a > segment. > Root cause: > In step 'Create HTable', it uses the lowercase as HTable name, and then in > step 'Convert Cuboid Data to HFile' , the HTable name is coverted to > uppercases. The HTable name is case sensitive. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4572) Add "build/VERSION" to .gitignore
Guangxu Cheng created KYLIN-4572: Summary: Add "build/VERSION" to .gitignore Key: KYLIN-4572 URL: https://issues.apache.org/jira/browse/KYLIN-4572 Project: Kylin Issue Type: Improvement Reporter: Guangxu Cheng Assignee: Guangxu Cheng A temporary file build/VERSION will be generated when executing build/script/package.sh -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4567) Improve TopN merge performance in MR engine
[
https://issues.apache.org/jira/browse/KYLIN-4567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135521#comment-17135521
]
Guangxu Cheng commented on KYLIN-4567:
--
{noformat}
2020-06-15 13:57:14,386 INFO [main] org.apache.kylin.engine.mr.KylinMapper:
Accepting Mapper Key with ordinal: 1
2020-06-15 13:57:14,386 INFO [main] org.apache.kylin.engine.mr.KylinMapper: Do
map, available memory: 3480m
2020-06-15 13:57:46,733 INFO [main] org.apache.kylin.engine.mr.KylinMapper:
Accepting Mapper Key with ordinal: 11
2020-06-15 13:57:46,733 INFO [main] org.apache.kylin.engine.mr.KylinMapper: Do
map, available memory: 2806m
2020-06-15 13:58:13,424 INFO [main] org.apache.kylin.engine.mr.KylinMapper:
Accepting Mapper Key with ordinal: 21
2020-06-15 13:58:13,424 INFO [main] org.apache.kylin.engine.mr.KylinMapper: Do
map, available memory: 2336m
2020-06-15 13:58:16,037 INFO [main] org.apache.hadoop.mapred.MapTask: Spilling
map output
2020-06-15 13:58:16,037 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart
= 0; bufend = 80520185; bufvoid = 104857600
2020-06-15 13:58:16,037 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart =
26214396(104857584); kvend = 25372888(101491552); length = 841509/6553600
2020-06-15 13:58:16,037 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR)
81363241 kvi 20340804(81363216)
2020-06-15 13:58:16,106 INFO [SpillThread]
org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.deflate]
2020-06-15 13:58:16,116 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Do setup, available memory: 2577m
2020-06-15 13:58:16,119 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer Key with ordinal: 1
2020-06-15 13:58:16,120 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Do reduce, available memory: 2576m
2020-06-15 13:58:16,120 INFO [SpillThread]
org.apache.kylin.engine.mr.steps.CuboidReducer: Handling value with ordinal
(This is not KV number!): 1
2020-06-15 13:58:17,975 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Do cleanup, available memory: 2991m
2020-06-15 13:58:17,975 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Total rows: 1
2020-06-15 13:58:17,979 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Do setup, available memory: 2991m
2020-06-15 13:58:17,980 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer Key with ordinal: 1
2020-06-15 13:58:17,980 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Do reduce, available memory: 2991m
2020-06-15 13:58:17,980 INFO [SpillThread]
org.apache.kylin.engine.mr.steps.CuboidReducer: Handling value with ordinal
(This is not KV number!): 1
2020-06-15 13:58:28,764 INFO [main] org.apache.kylin.engine.mr.KylinMapper: Do
cleanup, available memory: 3283m
2020-06-15 13:58:28,764 INFO [main] org.apache.kylin.engine.mr.KylinMapper:
Total rows: 255999
2020-06-15 13:58:28,767 INFO [main] org.apache.hadoop.mapred.MapTask: Starting
flush of map output
2020-06-15 13:58:33,280 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Do cleanup, available memory: 3048m
2020-06-15 13:58:33,280 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Total rows: 1
2020-06-15 13:58:33,282 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Do setup, available memory: 3048m
2020-06-15 13:58:33,283 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer Key with ordinal: 1
2020-06-15 13:58:33,283 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Do reduce, available memory: 3048m
2020-06-15 13:58:33,283 INFO [SpillThread]
org.apache.kylin.engine.mr.steps.CuboidReducer: Handling value with ordinal
(This is not KV number!): 1
2020-06-15 13:59:22,359 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Do cleanup, available memory: 3305m
2020-06-15 13:59:22,359 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Total rows: 1
2020-06-15 13:59:22,374 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Do setup, available memory: 3305m
2020-06-15 13:59:22,374 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Accepting Reducer Key with ordinal: 1
2020-06-15 13:59:22,374 INFO [SpillThread]
org.apache.kylin.engine.mr.KylinReducer: Do reduce, available memory: 3305m
2020-06-15 13:59:22,374 INFO [SpillThread]
org.apache.kylin.engine.mr.steps.CuboidReducer: Handling value with ordinal
(This is not KV number!): 1
2020-06-15 14:39:34,082 INFO [SpillThread]
org.apache.kylin.engine.mr.steps.CuboidReducer: Handling value with ordinal
(This is not KV number!): 11
{noformat}
>From the map logs we can find that calculating the TOPN of 13 columns takes
>about 40 minutes to process 100k rows, which is too slow.
> Improve TopN merge performance in MR engine
> ---
>
> Key: KYLIN-4567
>
[jira] [Created] (KYLIN-4567) Improve TopN merge performance in MR engine
Guangxu Cheng created KYLIN-4567:
Summary: Improve TopN merge performance in MR engine
Key: KYLIN-4567
URL: https://issues.apache.org/jira/browse/KYLIN-4567
Project: Kylin
Issue Type: Improvement
Components: Measure - TopN
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
We have a cube that needs to calculate the TOPN of 13 columns. The number of
data source is only 500k. But, the cubing job always fail when building base
cuboid.
we found that the map task always killed by the ApplicationMaster due to time
out
{noformat}
ERROR-[-10001]-[MR]:[Mr Task
Timeout]:[AttemptID:attempt_1591996262448_229922_m_00_1 Timed out after
3600 secs!] ERROR-[-10015]-[MR]:[Container Exit Accidentally]:[Container killed
by the ApplicationMaster. Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143 ]
{noformat}
the stack information as below:
{noformat}
"SpillThread" #35 daemon prio=5 os_prio=0 tid=0x7f9a89771800 nid=0x133a2
runnable [0x7f9a56e3f000]
java.lang.Thread.State: RUNNABLE
at java.util.LinkedList.toArray(LinkedList.java:1052)
at java.util.List.sort(List.java:477)
at java.util.Collections.sort(Collections.java:175)
at
org.apache.kylin.measure.topn.TopNCounter.sortAndRetain(TopNCounter.java:96)
at org.apache.kylin.measure.topn.TopNCounter.merge(TopNCounter.java:183)
at
org.apache.kylin.measure.topn.TopNAggregator.aggregate(TopNAggregator.java:44)
at
org.apache.kylin.measure.topn.TopNAggregator.aggregate(TopNAggregator.java:27)
at
org.apache.kylin.measure.MeasureAggregators.aggregate(MeasureAggregators.java:83)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.doReduce(CuboidReducer.java:108)
at
org.apache.kylin.engine.mr.steps.CuboidReducer.doReduce(CuboidReducer.java:44)
at org.apache.kylin.engine.mr.KylinReducer.reduce(KylinReducer.java:77)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at
org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1688)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1645)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:884)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1540)
{noformat}
>From the stack information, we found that sorting takes a lot of time. After
>merge another counter into this counter, need to re-sort this counter. Maybe
>we can reduce the frequency of sorting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4553) Expose job filtered with more job types
[ https://issues.apache.org/jira/browse/KYLIN-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133220#comment-17133220 ] Guangxu Cheng edited comment on KYLIN-4553 at 6/12/20, 9:21 AM: Sorry, I miss this issue before creating KYLIN-4556, this issue is part of KYLIN-4556 which has been implenmented. was (Author: andrewcheng): Sorry, I miss this issue before creating KYLIN-4553, this issue is part of KYLIN-4553 which has been implenmented. > Expose job filtered with more job types > --- > > Key: KYLIN-4553 > URL: https://issues.apache.org/jira/browse/KYLIN-4553 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Zhong Yanghong >Assignee: Julian Pan >Priority: Major > > It's better to expose jobs by filtering with all job types. Currently jobs > only be filtered by "Cubing" & "CheckPoint". There're more job types, like > "Lookup snapshot", "Calculate Cardinality", etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4553) Expose job filtered with more job types
[ https://issues.apache.org/jira/browse/KYLIN-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133220#comment-17133220 ] Guangxu Cheng commented on KYLIN-4553: -- Sorry, I miss this issue before creating KYLIN-4553, this issue is part of KYLIN-4553 which has been implenmented. > Expose job filtered with more job types > --- > > Key: KYLIN-4553 > URL: https://issues.apache.org/jira/browse/KYLIN-4553 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Zhong Yanghong >Assignee: Julian Pan >Priority: Major > > It's better to expose jobs by filtering with all job types. Currently jobs > only be filtered by "Cubing" & "CheckPoint". There're more job types, like > "Lookup snapshot", "Calculate Cardinality", etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4561) overall job number statistics in monitor page is incorrect after change the job status
[ https://issues.apache.org/jira/browse/KYLIN-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-4561: - Parent: KYLIN-4556 Issue Type: Sub-task (was: Bug) > overall job number statistics in monitor page is incorrect after change the > job status > -- > > Key: KYLIN-4561 > URL: https://issues.apache.org/jira/browse/KYLIN-4561 > Project: Kylin > Issue Type: Sub-task >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: image-2020-06-11-15-40-18-370.png, > image-2020-06-11-15-41-36-058.png > > > !image-2020-06-11-15-41-36-058.png|width=587,height=45! > When the job status is changed, the statistics in monitor page will not > change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4561) overall job number statistics in monitor page is incorrect after change the job status
Guangxu Cheng created KYLIN-4561: Summary: overall job number statistics in monitor page is incorrect after change the job status Key: KYLIN-4561 URL: https://issues.apache.org/jira/browse/KYLIN-4561 Project: Kylin Issue Type: Bug Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-06-11-15-40-18-370.png, image-2020-06-11-15-41-36-058.png !image-2020-06-11-15-41-36-058.png|width=587,height=45! When the job status is changed, the statistics in monitor page will not change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4560) support to re-run/delete cardinality and lookup snapshot job
Guangxu Cheng created KYLIN-4560: Summary: support to re-run/delete cardinality and lookup snapshot job Key: KYLIN-4560 URL: https://issues.apache.org/jira/browse/KYLIN-4560 Project: Kylin Issue Type: Sub-task Reporter: Guangxu Cheng Assignee: Guangxu Cheng -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4559) show cardinality and lookup snapshot job on job page
Guangxu Cheng created KYLIN-4559: Summary: show cardinality and lookup snapshot job on job page Key: KYLIN-4559 URL: https://issues.apache.org/jira/browse/KYLIN-4559 Project: Kylin Issue Type: Sub-task Reporter: Guangxu Cheng Assignee: Guangxu Cheng -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4558) get all chained executable jobs through job API
Guangxu Cheng created KYLIN-4558: Summary: get all chained executable jobs through job API Key: KYLIN-4558 URL: https://issues.apache.org/jira/browse/KYLIN-4558 Project: Kylin Issue Type: Sub-task Reporter: Guangxu Cheng Assignee: Guangxu Cheng -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4557) Refactor JobService to improve code readability
Guangxu Cheng created KYLIN-4557: Summary: Refactor JobService to improve code readability Key: KYLIN-4557 URL: https://issues.apache.org/jira/browse/KYLIN-4557 Project: Kylin Issue Type: Sub-task Reporter: Guangxu Cheng Assignee: Guangxu Cheng -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4556) improve job service
Guangxu Cheng created KYLIN-4556: Summary: improve job service Key: KYLIN-4556 URL: https://issues.apache.org/jira/browse/KYLIN-4556 Project: Kylin Issue Type: Task Reporter: Guangxu Cheng Assignee: Guangxu Cheng At present, only cubing and checkpoint jobs can be obtained through job API,for cardinality and lookup snapshot job, we can't get them which is very inconvenient. when the cardinality task fails, we can only find the reason why the task failed from the system log, and we cannot rerun or delete these failed tasks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4539) Spark build failed due to com.codahale.metrics.json.MetricsModule loaded from kylin job jar
[
https://issues.apache.org/jira/browse/KYLIN-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guangxu Cheng updated KYLIN-4539:
-
Description:
{code:sh}
[Loaded com.fasterxml.jackson.databind.Module from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/jackson-databind-2.9.4.jar]
[Loaded org.apache.kylin.job.shaded.com.fasterxml.jackson.core.Versioned from
file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
[Loaded org.apache.kylin.job.shaded.com.fasterxml.jackson.databind.Module from
file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
[Loaded com.codahale.metrics.json.MetricsModule from
file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
[Loaded java.lang.VerifyError from
/data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
scala.util.control.NonFatal$ from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/scala-library-2.11.8.jar]
[Loaded org.apache.spark.SparkUserAppException from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/spark-core_2.11-3.12.0.jar]
[Loaded org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4$$anonfun$run$11
from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/spark-yarn_2.11-3.12.0.jar]
[Loaded org.apache.log4j.spi.ThrowableInformation from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
[Loaded org.apache.log4j.spi.ThrowableRenderer from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
[Loaded org.apache.log4j.DefaultThrowableRenderer from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
[Loaded java.lang.Throwable$PrintStreamOrWriter from
/data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
java.lang.Throwable$WrappedPrintWriter from
/data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
java.io.LineNumberReader from
/data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] User class threw
exception: java.lang.VerifyError: Bad type on operand stack Exception Details:
Location:
org/apache/spark/metrics/sink/MetricsServlet.(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V
@116: invokevirtual Reason: Type 'com/codahale/metrics/json/MetricsModule'
(current frame, stack[2]) is not assignable to
'com/fasterxml/jackson/databind/Module' Current Frame: bci: @116 flags: \{ }
locals: \{ 'org/apache/spark/metrics/sink/MetricsServlet',
'java/util/Properties', 'com/codahale/metrics/MetricRegistry',
'org/apache/spark/SecurityManager' } stack: \{
'org/apache/spark/metrics/sink/MetricsServlet',
'com/fasterxml/jackson/databind/ObjectMapper',
'com/codahale/metrics/json/MetricsModule' } Bytecode: 0x000: 2a2b b500 2a2a
2cb5 002f 2a2d b500 5c2a 0x010: b700 7e2a 1280 b500 322a 1282 b500 342a
0x020: 03b5 0037 2a2b 2ab6 0084 b600 8ab5 0039 0x030: 2ab2 008f 2b2a
b600 91b6 008a b600 95bb 0x040: 0014 592a b700 96b6 009c bb00 1659 2ab7
0x050: 009d b600 a1b8 00a7 b500 3b2a bb00 7159 0x060: b700 a8bb 00aa
59b2 00b0 b200 b32a b600 0x070: b5b7 00b8 b600 bcb5 003e b1
java.lang.VerifyError: Bad type on operand stack Exception Details: Location:
org/apache/spark/metrics/sink/MetricsServlet.(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V
@116: invokevirtual Reason: Type 'com/codahale/metrics/json/MetricsModule'
(current frame, stack[2]) is not assignable to
'com/fasterxml/jackson/databind/Module' Current Frame: bci: @116 flags: \{ }
locals: \{ 'org/apache/spark/metrics/sink/MetricsServlet',
'java/util/Properties', 'com/codahale/metrics/MetricRegistry',
'org/apache/spark/SecurityManager' } stack: \{
'org/apache/spark/metrics/sink/MetricsServlet',
'com/fasterxml/jackson/databind/ObjectMapper',
'com/codahale/metrics/json/MetricsModule' } Bytecode: 0x000: 2a2b b500 2a2a
2cb5 002f 2a2d b500 5c2a 0x010: b700 7e2a 1280 b500 322a 1282 b500 342a
0x020: 03b5 0037 2a2b 2ab6 0084 b600 8ab5 0039 0x030: 2ab2 008f 2b2a
b600 91b6 008a b600 95bb 0x040: 0014 592a b700 96b6 009c bb00 1659 2ab7
0x050: 009d b600 a1b8 00a7 b500 3b2a bb00 7159 0x060: b700 a8bb 00aa
59b2 00b0 b200 b32a b600 0x070: b5b7 00b8 b600 bcb5 003e b1 at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:348) at
org.apache.spark.util.Utils$.classForName(Utils.scala:238) at
org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:198)
at
org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:194)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at
[jira] [Created] (KYLIN-4539) Spark build failed due to com.codahale.metrics.json.MetricsModule loaded from kylin job jar
Guangxu Cheng created KYLIN-4539:
Summary: Spark build failed due to
com.codahale.metrics.json.MetricsModule loaded from kylin job jar
Key: KYLIN-4539
URL: https://issues.apache.org/jira/browse/KYLIN-4539
Project: Kylin
Issue Type: Bug
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
{code:shell}
[Loaded com.fasterxml.jackson.databind.Module from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/jackson-databind-2.9.4.jar]
[Loaded org.apache.kylin.job.shaded.com.fasterxml.jackson.core.Versioned from
file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
[Loaded org.apache.kylin.job.shaded.com.fasterxml.jackson.databind.Module from
file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
[Loaded com.codahale.metrics.json.MetricsModule from
file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
[Loaded java.lang.VerifyError from
/data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
scala.util.control.NonFatal$ from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/scala-library-2.11.8.jar]
[Loaded org.apache.spark.SparkUserAppException from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/spark-core_2.11-3.12.0.jar]
[Loaded org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4$$anonfun$run$11
from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/spark-yarn_2.11-3.12.0.jar]
[Loaded org.apache.log4j.spi.ThrowableInformation from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
[Loaded org.apache.log4j.spi.ThrowableRenderer from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
[Loaded org.apache.log4j.DefaultThrowableRenderer from
file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
[Loaded java.lang.Throwable$PrintStreamOrWriter from
/data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
java.lang.Throwable$WrappedPrintWriter from
/data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
java.io.LineNumberReader from
/data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] User class threw
exception: java.lang.VerifyError: Bad type on operand stack Exception Details:
Location:
org/apache/spark/metrics/sink/MetricsServlet.(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V
@116: invokevirtual Reason: Type 'com/codahale/metrics/json/MetricsModule'
(current frame, stack[2]) is not assignable to
'com/fasterxml/jackson/databind/Module' Current Frame: bci: @116 flags: \{ }
locals: \{ 'org/apache/spark/metrics/sink/MetricsServlet',
'java/util/Properties', 'com/codahale/metrics/MetricRegistry',
'org/apache/spark/SecurityManager' } stack: \{
'org/apache/spark/metrics/sink/MetricsServlet',
'com/fasterxml/jackson/databind/ObjectMapper',
'com/codahale/metrics/json/MetricsModule' } Bytecode: 0x000: 2a2b b500 2a2a
2cb5 002f 2a2d b500 5c2a 0x010: b700 7e2a 1280 b500 322a 1282 b500 342a
0x020: 03b5 0037 2a2b 2ab6 0084 b600 8ab5 0039 0x030: 2ab2 008f 2b2a
b600 91b6 008a b600 95bb 0x040: 0014 592a b700 96b6 009c bb00 1659 2ab7
0x050: 009d b600 a1b8 00a7 b500 3b2a bb00 7159 0x060: b700 a8bb 00aa
59b2 00b0 b200 b32a b600 0x070: b5b7 00b8 b600 bcb5 003e b1
java.lang.VerifyError: Bad type on operand stack Exception Details: Location:
org/apache/spark/metrics/sink/MetricsServlet.(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V
@116: invokevirtual Reason: Type 'com/codahale/metrics/json/MetricsModule'
(current frame, stack[2]) is not assignable to
'com/fasterxml/jackson/databind/Module' Current Frame: bci: @116 flags: \{ }
locals: \{ 'org/apache/spark/metrics/sink/MetricsServlet',
'java/util/Properties', 'com/codahale/metrics/MetricRegistry',
'org/apache/spark/SecurityManager' } stack: \{
'org/apache/spark/metrics/sink/MetricsServlet',
'com/fasterxml/jackson/databind/ObjectMapper',
'com/codahale/metrics/json/MetricsModule' } Bytecode: 0x000: 2a2b b500 2a2a
2cb5 002f 2a2d b500 5c2a 0x010: b700 7e2a 1280 b500 322a 1282 b500 342a
0x020: 03b5 0037 2a2b 2ab6 0084 b600 8ab5 0039 0x030: 2ab2 008f 2b2a
b600 91b6 008a b600 95bb 0x040: 0014 592a b700 96b6 009c bb00 1659 2ab7
0x050: 009d b600 a1b8 00a7 b500 3b2a bb00 7159 0x060: b700 a8bb 00aa
59b2 00b0 b200 b32a b600 0x070: b5b7 00b8 b600 bcb5 003e b1 at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:348) at
org.apache.spark.util.Utils$.classForName(Utils.scala:238) at
org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:198)
at
org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:194)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
[jira] [Created] (KYLIN-4537) Give a friendly tips to the user when getting task list fails
Guangxu Cheng created KYLIN-4537: Summary: Give a friendly tips to the user when getting task list fails Key: KYLIN-4537 URL: https://issues.apache.org/jira/browse/KYLIN-4537 Project: Kylin Issue Type: Improvement Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-05-27-21-47-34-128.png, image-2020-05-27-21-48-57-223.png On the monitoring page, when the task list acquisition fails, it will show "failed to load jobs", but we don't know the detail reason for the failure, whether it is due to permission problems or the system crash? Before: !image-2020-05-27-21-48-57-223.png|width=491,height=314! After: !image-2020-05-27-21-47-34-128.png|width=441,height=310! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4527) Beautify the drop-down list of the cube on query page
[ https://issues.apache.org/jira/browse/KYLIN-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-4527: - Attachment: (was: image-2020-05-27-12-21-54-146.png) > Beautify the drop-down list of the cube on query page > - > > Key: KYLIN-4527 > URL: https://issues.apache.org/jira/browse/KYLIN-4527 > Project: Kylin > Issue Type: Improvement >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: image-2020-05-27-12-05-49-425.png, > image-2020-05-27-12-22-19-097.png > > > The drop-down list of cube is very compact, which is not convenient to select > cube > Before: > !image-2020-05-27-12-05-49-425.png|width=424,height=212! > After: > !image-2020-05-27-12-22-19-097.png|width=429,height=249! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4527) Beautify the drop-down list of the cube on query page
[ https://issues.apache.org/jira/browse/KYLIN-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-4527: - Attachment: image-2020-05-27-12-22-19-097.png > Beautify the drop-down list of the cube on query page > - > > Key: KYLIN-4527 > URL: https://issues.apache.org/jira/browse/KYLIN-4527 > Project: Kylin > Issue Type: Improvement >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: image-2020-05-27-12-05-49-425.png, > image-2020-05-27-12-21-54-146.png, image-2020-05-27-12-22-19-097.png > > > !image-2020-05-27-12-05-49-425.png|width=336,height=168! > The drop-down list of cube is very compact, which is not convenient to select > cube > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4527) Beautify the drop-down list of the cube on query page
[ https://issues.apache.org/jira/browse/KYLIN-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-4527: - Description: The drop-down list of cube is very compact, which is not convenient to select cube Before: !image-2020-05-27-12-05-49-425.png|width=424,height=212! After: !image-2020-05-27-12-22-19-097.png|width=429,height=249! was: !image-2020-05-27-12-05-49-425.png|width=336,height=168! The drop-down list of cube is very compact, which is not convenient to select cube > Beautify the drop-down list of the cube on query page > - > > Key: KYLIN-4527 > URL: https://issues.apache.org/jira/browse/KYLIN-4527 > Project: Kylin > Issue Type: Improvement >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: image-2020-05-27-12-05-49-425.png, > image-2020-05-27-12-21-54-146.png, image-2020-05-27-12-22-19-097.png > > > The drop-down list of cube is very compact, which is not convenient to select > cube > Before: > !image-2020-05-27-12-05-49-425.png|width=424,height=212! > After: > !image-2020-05-27-12-22-19-097.png|width=429,height=249! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4527) Beautify the drop-down list of the cube on query page
[ https://issues.apache.org/jira/browse/KYLIN-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-4527: - Attachment: image-2020-05-27-12-21-54-146.png > Beautify the drop-down list of the cube on query page > - > > Key: KYLIN-4527 > URL: https://issues.apache.org/jira/browse/KYLIN-4527 > Project: Kylin > Issue Type: Improvement >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: image-2020-05-27-12-05-49-425.png, > image-2020-05-27-12-21-54-146.png > > > !image-2020-05-27-12-05-49-425.png|width=336,height=168! > The drop-down list of cube is very compact, which is not convenient to select > cube > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4527) Beautify the drop-down list of the cube on query page
Guangxu Cheng created KYLIN-4527: Summary: Beautify the drop-down list of the cube on query page Key: KYLIN-4527 URL: https://issues.apache.org/jira/browse/KYLIN-4527 Project: Kylin Issue Type: Improvement Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-05-27-12-05-49-425.png !image-2020-05-27-12-05-49-425.png|width=336,height=168! The drop-down list of cube is very compact, which is not convenient to select cube -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4526) Enhance get the hive table rows
Guangxu Cheng created KYLIN-4526: Summary: Enhance get the hive table rows Key: KYLIN-4526 URL: https://issues.apache.org/jira/browse/KYLIN-4526 Project: Kylin Issue Type: Task Reporter: Guangxu Cheng Assignee: Guangxu Cheng In kylin-4315, we get the rows of the hive table from metadata, but when we turn off hive's statistics feature(`hive.stats.autogather=false`), we can't get the correct rows of hive table from metadata -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4511) Support project admin user to calculate column cardinality by web
[ https://issues.apache.org/jira/browse/KYLIN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-4511: - Description: Now only the administrator of kylin can recalculate the cardinality of the hive table through the admin web, but the project administrator cannot. Add an entry to calculate cardinality on the table detail web # add calculate cardinality button !image-2020-05-21-13-56-19-962.png|width=902,height=303! # confirm !image-2020-05-21-13-57-29-350.png|width=901,height=280! was: Now only the administrator of kylin can recalculate the cardinality of the hive table through the admin web, but the project administrator cannot. Add an entry to calculate cardinality on the table detail web # add calculate cardinality button > Support project admin user to calculate column cardinality by web > - > > Key: KYLIN-4511 > URL: https://issues.apache.org/jira/browse/KYLIN-4511 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: image-2020-05-21-13-56-19-962.png, > image-2020-05-21-13-57-29-350.png > > > Now only the administrator of kylin can recalculate the cardinality of the > hive table through the admin web, but the project administrator cannot. > Add an entry to calculate cardinality on the table detail web > # add calculate cardinality button > !image-2020-05-21-13-56-19-962.png|width=902,height=303! > # confirm !image-2020-05-21-13-57-29-350.png|width=901,height=280! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4511) Support project admin user to calculate column cardinality by web
[ https://issues.apache.org/jira/browse/KYLIN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-4511: - Attachment: image-2020-05-21-13-57-29-350.png > Support project admin user to calculate column cardinality by web > - > > Key: KYLIN-4511 > URL: https://issues.apache.org/jira/browse/KYLIN-4511 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: image-2020-05-21-13-56-19-962.png, > image-2020-05-21-13-57-29-350.png > > > Now only the administrator of kylin can recalculate the cardinality of the > hive table through the admin web, but the project administrator cannot. > Add an entry to calculate cardinality on the table detail web > # add calculate cardinality button -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4511) Support project admin user to calculate column cardinality by web
[ https://issues.apache.org/jira/browse/KYLIN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-4511: - Attachment: image-2020-05-21-13-56-19-962.png > Support project admin user to calculate column cardinality by web > - > > Key: KYLIN-4511 > URL: https://issues.apache.org/jira/browse/KYLIN-4511 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: image-2020-05-21-13-56-19-962.png > > > Now only the administrator of kylin can recalculate the cardinality of the > hive table through the admin web, but the project administrator cannot. > Add an entry to calculate cardinality on the table detail web > # add calculate cardinality button -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4511) Support project admin user to calculate column cardinality by web
[ https://issues.apache.org/jira/browse/KYLIN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-4511: - Description: Now only the administrator of kylin can recalculate the cardinality of the hive table through the admin web, but the project administrator cannot. Add an entry to calculate cardinality on the table detail web # add calculate cardinality button was: Now only the administrator of kylin can recalculate the cardinality of the hive table through the admin web, but the project administrator cannot. Add an entry to calculate cardinality on the table detail web 1. add calculate cardinality button > Support project admin user to calculate column cardinality by web > - > > Key: KYLIN-4511 > URL: https://issues.apache.org/jira/browse/KYLIN-4511 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > > Now only the administrator of kylin can recalculate the cardinality of the > hive table through the admin web, but the project administrator cannot. > Add an entry to calculate cardinality on the table detail web > # add calculate cardinality button -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4511) Support project admin user to calculate column cardinality by web
[ https://issues.apache.org/jira/browse/KYLIN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-4511: - Description: Now only the administrator of kylin can recalculate the cardinality of the hive table through the admin web, but the project administrator cannot. Add an entry to calculate cardinality on the table detail web 1. add calculate cardinality button was: Now only the administrator of kylin can recalculate the cardinality of the hive table through the admin web, but the project administrator cannot. Add an entry to calculate cardinality on the table detail web > Support project admin user to calculate column cardinality by web > - > > Key: KYLIN-4511 > URL: https://issues.apache.org/jira/browse/KYLIN-4511 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > > Now only the administrator of kylin can recalculate the cardinality of the > hive table through the admin web, but the project administrator cannot. > Add an entry to calculate cardinality on the table detail web > 1. add calculate cardinality button -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4511) Support project admin user to calculate column cardinality by web
Guangxu Cheng created KYLIN-4511: Summary: Support project admin user to calculate column cardinality by web Key: KYLIN-4511 URL: https://issues.apache.org/jira/browse/KYLIN-4511 Project: Kylin Issue Type: Improvement Components: Web Reporter: Guangxu Cheng Assignee: Guangxu Cheng Now only the administrator of kylin can recalculate the cardinality of the hive table through the admin web, but the project administrator cannot. Add an entry to calculate cardinality on the table detail web -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4510) Automatically refresh the page after reload table
Guangxu Cheng created KYLIN-4510: Summary: Automatically refresh the page after reload table Key: KYLIN-4510 URL: https://issues.apache.org/jira/browse/KYLIN-4510 Project: Kylin Issue Type: Improvement Components: Web Reporter: Guangxu Cheng Assignee: Guangxu Cheng Now after reload table, we need to manually refresh the page to get the latest metadata information, which is unfriendly -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4509) get hive table rows from metadata when using beeline
Guangxu Cheng created KYLIN-4509: Summary: get hive table rows from metadata when using beeline Key: KYLIN-4509 URL: https://issues.apache.org/jira/browse/KYLIN-4509 Project: Kylin Issue Type: Improvement Affects Versions: v3.0.1, v2.6.5 Reporter: Guangxu Cheng Assignee: Guangxu Cheng Now by running SQL "select count(*) from table" to get the number of the hive table records, the performance is relatively poor and will consume a lot of resources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4490) Fix minor bug which add measure on frontend
Guangxu Cheng created KYLIN-4490: Summary: Fix minor bug which add measure on frontend Key: KYLIN-4490 URL: https://issues.apache.org/jira/browse/KYLIN-4490 Project: Kylin Issue Type: Bug Components: Web Affects Versions: v3.0.1, v2.6.5, v4.0.0-beta Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2020-05-11-14-52-30-494.png !image-2020-05-11-14-52-30-494.png|width=1227,height=194! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4481) Project-level ACL lookups not working for non-admin SAML-federated users
[
https://issues.apache.org/jira/browse/KYLIN-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guangxu Cheng reassigned KYLIN-4481:
Assignee: Guangxu Cheng
> Project-level ACL lookups not working for non-admin SAML-federated users
>
>
> Key: KYLIN-4481
> URL: https://issues.apache.org/jira/browse/KYLIN-4481
> Project: Kylin
> Issue Type: Bug
> Components: Security
>Affects Versions: v2.6.5, v3.0.1
>Reporter: Rafael Felix Correa
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v3.1.0, v3.0.2, v2.6.6
>
>
> Steps to reproduce:
> * setup kylin with SAML as described in
> [http://kylin.apache.org/docs/howto/howto_ldap_and_sso.html].
> kylin.properties:
> {code:java}
> kylin.security.profile=saml
> kylin.security.acl.admin-role=Kylin_Admins
> kylin.security.ldap.connection-server=ldap://openldap:389
> kylin.security.ldap.connection-username=cn=admin,dc=example,dc=org
> # set kylin.security.ldap.connection-password appropriately
> kylin.security.ldap.user-search-base=ou=people,dc=example,dc=org
> kylin.security.ldap.user-search-pattern=(uid={0})
> kylin.security.ldap.user-group-search-base=ou=groups,dc=example,dc=org
> kylin.security.saml.context-context-path=/kylin
> kylin.security.saml.context-scheme=https
> kylin.security.saml.context-server-name=kylin.validdomain.com
> kylin.security.saml.context-server-port=443
> kylin.security.saml.metadata-entity-base-url=https://kylin.validdomain.com/kylin{code}
> * on the LDAP server, make sure you have the following objects in place:
> {code:java}
> # example.user, people, example.org
> dn: uid=example.user,ou=people,dc=example,dc=org
> objectClass: top
> objectClass: account
> objectClass: posixAccount
> objectClass: shadowAccount
> gidNumber: 1
> uidNumber: 5000
> cn: Does not matter
> homeDirectory: /home/doesntmatter
> uid: example.user{code}
> *
> {code:java}
> # Kylin_Users, groups, example.org
> dn: cn=Kylin_Users,ou=groups,dc=example,dc=org
> objectClass: top
> objectClass: groupOfNames
> cn: Kylin_Users
> member: uid=example.user,ou=people,dc=example,dc=org{code}
> * as an ADMIN, create a sample project in kylin and grant QUERY, MANAGEMENT
> or OPERATION access to example.user.
> * now, try logging into kylin.validdomain.com's Web UI as
> [example.u...@validdomain.com.|mailto:example.u...@validdomain.com.]
> Expected result:
> * example.user is logged in, able to select the project from the dropdown
> box at the top left corner and navigate through its properties.
> Actual result:
> * example.user is logged in, but no projects are listed in the dropdown box.
> As if he/she had no permissions in any project.
>
> With LDAP-pure installations (no SAML), this configuration works as expected.
>
> Worth noting:
> [https://github.com/apache/kylin/blob/kylin-3.0.1/server-base/src/main/java/org/apache/kylin/rest/security/SAMLUserDetailsService.java#L40-L54]
> splits the user in the '@' char for performing LDAP lookups. However, by
> editing kylin_metadata manually and appending the @validdomain.com to the
> corresponding object under /acls, the lookup works as it should and the
> non-admin user gets to access the sample project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KYLIN-4145) compile failed due to incompatible version between scala and scala-maven-plugin
[
https://issues.apache.org/jira/browse/KYLIN-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guangxu Cheng resolved KYLIN-4145.
--
Resolution: Won't Fix
> compile failed due to incompatible version between scala and
> scala-maven-plugin
> ---
>
> Key: KYLIN-4145
> URL: https://issues.apache.org/jira/browse/KYLIN-4145
> Project: Kylin
> Issue Type: Bug
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
>
> The version of scala-maven-plugin is not specified, and the latest
> version(4.1.1) will be used when compiling the project, which is incompatible
> with the version of scala(2.11.0).
> So, it would be better to specify the version number of scala-maven-plugin.
> {code:xml}
> [ERROR] Failed to execute goal
> net.alchim31.maven:scala-maven-plugin:4.1.1:compile (scala-compile-first) on
> project kylin-engine-spark: Execution scala-compile-first of goal
> net.alchim31.maven:scala-maven-plugin:4.1.1:compile failed: An API
> incompatibility was encountered while executing
> net.alchim31.maven:scala-maven-plugin:4.1.1:compile:
> java.lang.NoSuchMethodError:
> xsbt.ZincCompiler.classPath()Lscala/tools/nsc/util/ClassFileLookup;
> [ERROR] -
> [ERROR] realm =plugin>net.alchim31.maven:scala-maven-plugin:4.1.1
> [ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
> [ERROR] urls[0] =
> file:/Users/cheng/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.1.1/scala-maven-plugin-4.1.1.jar
> [ERROR] urls[1] =
> file:/Users/cheng/.m2/repository/org/apache/maven/maven-builder-support/3.3.9/maven-builder-support-3.3.9.jar
> [ERROR] urls[2] =
> file:/Users/cheng/.m2/repository/com/google/guava/guava/18.0/guava-18.0.jar
> [ERROR] urls[3] =
> file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.21/plexus-interpolation-1.21.jar
> [ERROR] urls[4] =
> file:/Users/cheng/.m2/repository/javax/enterprise/cdi-api/1.0/cdi-api-1.0.jar
> [ERROR] urls[5] =
> file:/Users/cheng/.m2/repository/org/eclipse/sisu/org.eclipse.sisu.inject/0.3.2/org.eclipse.sisu.inject-0.3.2.jar
> [ERROR] urls[6] =
> file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-component-annotations/1.6/plexus-component-annotations-1.6.jar
> [ERROR] urls[7] =
> file:/Users/cheng/.m2/repository/org/apache/maven/reporting/maven-reporting-api/3.0/maven-reporting-api-3.0.jar
> [ERROR] urls[8] =
> file:/Users/cheng/.m2/repository/org/eclipse/aether/aether-util/1.0.2.v20150114/aether-util-1.0.2.v20150114.jar
> [ERROR] urls[9] =
> file:/Users/cheng/.m2/repository/com/google/inject/guice/4.0/guice-4.0-no_aop.jar
> [ERROR] urls[10] =
> file:/Users/cheng/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar
> [ERROR] urls[11] =
> file:/Users/cheng/.m2/repository/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.jar
> [ERROR] urls[12] =
> file:/Users/cheng/.m2/repository/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.jar
> [ERROR] urls[13] =
> file:/Users/cheng/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
> [ERROR] urls[14] =
> file:/Users/cheng/.m2/repository/org/apache/maven/shared/maven-dependency-tree/3.0.1/maven-dependency-tree-3.0.1.jar
> [ERROR] urls[15] =
> file:/Users/cheng/.m2/repository/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar
> [ERROR] urls[16] =
> file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-utils/3.2.1/plexus-utils-3.2.1.jar
> [ERROR] urls[17] =
> file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-archiver/4.1.0/plexus-archiver-4.1.0.jar
> [ERROR] urls[18] =
> file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-io/3.1.1/plexus-io-3.1.1.jar
> [ERROR] urls[19] =
> file:/Users/cheng/.m2/repository/org/apache/commons/commons-compress/1.18/commons-compress-1.18.jar
> [ERROR] urls[20] =
> file:/Users/cheng/.m2/repository/org/iq80/snappy/snappy/0.4/snappy-0.4.jar
> [ERROR] urls[21] =
> file:/Users/cheng/.m2/repository/org/tukaani/xz/1.8/xz-1.8.jar
> [ERROR] urls[22] =
> file:/Users/cheng/.m2/repository/backport-util-concurrent/backport-util-concurrent/3.1/backport-util-concurrent-3.1.jar
> [ERROR] urls[23] =
> file:/Users/cheng/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar
> [ERROR] urls[24] =
> file:/Users/cheng/.m2/repository/org/apache/maven/maven-archiver/3.4.0/maven-archiver-3.4.0.jar
> [ERROR] urls[25] =
> file:/Users/cheng/.m2/repository/org/apache/maven/shared/maven-shared-utils/3.2.1/maven-shared-utils-3.2.1.jar
> [ERROR] urls[26] =
> file:/Users/cheng/.m2/repository/commons-io/commons-io/2.5/commons-io-2.5.jar
> [ERROR] urls[27] =
> file:/Users/cheng/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.9/doxia-sink-api-1.9.jar
> [ERROR] urls[28] =
[jira] [Created] (KYLIN-4235) Failed to load table metadata from JDBC data source
Guangxu Cheng created KYLIN-4235: Summary: Failed to load table metadata from JDBC data source Key: KYLIN-4235 URL: https://issues.apache.org/jira/browse/KYLIN-4235 Project: Kylin Issue Type: Bug Affects Versions: v2.6.4, v3.0.0-alpha2 Reporter: Guangxu Cheng Assignee: Guangxu Cheng Attachments: image-2019-11-01-23-45-56-319.png After adding the following configurations to the project, the table metadata still cannot be loaded !image-2019-11-01-23-45-56-319.png|width=480,height=373! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4145) compile failed due to incompatible version between scala and scala-maven-plugin
Guangxu Cheng created KYLIN-4145:
Summary: compile failed due to incompatible version between scala
and scala-maven-plugin
Key: KYLIN-4145
URL: https://issues.apache.org/jira/browse/KYLIN-4145
Project: Kylin
Issue Type: Bug
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
The version of scala-maven-plugin is not specified, and the latest
version(4.1.1) will be used when compiling the project, which is incompatible
with the version of scala(2.11.0).
So, it would be better to specify the version number of scala-maven-plugin.
{code:xml}
[ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:4.1.1:compile (scala-compile-first) on
project kylin-engine-spark: Execution scala-compile-first of goal
net.alchim31.maven:scala-maven-plugin:4.1.1:compile failed: An API
incompatibility was encountered while executing
net.alchim31.maven:scala-maven-plugin:4.1.1:compile:
java.lang.NoSuchMethodError:
xsbt.ZincCompiler.classPath()Lscala/tools/nsc/util/ClassFileLookup;
[ERROR] -
[ERROR] realm =plugin>net.alchim31.maven:scala-maven-plugin:4.1.1
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] =
file:/Users/cheng/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.1.1/scala-maven-plugin-4.1.1.jar
[ERROR] urls[1] =
file:/Users/cheng/.m2/repository/org/apache/maven/maven-builder-support/3.3.9/maven-builder-support-3.3.9.jar
[ERROR] urls[2] =
file:/Users/cheng/.m2/repository/com/google/guava/guava/18.0/guava-18.0.jar
[ERROR] urls[3] =
file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.21/plexus-interpolation-1.21.jar
[ERROR] urls[4] =
file:/Users/cheng/.m2/repository/javax/enterprise/cdi-api/1.0/cdi-api-1.0.jar
[ERROR] urls[5] =
file:/Users/cheng/.m2/repository/org/eclipse/sisu/org.eclipse.sisu.inject/0.3.2/org.eclipse.sisu.inject-0.3.2.jar
[ERROR] urls[6] =
file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-component-annotations/1.6/plexus-component-annotations-1.6.jar
[ERROR] urls[7] =
file:/Users/cheng/.m2/repository/org/apache/maven/reporting/maven-reporting-api/3.0/maven-reporting-api-3.0.jar
[ERROR] urls[8] =
file:/Users/cheng/.m2/repository/org/eclipse/aether/aether-util/1.0.2.v20150114/aether-util-1.0.2.v20150114.jar
[ERROR] urls[9] =
file:/Users/cheng/.m2/repository/com/google/inject/guice/4.0/guice-4.0-no_aop.jar
[ERROR] urls[10] =
file:/Users/cheng/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar
[ERROR] urls[11] =
file:/Users/cheng/.m2/repository/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.jar
[ERROR] urls[12] =
file:/Users/cheng/.m2/repository/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.jar
[ERROR] urls[13] =
file:/Users/cheng/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
[ERROR] urls[14] =
file:/Users/cheng/.m2/repository/org/apache/maven/shared/maven-dependency-tree/3.0.1/maven-dependency-tree-3.0.1.jar
[ERROR] urls[15] =
file:/Users/cheng/.m2/repository/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar
[ERROR] urls[16] =
file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-utils/3.2.1/plexus-utils-3.2.1.jar
[ERROR] urls[17] =
file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-archiver/4.1.0/plexus-archiver-4.1.0.jar
[ERROR] urls[18] =
file:/Users/cheng/.m2/repository/org/codehaus/plexus/plexus-io/3.1.1/plexus-io-3.1.1.jar
[ERROR] urls[19] =
file:/Users/cheng/.m2/repository/org/apache/commons/commons-compress/1.18/commons-compress-1.18.jar
[ERROR] urls[20] =
file:/Users/cheng/.m2/repository/org/iq80/snappy/snappy/0.4/snappy-0.4.jar
[ERROR] urls[21] =
file:/Users/cheng/.m2/repository/org/tukaani/xz/1.8/xz-1.8.jar
[ERROR] urls[22] =
file:/Users/cheng/.m2/repository/backport-util-concurrent/backport-util-concurrent/3.1/backport-util-concurrent-3.1.jar
[ERROR] urls[23] =
file:/Users/cheng/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar
[ERROR] urls[24] =
file:/Users/cheng/.m2/repository/org/apache/maven/maven-archiver/3.4.0/maven-archiver-3.4.0.jar
[ERROR] urls[25] =
file:/Users/cheng/.m2/repository/org/apache/maven/shared/maven-shared-utils/3.2.1/maven-shared-utils-3.2.1.jar
[ERROR] urls[26] =
file:/Users/cheng/.m2/repository/commons-io/commons-io/2.5/commons-io-2.5.jar
[ERROR] urls[27] =
file:/Users/cheng/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.9/doxia-sink-api-1.9.jar
[ERROR] urls[28] =
file:/Users/cheng/.m2/repository/org/apache/maven/doxia/doxia-logging-api/1.9/doxia-logging-api-1.9.jar
[ERROR] urls[29] =
file:/Users/cheng/.m2/repository/org/apache/maven/shared/maven-invoker/3.0.1/maven-invoker-3.0.1.jar
[ERROR] urls[30] =
file:/Users/cheng/.m2/repository/org/scala-sbt/zinc_2.12/1.3.0-M8/zinc_2.12-1.3.0-M8.jar
[ERROR] urls[31] =
[jira] [Updated] (KYLIN-3936) MR/Spark task will still run after the job is stopped.
[ https://issues.apache.org/jira/browse/KYLIN-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-3936: - Description: The command "pause" only sets status of the job to "stopped" and does not reset the status of the subtask. So, In SparkExecutable, we can't get the real status of the running task. > MR/Spark task will still run after the job is stopped. > -- > > Key: KYLIN-3936 > URL: https://issues.apache.org/jira/browse/KYLIN-3936 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > > The command "pause" only sets status of the job to "stopped" and does not > reset the status of the subtask. > So, In SparkExecutable, we can't get the real status of the running task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3936) MR/Spark task will still run after the job is stopped.
Guangxu Cheng created KYLIN-3936: Summary: MR/Spark task will still run after the job is stopped. Key: KYLIN-3936 URL: https://issues.apache.org/jira/browse/KYLIN-3936 Project: Kylin Issue Type: Bug Components: Job Engine Reporter: Guangxu Cheng Assignee: Guangxu Cheng -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3895) Failed to register new MBean when "kylin.server.query-metrics-enabled" set true
Guangxu Cheng created KYLIN-3895:
Summary: Failed to register new MBean when
"kylin.server.query-metrics-enabled" set true
Key: KYLIN-3895
URL: https://issues.apache.org/jira/browse/KYLIN-3895
Project: Kylin
Issue Type: Bug
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
{code}
2019-03-20 10:17:25,753 WARN [Query 46cd99cc-8eb2-8370-d24c-6c10f18da9e0-54]
util.MBeans:94 : Error creating MBean object name: Ha
doop:service=Kylin,name=KYLIN_SYSTEM,sub=CUBE[name=KYLIN_HIVE_METRICS_JOB_QA]
org.apache.hadoop.metrics2.MetricsException:
javax.management.MalformedObjectNameException: Invalid character '=' in value
part of property
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:122)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:102)
at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:92)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55)
{code}
The subname can't contain '='
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3874) "Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS clusters
[
https://issues.apache.org/jira/browse/KYLIN-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795550#comment-16795550
]
Guangxu Cheng commented on KYLIN-3874:
--
bq.If you want to use different HDFS clusters, you should modify your
hdfs-site.xml .
Kylin supports set HBase's HDFS configuration information in a separate file
which specified by parameter "kylin.storage.hbase.cluster-hdfs-config-file". In
this case, the exception mentioned in this issue will occur. Thanks
> "Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS
> clusters
> ---
>
> Key: KYLIN-3874
> URL: https://issues.apache.org/jira/browse/KYLIN-3874
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v2.6.2
>
>
> {panel:title=exception}
> 2019-03-14 16:33:45,113 INFO [main]
> org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause:
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
> Couldn't create proxy provider class
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> ...
> Caused by: java.lang.RuntimeException: Could not find any configured
> addresses for URI
> hdfs://{color:#ff}hbase-ns-test{color}/kylin/kylin_metadata/kylin-cd4db32e-42b6-44f2-7fa9-ef6c46334249/derived_meta_global/hfile
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.(ConfiguredFailoverProxyProvider.java:116)
> ... 31 more
> {panel}
> {color:#ff}hbase-ns-test{color} is the nameservice of HDFS which is used
> by HBase.
> In our kylin environment, HBase and MR/HIVE use different HDFS clusters.
> {code:java|title=CubeHFileJob.java|borderStyle=solid}
> // use current hbase configuration
> Configuration configuration = new
> Configuration(HBaseConnection.getCurrentHBaseConfiguration());
> merge(configuration, getConf());
> {code}
> "getConf()" contains only the HDFS configuration of MR, and "configuration"
> contains the HDFS configuration both HBase and MR. See:
> [HBaseConnection#addHBaseClusterNNHAConfiguration()|https://github.com/apache/kylin/blob/d23d2bddab3263d39059492b91f71d8091198057/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseConnection.java#L205]
> So "merge(configuration, getConf())" will lead to HBASE's HDFS configuration
> to be covered. Mainly parameter "dfs.nameservices"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3888) TableNotDisabledException when running "Convert Lookup Table to HFile"
Guangxu Cheng created KYLIN-3888:
Summary: TableNotDisabledException when running "Convert Lookup
Table to HFile"
Key: KYLIN-3888
URL: https://issues.apache.org/jira/browse/KYLIN-3888
Project: Kylin
Issue Type: Bug
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
{code}
org.apache.hadoop.hbase.TableNotDisabledException:
KYLIN_LOOKUP_DEFAULT.F_ADVERTISER_DIM_KYLIN_TEST_9WCVUDIXI5
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.convertResult(HBaseAdmin.java:4756)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4714)
at
org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4647)
at
org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:904)
at
org.apache.kylin.storage.hbase.lookup.LookupTableToHFileJob.removeSnapshotIfExist(LookupTableToHFileJob.java:170)
at
org.apache.kylin.storage.hbase.lookup.LookupTableToHFileJob.run(LookupTableToHFileJob.java:104)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:131)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotDisabledException):
KYLIN_LOOKUP_DEFAULT.F_ADVERTISER_DIM_KYLIN_TEST_9WCVUDIXI5
at
org.apache.hadoop.hbase.master.HMaster.checkTableModifiable(HMaster.java:2266)
at
org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.prepareDelete(DeleteTableProcedure.java:253)
at
org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:102)
at
org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:59)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:498)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1147)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:942)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:895)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:77)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:497)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3874) "Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS clusters
Guangxu Cheng created KYLIN-3874:
Summary: "Convert Cuboid Data to HFile" failed when HBase and MR
use different HDFS clusters
Key: KYLIN-3874
URL: https://issues.apache.org/jira/browse/KYLIN-3874
Project: Kylin
Issue Type: Bug
Components: Job Engine
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng
{panel:title=exception}
2019-03-14 16:33:45,113 INFO [main] org.apache.hadoop.service.AbstractService:
Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED;
cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.io.IOException: Couldn't create proxy provider class
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
...
Caused by: java.lang.RuntimeException: Could not find any configured addresses
for URI
hdfs://{color:#ff}hbase-ns-test{color}/kylin/kylin_metadata/kylin-cd4db32e-42b6-44f2-7fa9-ef6c46334249/derived_meta_global/hfile
at
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.(ConfiguredFailoverProxyProvider.java:116)
... 31 more
{panel}
{color:#ff}hbase-ns-test{color} is the nameservice of HDFS which is used by
HBase.
In our kylin environment, HBase and MR/HIVE use different HDFS clusters.
{code:java|title=CubeHFileJob.java|borderStyle=solid}
// use current hbase configuration
Configuration configuration = new
Configuration(HBaseConnection.getCurrentHBaseConfiguration());
merge(configuration, getConf());
{code}
"getConf()" contains only the HDFS configuration of MR, and "configuration"
contains the HDFS configuration both HBase and MR. See:
[HBaseConnection#addHBaseClusterNNHAConfiguration()|https://github.com/apache/kylin/blob/d23d2bddab3263d39059492b91f71d8091198057/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseConnection.java#L205]
So "merge(configuration, getConf())" will lead to HBASE's HDFS configuration to
be covered. Mainly parameter "dfs.nameservices"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792399#comment-16792399 ] Guangxu Cheng commented on KYLIN-3866: -- Thanks for reviewing and committing, [~Shaofengshi] :) > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: v2.6.2 > > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791612#comment-16791612 ] Guangxu Cheng commented on KYLIN-3866: -- Add parameter "kylin.engine.mr.use-local-classpath" and set the default value to true, mainly to keep the compatibility of versions. For old users, upgrading kylin to a higher version has no effect on them. For new users, this feature can be turned off with this parameter if it is not needed. If the changes can be listed in the upgrade document to remind the old user.I personally think it is okay to remove this logic. WDYT? Thanks > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[
https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791409#comment-16791409
]
Guangxu Cheng commented on KYLIN-3866:
--
{quote}While, I still not understand the requirment: in which case the
"mapreduce.application.classpath" is absent, and still not allow to use local
cmd to get the classpath? Usually Kylin is installed in a Hadoop client node,
which need the MR/Hive/HBase be well configured, the classpath configuration
should be correct for all Hadoop node. Could you please elaborate the problem
you encountered, and how common it is? Thank you!
{quote}
[~Shaofengshi] Thanks for your review.
1. "mapreduce.application.classpath" is not a required parameter. If the
parameter is not configured, the default value will be used. see:
[https://github.com/apache/hadoop/blob/5d8c8cd568ae54c1be366e786c738b63bd53e271/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java#L219]
In our cluster, we use the default value and do not need the user to configure
the parameter on the client side.
2. We are not deploying kylin on the nodes of the MR cluster because the MR
cluster is maintained by other departments within Tencent and we do not have
permission to deploy other applications on the nodes of the MR cluster. We also
don't know the installation path of the MR cluster.
In Tencent, Kylin is deployed separately, Hadoop/HBase/Hive client is deployed
on the kylin node, but the path of deploying Hadoop client may not be the same
as the MR cluster. So, the path obtained by the command "mapred classpath" is
only the path of Hadoop client not the path of the MR cluster deployment.
Now, kylin forces the user to configure this parameter. If it is not
configured, the path obtained by using the command "mapred classpath" (which
may be wrong value) will be assigned to the parameter, which I think it is
unreasonable.
So, it may be more reasonable for the user to decide whether to use the
parameter.
> Whether to set mapreduce.application.classpath is determined by the user
>
>
> Key: KYLIN-3866
> URL: https://issues.apache.org/jira/browse/KYLIN-3866
> Project: Kylin
> Issue Type: Bug
>Reporter: Guangxu Cheng
>Priority: Major
> Attachments: KYLIN-3866.master.001.patch
>
>
> In the kylin environment, if there is no configuration parameter
> "mapreduce.application.classpath", then kylin will use the command "mapred
> classpath" to get the local classpath and set it to
> "mapreduce.application.classpath", However, it will cause the following two
> problems:
> 1. The local classpath may not be the same as the MapReduce cluster.
> 2. There is no need to configure the parameter
> "mapreduce.application.classpath" on the client side in some cluster.
> So, it may be a better choice for users to decide whether they need to
> configure this parameter or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789357#comment-16789357 ] Guangxu Cheng commented on KYLIN-3866: -- [~Shaofengshi] mind taking a look at it ? thanks > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Priority: Major > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng updated KYLIN-3866: - Attachment: KYLIN-3866.master.001.patch > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Priority: Major > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
Guangxu Cheng created KYLIN-3866: Summary: Whether to set mapreduce.application.classpath is determined by the user Key: KYLIN-3866 URL: https://issues.apache.org/jira/browse/KYLIN-3866 Project: Kylin Issue Type: Bug Reporter: Guangxu Cheng In the kylin environment, if there is no configuration parameter "mapreduce.application.classpath", then kylin will use the command "mapred classpath" to get the local classpath and set it to "mapreduce.application.classpath", However, it will cause the following two problems: 1. The local classpath may not be the same as the MapReduce cluster. 2. There is no need to configure the parameter "mapreduce.application.classpath" on the client side in some cluster. So, it may be a better choice for users to decide whether they need to configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
