[jira] [Created] (KYLIN-4175) Support secondary hbase storage config for hbase cluster migration
Liu Shaohui created KYLIN-4175: -- Summary: Support secondary hbase storage config for hbase cluster migration Key: KYLIN-4175 URL: https://issues.apache.org/jira/browse/KYLIN-4175 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Currently when migrating a kylin cluster from on data center to the other or the hbase cluster the kylin depends on will be changed from one cluster to the other, there will be a long down time to migrating the history data from one cluster to the other or we must rebuild all the history data of cube in the other cluster. In xiaomi, we added the support of secondary hbase storage and made the kylin cluster can query cube data from the old hbase cluster during the migration. As a result, the migration is very smooth with minimum down time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4100) Add overall job number statistics in monitor page
[ https://issues.apache.org/jira/browse/KYLIN-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931201#comment-16931201 ] Liu Shaohui commented on KYLIN-4100: You can see the demo image in the attachments. > Add overall job number statistics in monitor page > - > > Key: KYLIN-4100 > URL: https://issues.apache.org/jira/browse/KYLIN-4100 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Attachments: x.png > > > Currently it's hard to get pending and running job number in mointor page, we > can only continue to click more until the end. > It's better to have an overall job number statistics in monitor page. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KYLIN-4100) Add overall job number statistics in monitor page
[ https://issues.apache.org/jira/browse/KYLIN-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-4100: --- Attachment: x.png > Add overall job number statistics in monitor page > - > > Key: KYLIN-4100 > URL: https://issues.apache.org/jira/browse/KYLIN-4100 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Attachments: x.png > > > Currently it's hard to get pending and running job number in mointor page, we > can only continue to click more until the end. > It's better to have an overall job number statistics in monitor page. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KYLIN-4112) Add hdfs keberos token delegation in Spark to support HBase and MR use different HDFS clusters
Liu Shaohui created KYLIN-4112: -- Summary: Add hdfs keberos token delegation in Spark to support HBase and MR use different HDFS clusters Key: KYLIN-4112 URL: https://issues.apache.org/jira/browse/KYLIN-4112 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Currently the SparkExecutable only delegate the token for yarn hdfs cluster, not for the hdfs cluster used by the HBase cluster. The spark job of Convert Cuboid Data to HFile will failed for kerberos issue. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4111) drop table failed with no valid privileges after KYLIN-3857
Liu Shaohui created KYLIN-4111:
--
Summary: drop table failed with no valid privileges after
KYLIN-3857
Key: KYLIN-4111
URL: https://issues.apache.org/jira/browse/KYLIN-4111
Project: Kylin
Issue Type: Bug
Reporter: Liu Shaohui
Assignee: Liu Shaohui
After KYLIN-3857, there will be quote ` around database and table.
The drop table sql will be:
{code:java}
DROP TABLE IF EXISTS
`kylin_onebox.kylin_intermediate_kylin_sales_cube_7be84be1_a153_07c4_3ce6_270e8d99ff85`;{code}
Hive (1.2)with sentry will throw exception:
{code:java}
Error: Error while compiling statement: FAILED: HiveAccessControlException No
valid privileges
Required privileges for this query:
Server=server1->Db=`kylin_onebox->Table=kylin_intermediate_kylin_sales_cube_7be84be1_a153_07c4_3ce6_270e8d99ff85`->action=drop;
Query log:
http://zjy-hadoop-prc-ct14.bj:18201/log?qid=898c7878-a961-443d-b120-cca0e2667d15_f486bd16-4bbd-4014-a0a7-c2ebfdbe6668
(state=42000,code=4)
{code}
The reason is that hive identify the databse be `kylin_onebox and table be:
kylin_intermediate_kylin_sales_cube_7be84be1_a153_07c4_3ce6_270e8d99ff85`
May be we can fix it in hive and sentry. Just create a jira to show this
problem.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (KYLIN-4109) CubeHFileMapperTest failed after commit: f4d2405f6aa978bbc3153c9ca9fa339b9d7e6c30
Liu Shaohui created KYLIN-4109:
--
Summary: CubeHFileMapperTest failed after commit:
f4d2405f6aa978bbc3153c9ca9fa339b9d7e6c30
Key: KYLIN-4109
URL: https://issues.apache.org/jira/browse/KYLIN-4109
Project: Kylin
Issue Type: Bug
Reporter: Liu Shaohui
CubeHFileMapperTest failed in 2.5.x-hadoop3.1 for following changes
{code:java}
- assertEquals("cf1", new String(p1.getSecond().getFamily(),
StandardCharsets.UTF_8));
- assertEquals("usd_amt", new String(p1.getSecond().getQualifier(),
StandardCharsets.UTF_8));
- assertEquals("35.43", new String(p1.getSecond().getValue(),
StandardCharsets.UTF_8));
+ assertEquals("cf1", new String(copy(p1.getSecond(;
+ assertEquals("usd_amt", new String(copy(p1.getSecond(;
+ assertEquals("35.43", new String(copy(p1.getSecond(;
assertEquals(key, p2.getFirst());
- assertEquals("cf1", new String(p2.getSecond().getFamily(),
StandardCharsets.UTF_8));
- assertEquals("item_count", new String(p2.getSecond().getQualifier(),
StandardCharsets.UTF_8));
- assertEquals("2", new String(p2.getSecond().getValue(),
StandardCharsets.UTF_8));
+ assertEquals("cf1", new String(copy(p2.getSecond(;
+ assertEquals("item_count", new String(copy(p2.getSecond(;
+ assertEquals("2", new String(copy(p2.getSecond(;
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (KYLIN-4103) Make the user string in granting operation of project is case insensitive
Liu Shaohui created KYLIN-4103: -- Summary: Make the user string in granting operation of project is case insensitive Key: KYLIN-4103 URL: https://issues.apache.org/jira/browse/KYLIN-4103 Project: Kylin Issue Type: Bug Reporter: Liu Shaohui Assignee: Liu Shaohui Currently the user name of login operation is case insensitive. User can login in kylin with lower case string or upper case string. But it is not granting operation. If we use lower case string of user name in project granting operation, there will be no exception but the user can not set the project. The reason is that the sid in AccessService/AclService is not case insensitive -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4101) set hive and spark job name when building cube
Liu Shaohui created KYLIN-4101:
--
Summary: set hive and spark job name when building cube
Key: KYLIN-4101
URL: https://issues.apache.org/jira/browse/KYLIN-4101
Project: Kylin
Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Currently the job name of spark is
{color:#22}org.apache.kylin.common.util.SparkEntry{color}, which is the
main class name of spark . The mapreduce job name of hive sql is substring of
the query, which is difficult to read.
It's better to set a more readable name for the hive and spark jobs
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (KYLIN-4100) Add overall job number statistics in monitor page
Liu Shaohui created KYLIN-4100: -- Summary: Add overall job number statistics in monitor page Key: KYLIN-4100 URL: https://issues.apache.org/jira/browse/KYLIN-4100 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Currently it's hard to get pending and running job number in mointor page, we can only continue to click more until the end. It's better to have an overall job number statistics in monitor page. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KYLIN-4099) Using no blocking RDD unpersist in spark cubing job
[
https://issues.apache.org/jira/browse/KYLIN-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-4099:
---
Summary: Using no blocking RDD unpersist in spark cubing job (was: Using
no blocking unpersist in spark cubing job )
> Using no blocking RDD unpersist in spark cubing job
>
>
> Key: KYLIN-4099
> URL: https://issues.apache.org/jira/browse/KYLIN-4099
> Project: Kylin
> Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
>
> By default, the unpersist operation of RDD in spark is blocking which may
> cost a lot time and
> some times it may failed for some spark executors lost.
> We can set blocking false to improve it.
> {code:java}
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:208)
> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:218)
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> scala.concurrent.Await$.result(package.scala:190)
> org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:81)
> org.apache.spark.storage.BlockManagerMaster.removeRdd(BlockManagerMaster.scala:127)
> org.apache.spark.SparkContext.unpersistRDD(SparkContext.scala:1709)
> org.apache.spark.rdd.RDD.unpersist(RDD.scala:216)
> org.apache.spark.api.java.JavaPairRDD.unpersist(JavaPairRDD.scala:73)
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:204)
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:653){code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (KYLIN-4099) Using no blocking unpersist in spark cubing job
Liu Shaohui created KYLIN-4099:
--
Summary: Using no blocking unpersist in spark cubing job
Key: KYLIN-4099
URL: https://issues.apache.org/jira/browse/KYLIN-4099
Project: Kylin
Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
By default, the unpersist operation of RDD in spark is blocking which may cost
a lot time and
some times it may failed for some spark executors lost.
We can set blocking false to improve it.
{code:java}
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:208)
scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:218)
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
scala.concurrent.Await$.result(package.scala:190)
org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:81)
org.apache.spark.storage.BlockManagerMaster.removeRdd(BlockManagerMaster.scala:127)
org.apache.spark.SparkContext.unpersistRDD(SparkContext.scala:1709)
org.apache.spark.rdd.RDD.unpersist(RDD.scala:216)
org.apache.spark.api.java.JavaPairRDD.unpersist(JavaPairRDD.scala:73)
org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:204)
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:653){code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (KYLIN-4098) Add cube auto merge api
Liu Shaohui created KYLIN-4098: -- Summary: Add cube auto merge api Key: KYLIN-4098 URL: https://issues.apache.org/jira/browse/KYLIN-4098 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Currently the auto merging of cube is triggered by the event of new segment is ready automatically. When the cluster restart, there may be too many merging job. It's better to have a rest api to trigger the merging and make it more controllable. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KYLIN-4098) Add cube auto merge api
[ https://issues.apache.org/jira/browse/KYLIN-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-4098: --- Priority: Minor (was: Major) > Add cube auto merge api > --- > > Key: KYLIN-4098 > URL: https://issues.apache.org/jira/browse/KYLIN-4098 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > > Currently the auto merging of cube is triggered by the event of new segment > is ready automatically. When the cluster restart, there may be too many > merging job. > It's better to have a rest api to trigger the merging and make it more > controllable. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4097) Throw exception when too many dict slice eviction in AppendTrieDictionary
Liu Shaohui created KYLIN-4097: -- Summary: Throw exception when too many dict slice eviction in AppendTrieDictionary Key: KYLIN-4097 URL: https://issues.apache.org/jira/browse/KYLIN-4097 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui When the global dict is too large than the spark executor memory, there will be too many dict slice evictions and loads in AppendTrieDictionary, and the build job will be very slow. It's better to throw an exception in advance in this case. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4096) Make cube metadata validator rules configuable
Liu Shaohui created KYLIN-4096: -- Summary: Make cube metadata validator rules configuable Key: KYLIN-4096 URL: https://issues.apache.org/jira/browse/KYLIN-4096 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui CubeMetadataValidator is very useful to format the cube creation. In xiaomi, we implements multi rules to reduce the operation cost. eg: ConfOverrideRule which make user set computing queue in cube configuration and forbid to set some configurations like: kylin.query.max-scan-bytes So it's better to make the rules configuable -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4095) Add RESOURCE_PATH_PREFIX option in ResourceTool
Liu Shaohui created KYLIN-4095: -- Summary: Add RESOURCE_PATH_PREFIX option in ResourceTool Key: KYLIN-4095 URL: https://issues.apache.org/jira/browse/KYLIN-4095 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui ResourceTool is very useful to fix the metadata with overlap segments. But downloading and uploading entire metadata is too heavy. It's better to have a RESOURCE_PATH_PREFIX option for downloading and uploading cmds. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Closed] (KYLIN-4094) Add script to create system tables and cubes automatically
[ https://issues.apache.org/jira/browse/KYLIN-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui closed KYLIN-4094. -- Resolution: Duplicate Assignee: (was: Liu Shaohui) Duplicated with KYLIN-3570 > Add script to create system tables and cubes automatically > -- > > Key: KYLIN-4094 > URL: https://issues.apache.org/jira/browse/KYLIN-4094 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Minor > > See: [http://kylin.apache.org/docs/tutorial/setup_systemcube.html] > It's a little complex to setup the the system cubes. We can add a scripts to > make it easier. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4094) Add script to create system tables and cubes automatically
Liu Shaohui created KYLIN-4094: -- Summary: Add script to create system tables and cubes automatically Key: KYLIN-4094 URL: https://issues.apache.org/jira/browse/KYLIN-4094 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui See: [http://kylin.apache.org/docs/tutorial/setup_systemcube.html] It's a little complex to setup the the system cubes. We can add a scripts to make it easier. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KYLIN-4094) Add script to create system tables and cubes automatically
[ https://issues.apache.org/jira/browse/KYLIN-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-4094: --- Priority: Minor (was: Critical) > Add script to create system tables and cubes automatically > -- > > Key: KYLIN-4094 > URL: https://issues.apache.org/jira/browse/KYLIN-4094 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > > See: [http://kylin.apache.org/docs/tutorial/setup_systemcube.html] > It's a little complex to setup the the system cubes. We can add a scripts to > make it easier. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KYLIN-4094) Add script to create system tables and cubes automatically
[ https://issues.apache.org/jira/browse/KYLIN-4094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-4094: --- Priority: Critical (was: Major) > Add script to create system tables and cubes automatically > -- > > Key: KYLIN-4094 > URL: https://issues.apache.org/jira/browse/KYLIN-4094 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Critical > > See: [http://kylin.apache.org/docs/tutorial/setup_systemcube.html] > It's a little complex to setup the the system cubes. We can add a scripts to > make it easier. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KYLIN-3913) Remove getAllOutputs api in ExecutableManager to avoid OOM for large metadata
[ https://issues.apache.org/jira/browse/KYLIN-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-3913: --- Priority: Minor (was: Major) > Remove getAllOutputs api in ExecutableManager to avoid OOM for large metadata > - > > Key: KYLIN-3913 > URL: https://issues.apache.org/jira/browse/KYLIN-3913 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: PENG Zhengshuai >Priority: Minor > > In a big cluster, there will be many job info left in the metadata. The kylin > server will be OOM when search the jobs with a long time range. The reason is > that ExecutableManager will load all job output info into memory when search > a job. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KYLIN-3913) Remove getAllOutputs api in ExecutableManager to avoid OOM for large metadata
[ https://issues.apache.org/jira/browse/KYLIN-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-3913: --- Affects Version/s: v2.5.2 > Remove getAllOutputs api in ExecutableManager to avoid OOM for large metadata > - > > Key: KYLIN-3913 > URL: https://issues.apache.org/jira/browse/KYLIN-3913 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: PENG Zhengshuai >Priority: Minor > > In a big cluster, there will be many job info left in the metadata. The kylin > server will be OOM when search the jobs with a long time range. The reason is > that ExecutableManager will load all job output info into memory when search > a job. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4093) Slow query pages should be open to all users of the project
Liu Shaohui created KYLIN-4093: -- Summary: Slow query pages should be open to all users of the project Key: KYLIN-4093 URL: https://issues.apache.org/jira/browse/KYLIN-4093 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Currently the show query page only can been seen for kylin admins. It's very useful for modlers and analysts for this project. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4092) Support setting seperate jvm params for kylin backgroud tools
Liu Shaohui created KYLIN-4092: -- Summary: Support setting seperate jvm params for kylin backgroud tools Key: KYLIN-4092 URL: https://issues.apache.org/jira/browse/KYLIN-4092 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Usually, the memory set in setenv.sh for query server is larger then 8G, which is not suitable for kylin background tools (meta cleaup, storage cleanup, health check) So It's better to have a seperate env for kylin tools -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4048) Too long spark cube building time for too many eviction and loading for dict slices
Liu Shaohui created KYLIN-4048:
--
Summary: Too long spark cube building time for too many eviction
and loading for dict slices
Key: KYLIN-4048
URL: https://issues.apache.org/jira/browse/KYLIN-4048
Project: Kylin
Issue Type: Improvement
Affects Versions: v2.5.2
Reporter: Liu Shaohui
In our cluster, a cube building costs too long. In the log for spark, we found
there are too many eviction and loading for dict slices in AppendTrieDictionary.
{code:java}
$ grep "read slice from" spark.log | wc -l
119721
$ grep "Evict slice with key" spark.log| wc -l
119634
{code}
The reason is that the memory of spark executor(4G) is not enough to hold all
the slices of dict(3.3G in hdfs) in memory, which cause the bad performance of
cube building.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-4029) Overwriting conflict when create a new data model
[
https://issues.apache.org/jira/browse/KYLIN-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui reassigned KYLIN-4029:
--
Assignee: Liu Shaohui
> Overwriting conflict when create a new data model
> --
>
> Key: KYLIN-4029
> URL: https://issues.apache.org/jira/browse/KYLIN-4029
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.2
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
>
> A creating model request failed for Overwriting conflicts when saving project
> metadata.
> It left a stale state in meta and user can not delete it or creating a one
> with same name.
> {code:java}
> 2019-05-31 16:35:11,668 ERROR [http-bio-7070-exec-57]
> controller.BasicController:63 :
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /project/BigBI_Hive.json, expect old TS 1559291698212, but it is
> 1559291711327
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:326)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:327)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:309)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:288)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
> at
> org.apache.kylin.metadata.project.ProjectManager.save(ProjectManager.java:373)
> at
> org.apache.kylin.metadata.project.ProjectManager.addModelToProject(ProjectManager.java:251)
> at
> org.apache.kylin.metadata.model.DataModelManager.createDataModelDesc(DataModelManager.java:256)
> at
> org.apache.kylin.rest.service.ModelService.createModelDesc(ModelService.java:148)
> at
> org.apache.kylin.rest.controller.ModelController.saveModelDesc(ModelController.java:128){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-4032) Add tools to show kylin instance which schedule the running job
[ https://issues.apache.org/jira/browse/KYLIN-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16855657#comment-16855657 ] Liu Shaohui commented on KYLIN-4032: I think this is same as https://issues.apache.org/jira/browse/KYLIN-3780 which committed to 2.6.1 > Add tools to show kylin instance which schedule the running job > --- > > Key: KYLIN-4032 > URL: https://issues.apache.org/jira/browse/KYLIN-4032 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Priority: Minor > > Hi team: > Sometime, the operator need to know the running/error job owner to trace > the log file among the kylin cluster. A simple tool to show this may be > helpful. > > > Best regards > > yuzhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-4029) Overwriting conflict when create a new data model
[
https://issues.apache.org/jira/browse/KYLIN-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-4029:
---
Issue Type: Bug (was: Improvement)
> Overwriting conflict when create a new data model
> --
>
> Key: KYLIN-4029
> URL: https://issues.apache.org/jira/browse/KYLIN-4029
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Priority: Minor
>
> A creating model request failed for Overwriting conflicts when saving project
> metadata.
> It left a stale state in meta and user can not delete it or creating a one
> with same name.
> {code:java}
> 2019-05-31 16:35:11,668 ERROR [http-bio-7070-exec-57]
> controller.BasicController:63 :
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /project/BigBI_Hive.json, expect old TS 1559291698212, but it is
> 1559291711327
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:326)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:327)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:309)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:288)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
> at
> org.apache.kylin.metadata.project.ProjectManager.save(ProjectManager.java:373)
> at
> org.apache.kylin.metadata.project.ProjectManager.addModelToProject(ProjectManager.java:251)
> at
> org.apache.kylin.metadata.model.DataModelManager.createDataModelDesc(DataModelManager.java:256)
> at
> org.apache.kylin.rest.service.ModelService.createModelDesc(ModelService.java:148)
> at
> org.apache.kylin.rest.controller.ModelController.saveModelDesc(ModelController.java:128){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-4029) Overwriting conflict when create a new data model
[
https://issues.apache.org/jira/browse/KYLIN-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-4029:
---
Affects Version/s: v2.5.2
> Overwriting conflict when create a new data model
> --
>
> Key: KYLIN-4029
> URL: https://issues.apache.org/jira/browse/KYLIN-4029
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.5.2
>Reporter: Liu Shaohui
>Priority: Minor
>
> A creating model request failed for Overwriting conflicts when saving project
> metadata.
> It left a stale state in meta and user can not delete it or creating a one
> with same name.
> {code:java}
> 2019-05-31 16:35:11,668 ERROR [http-bio-7070-exec-57]
> controller.BasicController:63 :
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /project/BigBI_Hive.json, expect old TS 1559291698212, but it is
> 1559291711327
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:326)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:327)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:309)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:288)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
> at
> org.apache.kylin.metadata.project.ProjectManager.save(ProjectManager.java:373)
> at
> org.apache.kylin.metadata.project.ProjectManager.addModelToProject(ProjectManager.java:251)
> at
> org.apache.kylin.metadata.model.DataModelManager.createDataModelDesc(DataModelManager.java:256)
> at
> org.apache.kylin.rest.service.ModelService.createModelDesc(ModelService.java:148)
> at
> org.apache.kylin.rest.controller.ModelController.saveModelDesc(ModelController.java:128){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-4029) Overwriting conflict when create a new data model
[
https://issues.apache.org/jira/browse/KYLIN-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-4029:
---
Component/s: Metadata
> Overwriting conflict when create a new data model
> --
>
> Key: KYLIN-4029
> URL: https://issues.apache.org/jira/browse/KYLIN-4029
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.2
>Reporter: Liu Shaohui
>Priority: Minor
>
> A creating model request failed for Overwriting conflicts when saving project
> metadata.
> It left a stale state in meta and user can not delete it or creating a one
> with same name.
> {code:java}
> 2019-05-31 16:35:11,668 ERROR [http-bio-7070-exec-57]
> controller.BasicController:63 :
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /project/BigBI_Hive.json, expect old TS 1559291698212, but it is
> 1559291711327
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:326)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:327)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:309)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:288)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
> at
> org.apache.kylin.metadata.project.ProjectManager.save(ProjectManager.java:373)
> at
> org.apache.kylin.metadata.project.ProjectManager.addModelToProject(ProjectManager.java:251)
> at
> org.apache.kylin.metadata.model.DataModelManager.createDataModelDesc(DataModelManager.java:256)
> at
> org.apache.kylin.rest.service.ModelService.createModelDesc(ModelService.java:148)
> at
> org.apache.kylin.rest.controller.ModelController.saveModelDesc(ModelController.java:128){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-4029) Overwriting conflict when create a new data model
Liu Shaohui created KYLIN-4029:
--
Summary: Overwriting conflict when create a new data model
Key: KYLIN-4029
URL: https://issues.apache.org/jira/browse/KYLIN-4029
Project: Kylin
Issue Type: Improvement
Reporter: Liu Shaohui
A creating model request failed for Overwriting conflicts when saving project
metadata.
It left a stale state in meta and user can not delete it or creating a one with
same name.
{code:java}
2019-05-31 16:35:11,668 ERROR [http-bio-7070-exec-57]
controller.BasicController:63 :
org.apache.kylin.common.persistence.WriteConflictException: Overwriting
conflict /project/BigBI_Hive.json, expect old TS 1559291698212, but it is
1559291711327
at
org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:326)
at
org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:327)
at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:309)
at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:288)
at
org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
at
org.apache.kylin.metadata.project.ProjectManager.save(ProjectManager.java:373)
at
org.apache.kylin.metadata.project.ProjectManager.addModelToProject(ProjectManager.java:251)
at
org.apache.kylin.metadata.model.DataModelManager.createDataModelDesc(DataModelManager.java:256)
at
org.apache.kylin.rest.service.ModelService.createModelDesc(ModelService.java:148)
at
org.apache.kylin.rest.controller.ModelController.saveModelDesc(ModelController.java:128){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-4026) Avoid too many file append operation in HiveProducer of hive metrics reporter
Liu Shaohui created KYLIN-4026: -- Summary: Avoid too many file append operation in HiveProducer of hive metrics reporter Key: KYLIN-4026 URL: https://issues.apache.org/jira/browse/KYLIN-4026 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Currently for each write in HiveProducer, there will be a hdfs append operation, which is heavy for HDFS. A improvement is to keep a FSDataOutputStream in HiveProducer and write data to it continuously -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-4026) Avoid too many file append operations in HiveProducer of hive metrics reporter
[ https://issues.apache.org/jira/browse/KYLIN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-4026: --- Summary: Avoid too many file append operations in HiveProducer of hive metrics reporter (was: Avoid too many file append operation in HiveProducer of hive metrics reporter) > Avoid too many file append operations in HiveProducer of hive metrics reporter > -- > > Key: KYLIN-4026 > URL: https://issues.apache.org/jira/browse/KYLIN-4026 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > Currently for each write in HiveProducer, there will be a hdfs append > operation, which is heavy for HDFS. > A improvement is to keep a FSDataOutputStream in HiveProducer and write data > to it continuously > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4025) Add detail exception in kylin http response
Liu Shaohui created KYLIN-4025: -- Summary: Add detail exception in kylin http response Key: KYLIN-4025 URL: https://issues.apache.org/jira/browse/KYLIN-4025 Project: Kylin Issue Type: New Feature Components: REST Service Affects Versions: v2.5.2 Reporter: Liu Shaohui Assignee: Liu Shaohui Currently there is no detailed in http response when the requests go wrong, because InternalErrorException in controller wrap the execption and remove the exception stack. It's better add the detail exception in kylin http response -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-4005) Saving Cube of a aggregation Groups(40 Dimensions, Max Dimension Combination:5) may cause kylin server OOM
[
https://issues.apache.org/jira/browse/KYLIN-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui reassigned KYLIN-4005:
--
Assignee: Liu Shaohui
> Saving Cube of a aggregation Groups(40 Dimensions, Max Dimension
> Combination:5) may cause kylin server OOM
> --
>
> Key: KYLIN-4005
> URL: https://issues.apache.org/jira/browse/KYLIN-4005
> Project: Kylin
> Issue Type: Bug
> Components: REST Service
>Affects Versions: v2.5.2
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Critical
> Fix For: Future
>
>
> A user try to save a cube with a aggregation Groups(40 Dimensions, Max
> Dimension Combination:5) caused the kylin server OOM. The reason is that the
> DefaultCuboidScheduler will cost a lot memory when calculating all cube ids.
> The stack is following
> {code}
> http-bio-7070-exec-35
> at java.lang.OutOfMemoryError.()V (OutOfMemoryError.java:48)
> at java.util.HashMap.resize()[Ljava/util/HashMap$Node; (HashMap.java:704)
> at
> java.util.HashMap.putVal(ILjava/lang/Object;Ljava/lang/Object;ZZ)Ljava/lang/Object;
> (HashMap.java:663)
> at
> java.util.HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
> (HashMap.java:612)
> at java.util.HashSet.add(Ljava/lang/Object;)Z (HashSet.java:220)
> at java.util.AbstractCollection.addAll(Ljava/util/Collection;)Z
> (AbstractCollection.java:344)
> at
> org.apache.kylin.cube.cuboid.DefaultCuboidScheduler.getOnTreeParentsByLayer(Ljava/util/Collection;)Ljava/util/Set;
> (DefaultCuboidScheduler.java:240)
> at
> org.apache.kylin.cube.cuboid.DefaultCuboidScheduler.buildTreeBottomUp()Lorg/apache/kylin/common/util/Pair;
> (DefaultCuboidScheduler.java:183)
> at
> org.apache.kylin.cube.cuboid.DefaultCuboidScheduler.(Lorg/apache/kylin/cube/model/CubeDesc;)V
> (DefaultCuboidScheduler.java:58)
> at
> sun.reflect.GeneratedConstructorAccessor140.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
> (Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
> (DelegatingConstructorAccessorImpl.java:45)
> at
> java.lang.reflect.Constructor.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
> (Constructor.java:423)
> at
> org.apache.kylin.cube.cuboid.CuboidScheduler.getInstance(Lorg/apache/kylin/cube/model/CubeDesc;)Lorg/apache/kylin/cube/cuboid/CuboidScheduler;
> (CuboidScheduler.java:41)
> at
> org.apache.kylin.cube.model.CubeDesc.getInitialCuboidScheduler()Lorg/apache/kylin/cube/cuboid/CuboidScheduler;
> (CubeDesc.java:750)
> at
> org.apache.kylin.cube.cuboid.CuboidCLI.simulateCuboidGeneration(Lorg/apache/kylin/cube/model/CubeDesc;Z)I
> (CuboidCLI.java:47)
> at
> org.apache.kylin.rest.service.CubeService.updateCubeAndDesc(Lorg/apache/kylin/cube/CubeInstance;Lorg/apache/kylin/cube/model/CubeDesc;Ljava/lang/String;Z)Lorg/apache/kylin/cube/model/CubeDesc;
> (CubeService.java:287)
> at
> org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(ILjava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (Unknown Source)
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (MethodProxy.java:204)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;Lorg/springframework/cglib/proxy/MethodProxy;)Ljava/lang/Object;
> (CglibAopProxy.java:669)
> at
> org.apache.kylin.rest.service.CubeService$$EnhancerBySpringCGLIB$$34de75c4.updateCubeAndDesc(Lorg/apache/kylin/cube/CubeInstance;Lorg/apache/kylin/cube/model/CubeDesc;Ljava/lang/String;Z)Lorg/apache/kylin/cube/model/CubeDesc;
> (Unknown Source)
> at
> org.apache.kylin.rest.controller.CubeController.updateCubeDesc(Lorg/apache/kylin/rest/request/CubeRequest;)Lorg/apache/kylin/rest/request/CubeReq
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-4021) Async Broadcast of project schema may cause creating cube failed
Liu Shaohui created KYLIN-4021:
--
Summary: Async Broadcast of project schema may cause creating cube
failed
Key: KYLIN-4021
URL: https://issues.apache.org/jira/browse/KYLIN-4021
Project: Kylin
Issue Type: Bug
Components: Metadata
Reporter: Liu Shaohui
In our prod cluster, we found some creating cube requests failed for the model
not found.
The problem is that users will create the cube right after creating the model
success. But the the two requests may be routed to two different servers.
When the other server receive creating cube request, the project schema may be
not updated for the async Broadcast and the server can not found the model
related to the cube.
the log at query server 1
{code:java}
kylin.log.11:2019-05-27 10:26:44,143 INFO [http-bio-7070-exec-962]
model.DataModelManager:248 : Saving Model model_k1_bb_83_uyyy3636 to
Project BigBI_Hive with bigbi_kylin as owner
kylin.log.11:2019-05-27 10:26:44,144 INFO [http-bio-7070-exec-962]
model.DataModelManager:185 : Model model_k1_bb_83_uyyy3636 is missing or
unloaded yet
kylin.log.11:2019-05-27 10:26:44,145 INFO [http-bio-7070-exec-962]
persistence.ResourceStore:309 : Update resource:
/model_desc/model_k1_bb_83_uyyy3636.json with content:{code}
and the log at query server 2
{code:java}
2019-05-27 10:26:44,296 WARN [http-bio-7070-exec-132] cube.CubeDescManager:195
: Broken cube desc CubeDesc [name=cube_b_bb_83_uyyy3636]
java.lang.NullPointerException: DateModelDesc(model_k1_bb_83_uyyy3636) not
found
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:235)
at org.apache.kylin.cube.model.CubeDesc.init(CubeDesc.java:664)
at
org.apache.kylin.cube.CubeDescManager.createCubeDesc(CubeDescManager.java:193)
at
org.apache.kylin.rest.service.CubeService.createCubeAndDesc(CubeService.java:216)
at
org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke()
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at
org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at
org.apache.kylin.rest.service.CubeService$$EnhancerBySpringCGLIB$$20946622.createCubeAndDesc()
at
org.apache.kylin.rest.controller.CubeController.saveCubeDesc(CubeController.java:735)
at sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-4005) Saving Cube of a aggregation Groups(40 Dimensions, Max Dimension Combination:5) may cause kylin server OOM
Liu Shaohui created KYLIN-4005:
--
Summary: Saving Cube of a aggregation Groups(40 Dimensions, Max
Dimension Combination:5) may cause kylin server OOM
Key: KYLIN-4005
URL: https://issues.apache.org/jira/browse/KYLIN-4005
Project: Kylin
Issue Type: Bug
Components: REST Service
Affects Versions: v2.5.2
Reporter: Liu Shaohui
Fix For: Future
A user try to save a cube with a aggregation Groups(40 Dimensions, Max
Dimension Combination:5) caused the kylin server OOM. The reason is that the
DefaultCuboidScheduler will cost a lot memory when calculating all cube ids.
The stack is following
{code}
http-bio-7070-exec-35
at java.lang.OutOfMemoryError.()V (OutOfMemoryError.java:48)
at java.util.HashMap.resize()[Ljava/util/HashMap$Node; (HashMap.java:704)
at
java.util.HashMap.putVal(ILjava/lang/Object;Ljava/lang/Object;ZZ)Ljava/lang/Object;
(HashMap.java:663)
at
java.util.HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
(HashMap.java:612)
at java.util.HashSet.add(Ljava/lang/Object;)Z (HashSet.java:220)
at java.util.AbstractCollection.addAll(Ljava/util/Collection;)Z
(AbstractCollection.java:344)
at
org.apache.kylin.cube.cuboid.DefaultCuboidScheduler.getOnTreeParentsByLayer(Ljava/util/Collection;)Ljava/util/Set;
(DefaultCuboidScheduler.java:240)
at
org.apache.kylin.cube.cuboid.DefaultCuboidScheduler.buildTreeBottomUp()Lorg/apache/kylin/common/util/Pair;
(DefaultCuboidScheduler.java:183)
at
org.apache.kylin.cube.cuboid.DefaultCuboidScheduler.(Lorg/apache/kylin/cube/model/CubeDesc;)V
(DefaultCuboidScheduler.java:58)
at
sun.reflect.GeneratedConstructorAccessor140.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
(DelegatingConstructorAccessorImpl.java:45)
at
java.lang.reflect.Constructor.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
(Constructor.java:423)
at
org.apache.kylin.cube.cuboid.CuboidScheduler.getInstance(Lorg/apache/kylin/cube/model/CubeDesc;)Lorg/apache/kylin/cube/cuboid/CuboidScheduler;
(CuboidScheduler.java:41)
at
org.apache.kylin.cube.model.CubeDesc.getInitialCuboidScheduler()Lorg/apache/kylin/cube/cuboid/CuboidScheduler;
(CubeDesc.java:750)
at
org.apache.kylin.cube.cuboid.CuboidCLI.simulateCuboidGeneration(Lorg/apache/kylin/cube/model/CubeDesc;Z)I
(CuboidCLI.java:47)
at
org.apache.kylin.rest.service.CubeService.updateCubeAndDesc(Lorg/apache/kylin/cube/CubeInstance;Lorg/apache/kylin/cube/model/CubeDesc;Ljava/lang/String;Z)Lorg/apache/kylin/cube/model/CubeDesc;
(CubeService.java:287)
at
org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(ILjava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(Unknown Source)
at
org.springframework.cglib.proxy.MethodProxy.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(MethodProxy.java:204)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;Lorg/springframework/cglib/proxy/MethodProxy;)Ljava/lang/Object;
(CglibAopProxy.java:669)
at
org.apache.kylin.rest.service.CubeService$$EnhancerBySpringCGLIB$$34de75c4.updateCubeAndDesc(Lorg/apache/kylin/cube/CubeInstance;Lorg/apache/kylin/cube/model/CubeDesc;Ljava/lang/String;Z)Lorg/apache/kylin/cube/model/CubeDesc;
(Unknown Source)
at
org.apache.kylin.rest.controller.CubeController.updateCubeDesc(Lorg/apache/kylin/rest/request/CubeRequest;)Lorg/apache/kylin/rest/request/CubeReq
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3818) After Cube disabled, auto-merge cube job still running
[ https://issues.apache.org/jira/browse/KYLIN-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836980#comment-16836980 ] Liu Shaohui commented on KYLIN-3818: I encountered the same problem yesterday in kylin 2.5.2. Thanks for the fix. > After Cube disabled, auto-merge cube job still running > -- > > Key: KYLIN-3818 > URL: https://issues.apache.org/jira/browse/KYLIN-3818 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.6.0 >Reporter: Na Zhai >Assignee: Na Zhai >Priority: Major > Fix For: v2.6.2, v3.0.0-alpha > > > *precondition* > There is a Cube that turns on the auto-merge feature. And it satisfied the > condition of the auto-merge. Then the job of merging segments begins. > After a few minutes, the job of merging segments goes into the error status, > so I discard the job. Then I disable this cube, but a new job of merging > segments begin to run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3997) Add a health check job of Kylin
[
https://issues.apache.org/jira/browse/KYLIN-3997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836016#comment-16836016
]
Liu Shaohui commented on KYLIN-3997:
[~nichunen] It will be a script or background task like the storage cleanup job.
> Add a health check job of Kylin
> ---
>
> Key: KYLIN-3997
> URL: https://issues.apache.org/jira/browse/KYLIN-3997
> Project: Kylin
> Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
>
> Kylin has many inner meta data and outer dependencies. There may be
> inconsistent for bugs or failures. It's better to have a a health check job
> to find these inconsistent issues in advance。
> The inconsistent issues we found in our clusters are followings
> * {color:#808080}the cubeid data not exist for cube merging{color}
> * {color:#808080}hbase table not exist or online for a segment{color}
> * {color:#808080}there are holes in cube segments(The build of some days
> failed, but user not found it){color}
> * {color:#808080}Too many segment(hbase tables){color}
> * {color:#808080}metadata of stale segment left in cube{color}
> * {color:#808080}Some cubes have not be updated/built for a long time{color}
> * {color:#808080}Some important parameters are no set in cube desc{color}
> * {color:#808080}...{color}
> Suggestions are welcomed, thanks~
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3997) Add a health check job of Kylin
[
https://issues.apache.org/jira/browse/KYLIN-3997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-3997:
---
Description:
Kylin has many inner meta data and outer dependencies. There may be
inconsistent for bugs or failures. It's better to have a a health check job to
find these inconsistent issues in advance。
The inconsistent issues we found in our clusters are followings
* {color:#808080}the cubeid data not exist for cube merging{color}
* {color:#808080}hbase table not exist or online for a segment{color}
* {color:#808080}there are holes in cube segments(The build of some days
failed, but user not found it){color}
* {color:#808080}Too many segment(hbase tables){color}
* {color:#808080}metadata of stale segment left in cube{color}
* {color:#808080}Some cubes have not be updated/built for a long time{color}
* {color:#808080}Some important parameters are no set in cube desc{color}
* {color:#808080}...{color}
Suggestions are welcomed, thanks~
was:
Kylin has many inner meta data and outer dependencies. There may be no
inconsistent for bugs or failures. It's better to have a a health check job to
find these inconsistent issues in advance。
The inconsistent issues we found in our clusters are followings
* {color:#808080}the cubeid data not exist for cube merging{color}
* {color:#808080}hbase table not exist or online for a segment{color}
* {color:#808080}there are holes in cube segments(The build of some days
failed, but user not found it){color}
* {color:#808080}Too many segment(hbase tables){color}
* {color:#808080}metadata of stale segment left in cube{color}
* {color:#808080}Some cubes have not be updated/built for a long time{color}
* {color:#808080}Some important parameters are no set in cube desc{color}
* {color:#808080}...{color}
Suggestions are welcomed, thanks~
> Add a health check job of Kylin
> ---
>
> Key: KYLIN-3997
> URL: https://issues.apache.org/jira/browse/KYLIN-3997
> Project: Kylin
> Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
>
> Kylin has many inner meta data and outer dependencies. There may be
> inconsistent for bugs or failures. It's better to have a a health check job
> to find these inconsistent issues in advance。
> The inconsistent issues we found in our clusters are followings
> * {color:#808080}the cubeid data not exist for cube merging{color}
> * {color:#808080}hbase table not exist or online for a segment{color}
> * {color:#808080}there are holes in cube segments(The build of some days
> failed, but user not found it){color}
> * {color:#808080}Too many segment(hbase tables){color}
> * {color:#808080}metadata of stale segment left in cube{color}
> * {color:#808080}Some cubes have not be updated/built for a long time{color}
> * {color:#808080}Some important parameters are no set in cube desc{color}
> * {color:#808080}...{color}
> Suggestions are welcomed, thanks~
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3997) Add a health check job of Kylin
[
https://issues.apache.org/jira/browse/KYLIN-3997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-3997:
---
Description:
Kylin has many inner meta data and outer dependencies. There may be no
inconsistent for bugs or failures. It's better to have a a health check job to
find these inconsistent issues in advance。
The inconsistent issues we found in our clusters are followings
* {color:#808080}the cubeid data not exist for cube merging{color}
* {color:#808080}hbase table not exist or online for a segment{color}
* {color:#808080}there are holes in cube segments(The build of some days
failed, but user not found it){color}
* {color:#808080}Too many segment(hbase tables){color}
* {color:#808080}metadata of stale segment left in cube{color}
* {color:#808080}Some cubes have not be updated/built for a long time{color}
* {color:#808080}Some important parameters are no set in cube desc{color}
* {color:#808080}...{color}
Suggestions are welcomed, thanks~
was:
Kylin has many inner meta data and outer dependencies. There may be no
inconsistent for bugs or failures. It's better to have a a health check job to
find these inconsistent issues in advance。
The inconsistent issues we found in our clusters are followings
* {color:#808080}the cubeid data not exist for cube merging
{color}
* {color:#808080}hbase table not exist or online for a segment{color}
* {color:#808080}there are holes in cube segments(The build of some days
failed, but user not found it){color}
* {color:#808080}Too many segment(hbase tables){color}
* {color:#808080}metadata of stale segment left in cube{color}
* {color:#808080}Some cubes have not be updated/built for a long time{color}
* {color:#808080}Some important parameters are no set in cube desc{color}
* {color:#808080}...{color}
> Add a health check job of Kylin
> ---
>
> Key: KYLIN-3997
> URL: https://issues.apache.org/jira/browse/KYLIN-3997
> Project: Kylin
> Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
>
> Kylin has many inner meta data and outer dependencies. There may be no
> inconsistent for bugs or failures. It's better to have a a health check job
> to find these inconsistent issues in advance。
> The inconsistent issues we found in our clusters are followings
> * {color:#808080}the cubeid data not exist for cube merging{color}
> * {color:#808080}hbase table not exist or online for a segment{color}
> * {color:#808080}there are holes in cube segments(The build of some days
> failed, but user not found it){color}
> * {color:#808080}Too many segment(hbase tables){color}
> * {color:#808080}metadata of stale segment left in cube{color}
> * {color:#808080}Some cubes have not be updated/built for a long time{color}
> * {color:#808080}Some important parameters are no set in cube desc{color}
> * {color:#808080}...{color}
> Suggestions are welcomed, thanks~
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3997) Add a health check job of Kylin
Liu Shaohui created KYLIN-3997:
--
Summary: Add a health check job of Kylin
Key: KYLIN-3997
URL: https://issues.apache.org/jira/browse/KYLIN-3997
Project: Kylin
Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Kylin has many inner meta data and outer dependencies. There may be no
inconsistent for bugs or failures. It's better to have a a health check job to
find these inconsistent issues in advance。
The inconsistent issues we found in our clusters are followings
* {color:#808080}the cubeid data not exist for cube merging
{color}
* {color:#808080}hbase table not exist or online for a segment{color}
* {color:#808080}there are holes in cube segments(The build of some days
failed, but user not found it){color}
* {color:#808080}Too many segment(hbase tables){color}
* {color:#808080}metadata of stale segment left in cube{color}
* {color:#808080}Some cubes have not be updated/built for a long time{color}
* {color:#808080}Some important parameters are no set in cube desc{color}
* {color:#808080}...{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3994) StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager
[ https://issues.apache.org/jira/browse/KYLIN-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui reassigned KYLIN-3994: -- Assignee: Liu Shaohui > StorageCleanupJob may delete cube id data of new built segment because of > cube cache in CubeManager > --- > > Key: KYLIN-3994 > URL: https://issues.apache.org/jira/browse/KYLIN-3994 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > > In our production cluster, we found that the cube id data of a new-built > segment is deleted by the StorageCleanupJob. > After checking the code of cleanUnusedHdfsFiles in StorageCleanupJob, we > found that there is a bug here: CubeManager read all cube meta in > initiation and cache it for later > listAllCubes operations, the metadata will be out of data after listing the > hdfs working dir. > So the working directory of a finished job may be deleted unexpectedly. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3994) StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager
Liu Shaohui created KYLIN-3994: -- Summary: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager Key: KYLIN-3994 URL: https://issues.apache.org/jira/browse/KYLIN-3994 Project: Kylin Issue Type: Bug Affects Versions: v2.5.2 Reporter: Liu Shaohui In our production cluster, we found that the cube id data of a new-built segment is deleted by the StorageCleanupJob. After checking the code of cleanUnusedHdfsFiles in StorageCleanupJob, we found that there is a bug here: CubeManager read all cube meta in initiation and cache it for later listAllCubes operations, the metadata will be out of data after listing the hdfs working dir. So the working directory of a finished job may be deleted unexpectedly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3962) Support streaming cubing using Spark Streaming or Flink
[ https://issues.apache.org/jira/browse/KYLIN-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-3962: --- Summary: Support streaming cubing using Spark Streaming or Flink (was: Support streaming cubing using Spark Streaming of Flink) > Support streaming cubing using Spark Streaming or Flink > --- > > Key: KYLIN-3962 > URL: https://issues.apache.org/jira/browse/KYLIN-3962 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > KYLIN-3654 introduced the Real-time Streaming, but in my opinion, the arch is > a little too complicated to handle. > As streaming frameworks like spark streaming, flink are widely used in many > companies.Can we use the streaming framework to support real time cubing in > Kylin. > This is just a proposal. More discussion and suggestions are welcomed~ > More details of this proposal will be added later. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3962) Support streaming cubing using Spark Streaming
Liu Shaohui created KYLIN-3962: -- Summary: Support streaming cubing using Spark Streaming Key: KYLIN-3962 URL: https://issues.apache.org/jira/browse/KYLIN-3962 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui KYLIN-3654 introduced the Real-time Streaming, but in my opinion, the arch is a little too complicated to handle. As streaming frameworks like spark streaming, flink are widely used in many companies.Can we use the streaming framework to support real time cubing in Kylin. This is just a proposal. More discussion and suggestions are welcomed~ More details of this proposal will be added later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3962) Support streaming cubing using Spark Streaming of Flink
[ https://issues.apache.org/jira/browse/KYLIN-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-3962: --- Summary: Support streaming cubing using Spark Streaming of Flink (was: Support streaming cubing using Spark Streaming) > Support streaming cubing using Spark Streaming of Flink > --- > > Key: KYLIN-3962 > URL: https://issues.apache.org/jira/browse/KYLIN-3962 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > KYLIN-3654 introduced the Real-time Streaming, but in my opinion, the arch is > a little too complicated to handle. > As streaming frameworks like spark streaming, flink are widely used in many > companies.Can we use the streaming framework to support real time cubing in > Kylin. > This is just a proposal. More discussion and suggestions are welcomed~ > More details of this proposal will be added later. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3758) Flink Cube Build Engine
[ https://issues.apache.org/jira/browse/KYLIN-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui reassigned KYLIN-3758: -- Assignee: Liu Shaohui (was: vinoyang) > Flink Cube Build Engine > --- > > Key: KYLIN-3758 > URL: https://issues.apache.org/jira/browse/KYLIN-3758 > Project: Kylin > Issue Type: New Feature > Components: Flink Engine >Reporter: vinoyang >Assignee: Liu Shaohui >Priority: Major > > Kylin currently supports Apache Spark as the engine for building Cubes. As a > next-generation computing engine, Apache Flink is dedicated to the > convergence of streams and batches, with superior performance and elegant > architecture. Flink and Kylin have many overlapping users in China. If Kylin > can support Flink as the engine for building Cube, then this will give many > users more choices. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3918) Add project name in cube and job pages
[ https://issues.apache.org/jira/browse/KYLIN-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812248#comment-16812248 ] Liu Shaohui commented on KYLIN-3918: [~Shaofengshi] In my opinion, it not too wide. The project info is useful and I can't not find another more suitable position. Any advice about the position? Thanks~ > Add project name in cube and job pages > -- > > Key: KYLIN-3918 > URL: https://issues.apache.org/jira/browse/KYLIN-3918 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: v2.6.2 > > > In a production cluster, there will be many projects and each project has > many cubes. It's useful to show project name in cube and job pages. > So the admin can be quick to known which project the abnormal cube or failed > job belongs to and get contact with the users. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3918) Add project name in cube and job pages
[ https://issues.apache.org/jira/browse/KYLIN-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-3918: --- Priority: Minor (was: Major) > Add project name in cube and job pages > -- > > Key: KYLIN-3918 > URL: https://issues.apache.org/jira/browse/KYLIN-3918 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Minor > > In a production cluster, there will be many projects and each project has > many cubes. It's useful to show project name in cube and job pages. > So the admin can be quick to known which project the abnormal cube or failed > job belongs to and get contact with the users. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3918) Add project name in cube and job pages
Liu Shaohui created KYLIN-3918: -- Summary: Add project name in cube and job pages Key: KYLIN-3918 URL: https://issues.apache.org/jira/browse/KYLIN-3918 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui In a production cluster, there will be many projects and each project has many cubes. It's useful to show project name in cube and job pages. So the admin can be quick to known which project the abnormal cube or failed job belongs to and get contact with the users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3917) Add max segment merge span to cleanup intermediate data of cube building
Liu Shaohui created KYLIN-3917: -- Summary: Add max segment merge span to cleanup intermediate data of cube building Key: KYLIN-3917 URL: https://issues.apache.org/jira/browse/KYLIN-3917 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Currently the intermediate data of building cube can not be deleted for it maybe used for later cubing merging. But it result in double space used in HDFS. In actual scenario, we only need month-level segment span in maximum. So if a span of segment is larger than a month, we think it don't need be merged and the intermediate data can be deleted. So we can add a config kylin.cube.max-segment-merge.span, default is -1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3913) Remove getAllOutputs api in ExecutableManager to avoid OOM for large metadata
Liu Shaohui created KYLIN-3913: -- Summary: Remove getAllOutputs api in ExecutableManager to avoid OOM for large metadata Key: KYLIN-3913 URL: https://issues.apache.org/jira/browse/KYLIN-3913 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui In a big cluster, there will be many job info left in the metadata. The kylin server will be OOM when search the jobs with a long time range. The reason is that ExecutableManager will load all job output info into memory when search a job. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3912) Support cube level mapreduce queue config for BeelineHiveClient
[
https://issues.apache.org/jira/browse/KYLIN-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-3912:
---
Summary: Support cube level mapreduce queue config for BeelineHiveClient
(was: Support cube level mapreduuce queue config for BeelineHiveClient)
> Support cube level mapreduce queue config for BeelineHiveClient
> ---
>
> Key: KYLIN-3912
> URL: https://issues.apache.org/jira/browse/KYLIN-3912
> Project: Kylin
> Issue Type: Improvement
>Reporter: Liu Shaohui
>Priority: Major
>
> To support multi tenants, we set different mapreduce queue config for
> different projects and cubes, but BeelineHiveClient don't use those configs.
> So the getHiveTableRows api always run on same queue in kylin_hive_conf or
> jdbc url, which cause computing resource competition.
>
> {code:java}
> 2018-11-28 15:37:27,261 ERROR [Scheduler 1950398337 Job
> 08b3ee43-c84d-4039-84c5-a36ecb2cff18-228] execution.AbstractExecutable:383 :
> job:08b3ee43-c84d-4039-84c5-a36ecb2cff18-01 execute finished with exception
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> downstreamError is null.
> Query log:
> http://zjy-hadoop-prc-ct14.bj:28911/log?qid=a05e1629-2072-46dd-9d71-b5722d04b2aa
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:277)
> at
> org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:376)
> at
> org.apache.kylin.source.hive.BeelineHiveClient.getHiveTableRows(BeelineHiveClient.java:108)
> at
> org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.computeRowCount(HiveMRInput.java:304)
> at
> org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.doWork(HiveMRInput.java:354)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3912) Support cube level mapreduuce queue config for BeelineHiveClient
[
https://issues.apache.org/jira/browse/KYLIN-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-3912:
---
Description:
To support multi tenants, we set different mapreduce queue config for different
projects and cubes, but BeelineHiveClient don't use those configs. So the
getHiveTableRows api always run on same queue in kylin_hive_conf or jdbc url,
which cause computing resource competition.
{code:java}
2018-11-28 15:37:27,261 ERROR [Scheduler 1950398337 Job
08b3ee43-c84d-4039-84c5-a36ecb2cff18-228] execution.AbstractExecutable:383 :
job:08b3ee43-c84d-4039-84c5-a36ecb2cff18-01 execute finished with exception
java.sql.SQLException: Error while processing statement: FAILED: Execution
Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
downstreamError is null.
Query log:
http://zjy-hadoop-prc-ct14.bj:28911/log?qid=a05e1629-2072-46dd-9d71-b5722d04b2aa
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:277)
at
org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:376)
at
org.apache.kylin.source.hive.BeelineHiveClient.getHiveTableRows(BeelineHiveClient.java:108)
at
org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.computeRowCount(HiveMRInput.java:304)
at
org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.doWork(HiveMRInput.java:354)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
was:
To support multi tenants, we set different mapreduce queue config for different
projects and cubes, but BeelineHiveClient don't use those configs. So the
getHiveTableRows api always run on same queue in kylin_hive_conf or jdbc url,
which cause computing resource competition.
> Support cube level mapreduuce queue config for BeelineHiveClient
>
>
> Key: KYLIN-3912
> URL: https://issues.apache.org/jira/browse/KYLIN-3912
> Project: Kylin
> Issue Type: Improvement
>Reporter: Liu Shaohui
>Priority: Major
>
> To support multi tenants, we set different mapreduce queue config for
> different projects and cubes, but BeelineHiveClient don't use those configs.
> So the getHiveTableRows api always run on same queue in kylin_hive_conf or
> jdbc url, which cause computing resource competition.
>
> {code:java}
> 2018-11-28 15:37:27,261 ERROR [Scheduler 1950398337 Job
> 08b3ee43-c84d-4039-84c5-a36ecb2cff18-228] execution.AbstractExecutable:383 :
> job:08b3ee43-c84d-4039-84c5-a36ecb2cff18-01 execute finished with exception
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> downstreamError is null.
> Query log:
> http://zjy-hadoop-prc-ct14.bj:28911/log?qid=a05e1629-2072-46dd-9d71-b5722d04b2aa
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:277)
> at
> org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:376)
> at
> org.apache.kylin.source.hive.BeelineHiveClient.getHiveTableRows(BeelineHiveClient.java:108)
> at
> org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.computeRowCount(HiveMRInput.java:304)
> at
> org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.doWork(HiveMRInput.java:354)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3912) Support cube level mapreduuce queue config for BeelineHiveClient
Liu Shaohui created KYLIN-3912: -- Summary: Support cube level mapreduuce queue config for BeelineHiveClient Key: KYLIN-3912 URL: https://issues.apache.org/jira/browse/KYLIN-3912 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui To support multi tenants, we set different mapreduce queue config for different projects and cubes, but BeelineHiveClient don't use those configs. So the getHiveTableRows api always run on same queue in kylin_hive_conf or jdbc url, which cause computing resource competition. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3911) Check if HBase table is enabled before diabling table in DeployCoprocessorCLI
Liu Shaohui created KYLIN-3911:
--
Summary: Check if HBase table is enabled before diabling table in
DeployCoprocessorCLI
Key: KYLIN-3911
URL: https://issues.apache.org/jira/browse/KYLIN-3911
Project: Kylin
Issue Type: Bug
Reporter: Liu Shaohui
HBase tables may be disabled for operation issues and the previous interrupted
DeployCoprocessorCLI, which cause the new DeployCoprocessorCLI failed.
{code:java}
2018-06-08 10:40:23,489 ERROR [pool-5-thread-6] util.DeployCoprocessorCLI:383 :
Error processing kylin_bigdata_prod:KYLIN_A9520J93GU
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
kylin_bigdata_prod:KYLIN_A9520J93GU
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:102)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:2609)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:2619)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44586)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2061)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:125)
at
org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:83)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
@c3-hadoop-prc-ct36.bj/10.136.14.13:33500
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:220)
ingCaller.java:86)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3239)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:919)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTable(HBaseAdmin.java:948)
at
org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:294)
at
org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
```{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3909) kylin job failed for MappeableRunContainer is not registered
Liu Shaohui created KYLIN-3909: -- Summary: kylin job failed for MappeableRunContainer is not registered Key: KYLIN-3909 URL: https://issues.apache.org/jira/browse/KYLIN-3909 Project: Kylin Issue Type: Bug Reporter: Liu Shaohui |Job aborted due to stage failure: Task 2 in stage 1.0 failed 4 times, most recent failure: Lost task 2.3 in stage 1.0 (TID 2621, zjy-hadoop-prc-st2587.bj, executor 53): com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.apache.kylin.job.shaded.org.roaringbitmap.buffer.MappeableRunContainer| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3900) Discard all expired ERROR or STOPPED jobs to cleanup kylin metadata
[ https://issues.apache.org/jira/browse/KYLIN-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800584#comment-16800584 ] Liu Shaohui commented on KYLIN-3900: [~Shaofengshi] The patch just discard the *expired(eg: two weeks ago)* ERROR or STOPPED jobs > Discard all expired ERROR or STOPPED jobs to cleanup kylin metadata > --- > > Key: KYLIN-3900 > URL: https://issues.apache.org/jira/browse/KYLIN-3900 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > Currently metadata cleanup job only delete expired discarded and succeed > jobs, ERROR or STOPPED jobs are left which may cause too many meta in hbase > in a long term. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3900) Discard all expired ERROR or STOPPED jobs to cleanup kylin metadata
[ https://issues.apache.org/jira/browse/KYLIN-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800582#comment-16800582 ] Liu Shaohui commented on KYLIN-3900: [~Shaofengshi] users may forgot to discard the stopped jobs which cause OOM of kylin server when searching jobs. In the current implements, kylin server will fetch all job info into memory when filtering the jobs. > Discard all expired ERROR or STOPPED jobs to cleanup kylin metadata > --- > > Key: KYLIN-3900 > URL: https://issues.apache.org/jira/browse/KYLIN-3900 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > Currently metadata cleanup job only delete expired discarded and succeed > jobs, ERROR or STOPPED jobs are left which may cause too many meta in hbase > in a long term. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3901) Use multi threads to speed up the storage cleanup job
Liu Shaohui created KYLIN-3901: -- Summary: Use multi threads to speed up the storage cleanup job Key: KYLIN-3901 URL: https://issues.apache.org/jira/browse/KYLIN-3901 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3901) Use multi threads to speed up the storage cleanup job
[ https://issues.apache.org/jira/browse/KYLIN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-3901: --- Description: Currently, the storage cleanup job only use one thread to clean up hbase table, hive table, and hdfs dirs. It''s better to use multi threads to speed it up. was:Currently, > Use multi threads to speed up the storage cleanup job > - > > Key: KYLIN-3901 > URL: https://issues.apache.org/jira/browse/KYLIN-3901 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > Currently, the storage cleanup job only use one thread to clean up hbase > table, hive table, and hdfs dirs. > It''s better to use multi threads to speed it up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3901) Use multi threads to speed up the storage cleanup job
[ https://issues.apache.org/jira/browse/KYLIN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-3901: --- Description: Currently, > Use multi threads to speed up the storage cleanup job > - > > Key: KYLIN-3901 > URL: https://issues.apache.org/jira/browse/KYLIN-3901 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Priority: Major > > Currently, -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3900) Discard all expired ERROR or STOPPED jobs to cleanup kylin metadata
Liu Shaohui created KYLIN-3900: -- Summary: Discard all expired ERROR or STOPPED jobs to cleanup kylin metadata Key: KYLIN-3900 URL: https://issues.apache.org/jira/browse/KYLIN-3900 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Currently metadata cleanup job only delete expired discarded and succeed jobs, ERROR or STOPPED jobs are left which may cause too many meta in hbase in a long term. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3893) Cube build failed for wrong row key column description
[
https://issues.apache.org/jira/browse/KYLIN-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui reassigned KYLIN-3893:
--
Assignee: Liu Shaohui
> Cube build failed for wrong row key column description
> --
>
> Key: KYLIN-3893
> URL: https://issues.apache.org/jira/browse/KYLIN-3893
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
>
> User created wrong RowKeyColDesc, eg,
> RowKeyColDesc\{column=MYSQL_FEEDBACK_USER_AUDIT.DATE,
> encoding=integer:undefined}
> which cause the cube build forever.
>
> {code:java}
> org.apache.kylin.engine.mr.exception.HadoopShellException:
> java.lang.NumberFormatException: For input string: "undefined" at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:580) at
> java.lang.Integer.parseInt(Integer.java:615) at
> org.apache.kylin.dimension.IntegerDimEnc$Factory.createDimensionEncoding(IntegerDimEnc.java:65)
> at
> org.apache.kylin.dimension.DimensionEncodingFactory.create(DimensionEncodingFactory.java:65)
> at org.apache.kylin.cube.kv.CubeDimEncMap.get(CubeDimEncMap.java:74)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMapFromRowCount(CubeStatsReader.java:206)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMap(CubeStatsReader.java:170)
> at
> org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:102)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92) at
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)result code:2 at
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3893) Cube build failed for wrong row key column description
Liu Shaohui created KYLIN-3893:
--
Summary: Cube build failed for wrong row key column description
Key: KYLIN-3893
URL: https://issues.apache.org/jira/browse/KYLIN-3893
Project: Kylin
Issue Type: Bug
Reporter: Liu Shaohui
User created wrong RowKeyColDesc, eg,
RowKeyColDesc\{column=MYSQL_FEEDBACK_USER_AUDIT.DATE,
encoding=integer:undefined}
which cause the cube build forever.
{code:java}
org.apache.kylin.engine.mr.exception.HadoopShellException:
java.lang.NumberFormatException: For input string: "undefined" at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580) at
java.lang.Integer.parseInt(Integer.java:615) at
org.apache.kylin.dimension.IntegerDimEnc$Factory.createDimensionEncoding(IntegerDimEnc.java:65)
at
org.apache.kylin.dimension.DimensionEncodingFactory.create(DimensionEncodingFactory.java:65)
at org.apache.kylin.cube.kv.CubeDimEncMap.get(CubeDimEncMap.java:74) at
org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMapFromRowCount(CubeStatsReader.java:206)
at
org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMap(CubeStatsReader.java:170)
at
org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:102)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92) at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)result code:2 at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3893) Cube build failed for wrong row key column description
[
https://issues.apache.org/jira/browse/KYLIN-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-3893:
---
Priority: Minor (was: Major)
> Cube build failed for wrong row key column description
> --
>
> Key: KYLIN-3893
> URL: https://issues.apache.org/jira/browse/KYLIN-3893
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Priority: Minor
>
> User created wrong RowKeyColDesc, eg,
> RowKeyColDesc\{column=MYSQL_FEEDBACK_USER_AUDIT.DATE,
> encoding=integer:undefined}
> which cause the cube build forever.
>
> {code:java}
> org.apache.kylin.engine.mr.exception.HadoopShellException:
> java.lang.NumberFormatException: For input string: "undefined" at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:580) at
> java.lang.Integer.parseInt(Integer.java:615) at
> org.apache.kylin.dimension.IntegerDimEnc$Factory.createDimensionEncoding(IntegerDimEnc.java:65)
> at
> org.apache.kylin.dimension.DimensionEncodingFactory.create(DimensionEncodingFactory.java:65)
> at org.apache.kylin.cube.kv.CubeDimEncMap.get(CubeDimEncMap.java:74)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMapFromRowCount(CubeStatsReader.java:206)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMap(CubeStatsReader.java:170)
> at
> org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:102)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92) at
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)result code:2 at
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3884) loading hfile to HBase failed for temporary dir in output path
[
https://issues.apache.org/jira/browse/KYLIN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794908#comment-16794908
]
Liu Shaohui edited comment on KYLIN-3884 at 3/18/19 10:32 AM:
--
[~Shaofengshi] When running spark job with spark.speculation=true, there may
be two running task for same task id at a time.
If a running task left after the the job is committed, it will create a new
_temporary dir in the output path.
See:
[https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java]
#196
Be default, fs mkdirs api will create the parent path if it not exist.
{code:java}
fs.mkdirs(writerPath);
configureStoragePolicy(conf, fs, tableAndFamily, writerPath);{code}
was (Author: liushaohui):
[~Shaofengshi] When running spark job with spark.speculation=true, there may
be too running task for same task id at a time.
If a running task left after the the job is committed, it will create a new
_temporary dir in the output path.
See:
[https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java]
#196
Be default, fs mkdirs api will create the parent path if it not exist.
{code:java}
fs.mkdirs(writerPath);
configureStoragePolicy(conf, fs, tableAndFamily, writerPath);{code}
> loading hfile to HBase failed for temporary dir in output path
> ---
>
> Key: KYLIN-3884
> URL: https://issues.apache.org/jira/browse/KYLIN-3884
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Priority: Major
>
> {code:java}
> 2019-03-14 20:18:46,591 DEBUG [Scheduler 2084224398 Job
> e48de76a-6e16-309f-a3a5-191c04071072-131] steps.BulkLoadJob:77 : Start to run
> LoadIncrementalHFiles
> 2019-03-14 20:18:46,642 WARN [Scheduler 2084224398 Job
> e48de76a-6e16-309f-a3a5-191c04071072-131] mapreduce.LoadIncrementalHFiles:197
> : Skipping non-directory
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_SUCCESS
> 2019-03-14 20:18:46,650 ERROR [Scheduler 2084224398 Job
> e48de76a-6e16-309f-a3a5-191c04071072-131] mapreduce.LoadIncrementalHFiles:352
> : -
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/4170d772384144848c1c10cba66152c3
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/50ec331ff3c648e3b6e4f54a7b1fe7e9
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/703ade3b535b4fedab39ee183e22aa7c
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/82019f8ca00a4f16b9d2b45356a55a3a
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/8cc8844bced24cb88fda52fecc7224d5
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/cbac78e0c6d74b5c96a7b64f99e0d0b3
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/e3844766a4d0486d89f287450034f378
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_temporary/0
> 2019-03-14 20:18:46,651 ERROR [Scheduler 2084224398 Job
> e48de76a-6e16-309f-a3a5-191c04071072-131] common.HadoopShellExecutable:65 :
> error execute
> HadoopShellExecutable{id=e48de76a-6e16-309f-a3a5-191c04071072-08, name=Load
> HFile to HBase Table, state=RUNNING}
> java.io.FileNotFoundException: Path is not a file:
> /user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_temporary/0
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path is
> not a file:
> /user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_tem
[jira] [Commented] (KYLIN-3884) loading hfile to HBase failed for temporary dir in output path
[
https://issues.apache.org/jira/browse/KYLIN-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794908#comment-16794908
]
Liu Shaohui commented on KYLIN-3884:
[~Shaofengshi] When running spark job with spark.speculation=true, there may
be too running task for same task id at a time.
If a running task left after the the job is committed, it will create a new
_temporary dir in the output path.
See:
[https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java]
#196
Be default, fs mkdirs api will create the parent path if it not exist.
{code:java}
fs.mkdirs(writerPath);
configureStoragePolicy(conf, fs, tableAndFamily, writerPath);{code}
> loading hfile to HBase failed for temporary dir in output path
> ---
>
> Key: KYLIN-3884
> URL: https://issues.apache.org/jira/browse/KYLIN-3884
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Priority: Major
>
> {code:java}
> 2019-03-14 20:18:46,591 DEBUG [Scheduler 2084224398 Job
> e48de76a-6e16-309f-a3a5-191c04071072-131] steps.BulkLoadJob:77 : Start to run
> LoadIncrementalHFiles
> 2019-03-14 20:18:46,642 WARN [Scheduler 2084224398 Job
> e48de76a-6e16-309f-a3a5-191c04071072-131] mapreduce.LoadIncrementalHFiles:197
> : Skipping non-directory
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_SUCCESS
> 2019-03-14 20:18:46,650 ERROR [Scheduler 2084224398 Job
> e48de76a-6e16-309f-a3a5-191c04071072-131] mapreduce.LoadIncrementalHFiles:352
> : -
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/4170d772384144848c1c10cba66152c3
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/50ec331ff3c648e3b6e4f54a7b1fe7e9
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/703ade3b535b4fedab39ee183e22aa7c
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/82019f8ca00a4f16b9d2b45356a55a3a
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/8cc8844bced24cb88fda52fecc7224d5
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/cbac78e0c6d74b5c96a7b64f99e0d0b3
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/e3844766a4d0486d89f287450034f378
>
> hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_temporary/0
> 2019-03-14 20:18:46,651 ERROR [Scheduler 2084224398 Job
> e48de76a-6e16-309f-a3a5-191c04071072-131] common.HadoopShellExecutable:65 :
> error execute
> HadoopShellExecutable{id=e48de76a-6e16-309f-a3a5-191c04071072-08, name=Load
> HFile to HBase Table, state=RUNNING}
> java.io.FileNotFoundException: Path is not a file:
> /user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_temporary/0
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path is
> not a file:
> /user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_temporary/0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3887) Query with decimal sum measure of double complied failed after KYLIN-3703
[
https://issues.apache.org/jira/browse/KYLIN-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794799#comment-16794799
]
Liu Shaohui commented on KYLIN-3887:
[~temple.zhou] Will you help to look at this issue? Revert the commit of
KYLIN-3703, the compile problem will disappear.
> Query with decimal sum measure of double complied failed after KYLIN-3703
> -
>
> Key: KYLIN-3887
> URL: https://issues.apache.org/jira/browse/KYLIN-3887
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Priority: Major
>
> After KYLIN-3703, Query with decimal sum measure of double complied failed.
> {code:java}
> Caused by: org.codehaus.commons.compiler.CompileException:
> Line 112, Column 42: Cannot cast "java.math.BigDecimal" to
> "java.lang.Double"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3887) Query with decimal sum measure of double complied failed after KYLIN-3703
Liu Shaohui created KYLIN-3887:
--
Summary: Query with decimal sum measure of double complied failed
after KYLIN-3703
Key: KYLIN-3887
URL: https://issues.apache.org/jira/browse/KYLIN-3887
Project: Kylin
Issue Type: Bug
Reporter: Liu Shaohui
After KYLIN-3703, Query with decimal sum measure of double complied failed.
{code:java}
Caused by: org.codehaus.commons.compiler.CompileException:
Line 112, Column 42: Cannot cast "java.math.BigDecimal" to
"java.lang.Double"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3886) Missing argument for options for yarn command
[
https://issues.apache.org/jira/browse/KYLIN-3886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-3886:
---
Description:
The app id is empty which cause the yarn command failed for missing argument
{code:java}
2019-03-13 11:48:08,604 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : Missing
argument for options
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : usage:
application
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -appStates
Works with -list to filter applications
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : based on
input comma-separated list of
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 :
application states. The valid application
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : state can
be one of the following:
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 :
ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 :
NING,FINISHED,FAILED,KILLED
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -appTypes
Works with -list to filter applications
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : based on
input comma-separated list of
2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 :
application types.
2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -help
Displays help for all commands.
2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -kill
Kills the application.
2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -list List
applications. Supports optional use
2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : of
-appTypes to filter applications based{code}
was:
2019-03-13 11:48:08,604 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : Missing
argument for options
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : usage:
application
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 :
-appStates Works with -list to filter applications
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 :
based on input comma-separated list of
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 :
application states. The valid application
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 :
state can be one of the following:
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 :
ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 :
NING,FINISHED,FAILED,KILLED
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -appTypes
Works with -list to filter applications
2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 :
based on input comma-separated list of
2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 :
application types.
2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job
f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -help
Displays help for all commands.
2019-03-13 11:48:08,607
[jira] [Created] (KYLIN-3886) Missing argument for options for yarn command
Liu Shaohui created KYLIN-3886: -- Summary: Missing argument for options for yarn command Key: KYLIN-3886 URL: https://issues.apache.org/jira/browse/KYLIN-3886 Project: Kylin Issue Type: Bug Reporter: Liu Shaohui 2019-03-13 11:48:08,604 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : Missing argument for options 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : usage: application 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -appStates Works with -list to filter applications 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : based on input comma-separated list of 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : application states. The valid application 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : state can be one of the following: 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : NING,FINISHED,FAILED,KILLED 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -appTypes Works with -list to filter applications 2019-03-13 11:48:08,606 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : based on input comma-separated list of 2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : application types. 2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -help Displays help for all commands. 2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -kill Kills the application. 2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : -list List applications. Supports optional use 2019-03-13 11:48:08,607 INFO [Scheduler 542945608 Job f918877a-deb0-704c-ec6f-82f33f5e39a5-323] spark.SparkExecutable:38 : of -appTypes to filter applications based -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3885) Build dimension dictionary job costs too long when using Spark fact distinct
[
https://issues.apache.org/jira/browse/KYLIN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui reassigned KYLIN-3885:
--
Assignee: Liu Shaohui
> Build dimension dictionary job costs too long when using Spark fact distinct
>
>
> Key: KYLIN-3885
> URL: https://issues.apache.org/jira/browse/KYLIN-3885
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
>
> Build dimension dictionary job costs less than 20 minutes when using
> mapreduce fact distinct,but but it costs more than 3 hours when using spark
> fact distinct.
> {code:java}
> "Scheduler 542945608 Job 05c62aca-853f-396e-9653-f20c9ebd8ebc-329" #329
> prio=5 os_prio=0 tid=0x7f312109c800 nid=0x2dc0b in Object.wait()
> [0x7f30d8d24000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.hadoop.ipc.Client.call(Client.java:1482)
> - locked <0x0005c3110fc0> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy33.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:573)
> at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:249)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:107)
> at com.sun.proxy.$Proxy34.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2057)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:682)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:696)
> at
> org.apache.hadoop.fs.FilterFileSystem.delete(FilterFileSystem.java:232)
> at
> org.apache.hadoop.fs.viewfs.ChRootedFileSystem.delete(ChRootedFileSystem.java:198)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.delete(ViewFileSystem.java:334)
> at
> org.apache.hadoop.hdfs.FederatedDFSFileSystem.delete(FederatedDFSFileSystem.java:232)
> at
> org.apache.kylin.dict.global.GlobalDictHDFSStore.deleteSlice(GlobalDictHDFSStore.java:211)
> at
> org.apache.kylin.dict.global.AppendTrieDictionaryBuilder.flushCurrentNode(AppendTrieDictionaryBuilder.java:137)
> at
> org.apache.kylin.dict.global.AppendTrieDictionaryBuilder.addValue(AppendTrieDictionaryBuilder.java:97)
> at
> org.apache.kylin.dict.GlobalDictionaryBuilder.addValue(GlobalDictionaryBuilder.java:85)
> at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:82)
> at
> org.apache.kylin.dict.DictionaryManager.buildDictFromReadableTable(DictionaryManager.java:303)
> at
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:290)
> at
> org.apache.kylin.cube.CubeManager$DictionaryAssist.buildDictionary(CubeManager.java:1043)
> at
> org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:1012)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:72)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:50)
> at
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92)
> at
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
> java.util.concurrent.ThreadPoolExecuto
[jira] [Created] (KYLIN-3885) Build dimension dictionary job costs too long when using Spark fact distinct
Liu Shaohui created KYLIN-3885:
--
Summary: Build dimension dictionary job costs too long when using
Spark fact distinct
Key: KYLIN-3885
URL: https://issues.apache.org/jira/browse/KYLIN-3885
Project: Kylin
Issue Type: Bug
Reporter: Liu Shaohui
Build dimension dictionary job costs less than 20 minutes when using mapreduce
fact distinct,but but it costs more than 3 hours when using spark fact distinct.
{code:java}
"Scheduler 542945608 Job 05c62aca-853f-396e-9653-f20c9ebd8ebc-329" #329 prio=5
os_prio=0 tid=0x7f312109c800 nid=0x2dc0b in Object.wait()
[0x7f30d8d24000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.hadoop.ipc.Client.call(Client.java:1482)
- locked <0x0005c3110fc0> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy33.delete(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:573)
at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:249)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:107)
at com.sun.proxy.$Proxy34.delete(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2057)
at
org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:682)
at
org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:696)
at
org.apache.hadoop.fs.FilterFileSystem.delete(FilterFileSystem.java:232)
at
org.apache.hadoop.fs.viewfs.ChRootedFileSystem.delete(ChRootedFileSystem.java:198)
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.delete(ViewFileSystem.java:334)
at
org.apache.hadoop.hdfs.FederatedDFSFileSystem.delete(FederatedDFSFileSystem.java:232)
at
org.apache.kylin.dict.global.GlobalDictHDFSStore.deleteSlice(GlobalDictHDFSStore.java:211)
at
org.apache.kylin.dict.global.AppendTrieDictionaryBuilder.flushCurrentNode(AppendTrieDictionaryBuilder.java:137)
at
org.apache.kylin.dict.global.AppendTrieDictionaryBuilder.addValue(AppendTrieDictionaryBuilder.java:97)
at
org.apache.kylin.dict.GlobalDictionaryBuilder.addValue(GlobalDictionaryBuilder.java:85)
at
org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:82)
at
org.apache.kylin.dict.DictionaryManager.buildDictFromReadableTable(DictionaryManager.java:303)
at
org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:290)
at
org.apache.kylin.cube.CubeManager$DictionaryAssist.buildDictionary(CubeManager.java:1043)
at
org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:1012)
at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:72)
at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:50)
at
org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92)
at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3884) loading hfile to HBase failed for temporary dir in output path
Liu Shaohui created KYLIN-3884:
--
Summary: loading hfile to HBase failed for temporary dir in
output path
Key: KYLIN-3884
URL: https://issues.apache.org/jira/browse/KYLIN-3884
Project: Kylin
Issue Type: Bug
Reporter: Liu Shaohui
{code:java}
2019-03-14 20:18:46,591 DEBUG [Scheduler 2084224398 Job
e48de76a-6e16-309f-a3a5-191c04071072-131] steps.BulkLoadJob:77 : Start to run
LoadIncrementalHFiles
2019-03-14 20:18:46,642 WARN [Scheduler 2084224398 Job
e48de76a-6e16-309f-a3a5-191c04071072-131] mapreduce.LoadIncrementalHFiles:197 :
Skipping non-directory
hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_SUCCESS
2019-03-14 20:18:46,650 ERROR [Scheduler 2084224398 Job
e48de76a-6e16-309f-a3a5-191c04071072-131] mapreduce.LoadIncrementalHFiles:352 :
-
hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/4170d772384144848c1c10cba66152c3
hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/50ec331ff3c648e3b6e4f54a7b1fe7e9
hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/703ade3b535b4fedab39ee183e22aa7c
hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/82019f8ca00a4f16b9d2b45356a55a3a
hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/8cc8844bced24cb88fda52fecc7224d5
hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/cbac78e0c6d74b5c96a7b64f99e0d0b3
hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/F1/e3844766a4d0486d89f287450034f378
hdfs://zjyprc-xiaomi/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_temporary/0
2019-03-14 20:18:46,651 ERROR [Scheduler 2084224398 Job
e48de76a-6e16-309f-a3a5-191c04071072-131] common.HadoopShellExecutable:65 :
error execute HadoopShellExecutable{id=e48de76a-6e16-309f-a3a5-191c04071072-08,
name=Load HFile to HBase Table, state=RUNNING}
java.io.FileNotFoundException: Path is not a file:
/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_temporary/0
Caused by:
org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path is
not a file:
/user/s_kylin/kylin_zjyprc_bigdata_staging/kylin_zjyprc_bigdata_staging-kylin_metadata/kylin-e48de76a-6e16-309f-a3a5-191c04071072/total_user_cube/hfile/_temporary/0{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3882) kylin master build failed for pom issues
Liu Shaohui created KYLIN-3882:
--
Summary: kylin master build failed for pom issues
Key: KYLIN-3882
URL: https://issues.apache.org/jira/browse/KYLIN-3882
Project: Kylin
Issue Type: Bug
Reporter: Liu Shaohui
As title.
1, Kyligence repo id : nexus conflicts with local maven settings.xml
{code:java}
[ERROR] Failed to execute goal on project kylin-core-metadata: Could not
resolve dependencies for project
org.apache.kylin:kylin-core-metadata:jar:3.0.0-SNAPSHOT: Failure to find
org.apache.calcite:calcite-core:jar:1.16.0-kylin-r2 in
http://nexus.x./nexus/content/groups/public was cached in the local
repository, resolution will not be reattempted until the update interval of
nexus has elapsed or updates are forced -> [Help 1]
{code}
2, maven.compiler.source/target is not set
{code:java}
[INFO] Compiling 2 Scala sources and 18 Java sources to
/ssd/liushaohui/workspace/computing/kylin/engine-spark/target/classes ...
[WARNING] [Warn] : bootstrap class path not set in conjunction with -source 1.6
[ERROR] [Error]
/ssd/liushaohui/workspace/computing/kylin/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkBatchCubingJobBuilder2.java:148:
diamond operator is not supported in -source 1.6
(use -source 7 or higher to enable diamond operator)
[ERROR] [Error]
/ssd/liushaohui/workspace/computing/kylin/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkCubingByLayer.java:239:
try-with-resources is not supported in -source 1.6
(use -source 7 or higher to enable try-with-resources)
[ERROR] [Error]
/ssd/liushaohui/workspace/computing/kylin/engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkCubingByLayer.java:251:
diamond operator is not supported in -source 1.6
(use -source 7 or higher to enable diamond operator){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3880) DataType is incompatible in Kylin HBase coprocessor
Liu Shaohui created KYLIN-3880:
--
Summary: DataType is incompatible in Kylin HBase coprocessor
Key: KYLIN-3880
URL: https://issues.apache.org/jira/browse/KYLIN-3880
Project: Kylin
Issue Type: Bug
Reporter: Liu Shaohui
During upgrade kylin from 2.4.1 to 2.5.2, the query will failed for the
incompatible class in Kylin HBase coprocessor
{code:java}
2019-03-12,17:48:11,530 INFO
[FifoRWQ.default.readRpcServer.handler=197,queue=13,port=24600]
org.apache.hadoop.hdfs.DFSClient: Access token was invalid when connecting to
/10.152.33.45:22402 :
org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException: Got
access token error for OP_READ_BLOCK, self=/10.152.33.44:55387,
remote=/10.152.33.45:22402, for file
/hbase/zjyprc-xiaomi/data/miui_sec/data/4b88a72f5bd37daca00efb842e676ca8/C/6593503eb213431998db117cf3dab3a6,
for pool BP-792581576-10.152.48.22-1510572454905 block 1899006034_825272806
2019-03-12,17:48:12,135 INFO
[FifoRWQ.default.readRpcServer.handler=231,queue=15,port=24600]
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService:
start query dc0fadcf-3689-5508-9a45-559aaebfd4e0 in thread
FifoRWQ.default.readRpcServer.handler=231,queue=15,port=24600
2019-03-12,17:48:12,135 ERROR
[FifoRWQ.default.readRpcServer.handler=231,queue=15,port=24600]
org.apache.hadoop.ipc.RpcServer: Unexpected throwable object
java.lang.RuntimeException: java.io.InvalidClassException:
org.apache.kylin.metadata.datatype.DataType; local class incompatible: stream
classdesc serialVersionUID = -8891652700267537109, local class serialVersionUID
= -406124487097947
at
org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem.readDimensionEncoding(TrimmedCubeCodeSystem.java:87)
at
org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem$1.deserialize(TrimmedCubeCodeSystem.java:122)
at
org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem$1.deserialize(TrimmedCubeCodeSystem.java:91)
at org.apache.kylin.gridtable.GTInfo$1.deserialize(GTInfo.java:346)
at org.apache.kylin.gridtable.GTInfo$1.deserialize(GTInfo.java:307)
at
org.apache.kylin.gridtable.GTScanRequest$2.deserialize(GTScanRequest.java:466)
at
org.apache.kylin.gridtable.GTScanRequest$2.deserialize(GTScanRequest.java:412)
at
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259)
at
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6625)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:4336)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:4318)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34964)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2059)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:126)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:152)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:128)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.InvalidClassException:
org.apache.kylin.metadata.datatype.DataType; local class incompatible: stream
classdesc serialVersionUID = -8891652700267537109, local class serialVersionUID
= -406124487097947
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at
org.apache.kylin.dimension.AbstractDateDimEnc.readExternal(AbstractDateDimEnc.java:137)
at
java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2118)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at
org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem.
[jira] [Commented] (KYLIN-3817) Duration in Cube building is a negative number
[ https://issues.apache.org/jira/browse/KYLIN-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16793699#comment-16793699 ] Liu Shaohui commented on KYLIN-3817: Sorry for this bug which is introduced in KYLIN-3780.I will fix later. > Duration in Cube building is a negative number > -- > > Key: KYLIN-3817 > URL: https://issues.apache.org/jira/browse/KYLIN-3817 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.6.0 >Reporter: Na Zhai >Assignee: Na Zhai >Priority: Major > Fix For: v2.6.2 > > Attachments: Duration error.png > > > After building the cube for a few minutes, click the Refresh button and the > duration of the job becomes negative. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3809) Support Zookeeper based rest server discovery
[ https://issues.apache.org/jira/browse/KYLIN-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui reassigned KYLIN-3809: -- Assignee: Liu Shaohui > Support Zookeeper based rest server discovery > - > > Key: KYLIN-3809 > URL: https://issues.apache.org/jira/browse/KYLIN-3809 > Project: Kylin > Issue Type: New Feature >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > > Currently to broadcast config or meta changes, all kylin servers must be set > in kylin.properties. It's not convenient when adding or removing kylin server > especially in k8s env. > > So we can register the endpoint to zk and make the rest server discovery > automatically. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3809) Support Zookeeper based rest server discovery
Liu Shaohui created KYLIN-3809: -- Summary: Support Zookeeper based rest server discovery Key: KYLIN-3809 URL: https://issues.apache.org/jira/browse/KYLIN-3809 Project: Kylin Issue Type: New Feature Reporter: Liu Shaohui Currently to broadcast config or meta changes, all kylin servers must be set in kylin.properties. It's not convenient when adding or removing kylin server especially in k8s env. So we can register the endpoint to zk and make the rest server discovery automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3780) Add built instance in Job info
Liu Shaohui created KYLIN-3780: -- Summary: Add built instance in Job info Key: KYLIN-3780 URL: https://issues.apache.org/jira/browse/KYLIN-3780 Project: Kylin Issue Type: New Feature Reporter: Liu Shaohui Assignee: Liu Shaohui In DistributedScheduler, it's hard to known which machine the kylin job is running on. But this info is helpful to debug the failed jobs. So we add the job built instance info in job info and kylin web ui. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3494) Build cube with spark reports ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/KYLIN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748388#comment-16748388 ] Liu Shaohui commented on KYLIN-3494: [~Shaofengshi] Please help to review the pull request: [https://github.com/apache/kylin/pull/441/.|https://github.com/apache/kylin/pull/441/] Thanks~ > Build cube with spark reports ArrayIndexOutOfBoundsException > - > > Key: KYLIN-3494 > URL: https://issues.apache.org/jira/browse/KYLIN-3494 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.4.0 >Reporter: jilai liu >Assignee: Liu Shaohui >Priority: Major > Labels: patch > Fix For: v2.5.3, v2.6.1 > > > Logged in as: dr.who > Application > About > Jobs > Tools > Log Type: stderr > Log Upload Time: Mon Aug 13 15:50:10 +0800 2018 > Log Length: 74544 > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/data3/test/data/hadoop/hdfs/data/usercache/hadoop/filecache/17809/__spark_libs__6649521663189541594.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/data1/test/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 18/08/13 15:49:38 INFO util.SignalUtils: Registered signal handler for TERM > 18/08/13 15:49:38 INFO util.SignalUtils: Registered signal handler for HUP > 18/08/13 15:49:38 INFO util.SignalUtils: Registered signal handler for INT > 18/08/13 15:49:38 INFO yarn.ApplicationMaster: Preparing Local resources > 18/08/13 15:49:39 INFO yarn.ApplicationMaster: ApplicationAttemptId: > appattempt_1533616206085_5657_01 > 18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls to: hadoop > 18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls to: hadoop > 18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls groups to: > 18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls groups to: > 18/08/13 15:49:39 INFO spark.SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(hadoop); groups > with view permissions: Set(); users with modify permissions: Set(hadoop); > groups with modify permissions: Set() > 18/08/13 15:49:39 INFO yarn.ApplicationMaster: Starting the user application > in a separate Thread > 18/08/13 15:49:39 INFO yarn.ApplicationMaster: Waiting for spark context > initialization... > 18/08/13 15:49:39 INFO spark.SparkContext: Running Spark version 2.1.2 > 18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls to: hadoop > 18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls to: hadoop > 18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls groups to: > 18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls groups to: > 18/08/13 15:49:39 INFO spark.SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(hadoop); groups > with view permissions: Set(); users with modify permissions: Set(hadoop); > groups with modify permissions: Set() > 18/08/13 15:49:40 INFO util.Utils: Successfully started service 'sparkDriver' > on port 40358. > 18/08/13 15:49:40 INFO spark.SparkEnv: Registering MapOutputTracker > 18/08/13 15:49:40 INFO spark.SparkEnv: Registering BlockManagerMaster > 18/08/13 15:49:40 INFO storage.BlockManagerMasterEndpoint: Using > org.apache.spark.storage.DefaultTopologyMapper for getting topology > information > 18/08/13 15:49:40 INFO storage.BlockManagerMasterEndpoint: > BlockManagerMasterEndpoint up > 18/08/13 15:49:40 INFO storage.DiskBlockManager: Created local directory at > /data1/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/blockmgr-4cd0fed9-78ae-4e2c-826d-b42a8d6364d2 > 18/08/13 15:49:40 INFO storage.DiskBlockManager: Created local directory at > /data2/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/blockmgr-8cca22e9-ece0-469b-b7fa-3cd9567504d9 > 18/08/13 15:49:40 INFO storage.DiskBlockManager: Created local directory at > /data3/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/blockmgr-15f5073d-5ea2-4766-ab22-b8c68834fb80 > 18/08/13 15:49:40 INFO memory.MemoryStore: MemoryStore started with capacity > 305.3 MB > 18/08/13 15:49:40 INFO spark.SparkEnv: Registering OutputCommitCoordinator > 18/08/13 15:49:40 INFO util.log: Logging initialized @2958ms > 18/08/13 15:49:40 INFO ui.JettyUtils: Adding filter: > org.apache.hadoop.yarn.s
[jira] [Commented] (KYLIN-3494) build cube with spark ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/KYLIN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16747895#comment-16747895
]
Liu Shaohui commented on KYLIN-3494:
[~Shaofengshi] this issue is introduced in commit:
{code:java}
commit 0fdf55abb2c709e04bf327d15299d695dce7b219
Author: shaofengshi
Date: Mon Jun 4 14:59:55 2018 +0800
KYLIN-3137 Spark cubing without hive
{code}
After the commit, kylin will use sc.sequenceFile to read sequence file and
split the field using string splitting. It will lose null column fields and
cause ArrayIndexOutOfBoundsException in BaseCuboidBuilder.
After rolling back to code to use sparkSession.table to read hive table, this
problem disappeared.
> build cube with spark ArrayIndexOutOfBoundsException
> -
>
> Key: KYLIN-3494
> URL: https://issues.apache.org/jira/browse/KYLIN-3494
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.4.0
>Reporter: jilai liu
>Assignee: Liu Shaohui
>Priority: Major
> Labels: patch
>
> Logged in as: dr.who
> Application
> About
> Jobs
> Tools
> Log Type: stderr
> Log Upload Time: Mon Aug 13 15:50:10 +0800 2018
> Log Length: 74544
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/data3/test/data/hadoop/hdfs/data/usercache/hadoop/filecache/17809/__spark_libs__6649521663189541594.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/data1/test/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 18/08/13 15:49:38 INFO util.SignalUtils: Registered signal handler for TERM
> 18/08/13 15:49:38 INFO util.SignalUtils: Registered signal handler for HUP
> 18/08/13 15:49:38 INFO util.SignalUtils: Registered signal handler for INT
> 18/08/13 15:49:38 INFO yarn.ApplicationMaster: Preparing Local resources
> 18/08/13 15:49:39 INFO yarn.ApplicationMaster: ApplicationAttemptId:
> appattempt_1533616206085_5657_01
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls to: hadoop
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls to: hadoop
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls groups to:
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls groups to:
> 18/08/13 15:49:39 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(hadoop); groups
> with view permissions: Set(); users with modify permissions: Set(hadoop);
> groups with modify permissions: Set()
> 18/08/13 15:49:39 INFO yarn.ApplicationMaster: Starting the user application
> in a separate Thread
> 18/08/13 15:49:39 INFO yarn.ApplicationMaster: Waiting for spark context
> initialization...
> 18/08/13 15:49:39 INFO spark.SparkContext: Running Spark version 2.1.2
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls to: hadoop
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls to: hadoop
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls groups to:
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls groups to:
> 18/08/13 15:49:39 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(hadoop); groups
> with view permissions: Set(); users with modify permissions: Set(hadoop);
> groups with modify permissions: Set()
> 18/08/13 15:49:40 INFO util.Utils: Successfully started service 'sparkDriver'
> on port 40358.
> 18/08/13 15:49:40 INFO spark.SparkEnv: Registering MapOutputTracker
> 18/08/13 15:49:40 INFO spark.SparkEnv: Registering BlockManagerMaster
> 18/08/13 15:49:40 INFO storage.BlockManagerMasterEndpoint: Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> 18/08/13 15:49:40 INFO storage.BlockManagerMasterEndpoint:
> BlockManagerMasterEndpoint up
> 18/08/13 15:49:40 INFO storage.DiskBlockManager: Created local directory at
> /data1/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/blockmgr-4cd0fed9-78ae-4e2c-826d-b42a8d6364d2
> 18/08/13 15:49:40 INFO storage.DiskBlockManager: Created local directory at
> /data2/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/blockmgr-8cca22e9-ece0-469b-b7fa-3cd9567504d9
> 18/08/13 15:49:40 INFO storage.DiskBlockManager: Created local directory at
> /data3/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/blockmgr-15f5073d-5ea2-4766-ab22-b8c68
[jira] [Assigned] (KYLIN-3494) build cube with spark ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/KYLIN-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui reassigned KYLIN-3494:
--
Assignee: Liu Shaohui
> build cube with spark ArrayIndexOutOfBoundsException
> -
>
> Key: KYLIN-3494
> URL: https://issues.apache.org/jira/browse/KYLIN-3494
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.4.0
>Reporter: jilai liu
>Assignee: Liu Shaohui
>Priority: Major
> Labels: patch
>
> Logged in as: dr.who
> Application
> About
> Jobs
> Tools
> Log Type: stderr
> Log Upload Time: Mon Aug 13 15:50:10 +0800 2018
> Log Length: 74544
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/data3/test/data/hadoop/hdfs/data/usercache/hadoop/filecache/17809/__spark_libs__6649521663189541594.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/data1/test/soft/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 18/08/13 15:49:38 INFO util.SignalUtils: Registered signal handler for TERM
> 18/08/13 15:49:38 INFO util.SignalUtils: Registered signal handler for HUP
> 18/08/13 15:49:38 INFO util.SignalUtils: Registered signal handler for INT
> 18/08/13 15:49:38 INFO yarn.ApplicationMaster: Preparing Local resources
> 18/08/13 15:49:39 INFO yarn.ApplicationMaster: ApplicationAttemptId:
> appattempt_1533616206085_5657_01
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls to: hadoop
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls to: hadoop
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls groups to:
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls groups to:
> 18/08/13 15:49:39 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(hadoop); groups
> with view permissions: Set(); users with modify permissions: Set(hadoop);
> groups with modify permissions: Set()
> 18/08/13 15:49:39 INFO yarn.ApplicationMaster: Starting the user application
> in a separate Thread
> 18/08/13 15:49:39 INFO yarn.ApplicationMaster: Waiting for spark context
> initialization...
> 18/08/13 15:49:39 INFO spark.SparkContext: Running Spark version 2.1.2
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls to: hadoop
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls to: hadoop
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing view acls groups to:
> 18/08/13 15:49:39 INFO spark.SecurityManager: Changing modify acls groups to:
> 18/08/13 15:49:39 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(hadoop); groups
> with view permissions: Set(); users with modify permissions: Set(hadoop);
> groups with modify permissions: Set()
> 18/08/13 15:49:40 INFO util.Utils: Successfully started service 'sparkDriver'
> on port 40358.
> 18/08/13 15:49:40 INFO spark.SparkEnv: Registering MapOutputTracker
> 18/08/13 15:49:40 INFO spark.SparkEnv: Registering BlockManagerMaster
> 18/08/13 15:49:40 INFO storage.BlockManagerMasterEndpoint: Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> 18/08/13 15:49:40 INFO storage.BlockManagerMasterEndpoint:
> BlockManagerMasterEndpoint up
> 18/08/13 15:49:40 INFO storage.DiskBlockManager: Created local directory at
> /data1/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/blockmgr-4cd0fed9-78ae-4e2c-826d-b42a8d6364d2
> 18/08/13 15:49:40 INFO storage.DiskBlockManager: Created local directory at
> /data2/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/blockmgr-8cca22e9-ece0-469b-b7fa-3cd9567504d9
> 18/08/13 15:49:40 INFO storage.DiskBlockManager: Created local directory at
> /data3/test/data/hadoop/hdfs/data/usercache/hadoop/appcache/application_1533616206085_5657/blockmgr-15f5073d-5ea2-4766-ab22-b8c68834fb80
> 18/08/13 15:49:40 INFO memory.MemoryStore: MemoryStore started with capacity
> 305.3 MB
> 18/08/13 15:49:40 INFO spark.SparkEnv: Registering OutputCommitCoordinator
> 18/08/13 15:49:40 INFO util.log: Logging initialized @2958ms
> 18/08/13 15:49:40 INFO ui.JettyUtils: Adding filter:
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
> 18/08/13 15:49:40 INFO server.Server: jetty-9.2.z-SNAPSHOT
> 18/08/13 15:49:40 INFO handler.ContextHandler: Started
> o.s.j.s.ServletContextHandler@2506206a\{/jobs,null,AVAILABLE,@Spark}
> 18/08/13
[jira] [Commented] (KYLIN-3613) Kylin with Standalone HBase Cluster (enabled kerberos) could not find the main cluster namespace at "Create HTable" step
[ https://issues.apache.org/jira/browse/KYLIN-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744135#comment-16744135 ] Liu Shaohui commented on KYLIN-3613: + 1 for encountering the same problem. Thanks for fixing it~ > Kylin with Standalone HBase Cluster (enabled kerberos) could not find the > main cluster namespace at "Create HTable" step > - > > Key: KYLIN-3613 > URL: https://issues.apache.org/jira/browse/KYLIN-3613 > Project: Kylin > Issue Type: Bug > Components: Environment >Affects Versions: v2.4.0, v2.4.1, v2.5.0 >Reporter: powerinf >Assignee: WangSheng >Priority: Major > Fix For: v2.4.2, v2.5.1 > > Attachments: KYLIN-3613.patch, Kylin_standalone_hbase.log > > > I deployed two hadoop cluster(also enabled kerberos ,with cross-realm trust) > the main cluster and hbase cluster,Kylin Server can access both clusters > using hdfs shell with fully qualifiered path ,can submit MR job to main > cluster, and can use hive shell to access data warehouse > on Kylin Server, the configurations of hadoop and hive points to main > cluster,and can access hbase cluster using hbase shell. > when I build the cube, at "Create HTable" step, it reported the error > "java.net.UnknownHostException: ctyunbigdata Set hbase.table.sanity.checks to > false > at conf or table descriptor if you want to bypass sanity checks",but after I > restart Kylin serer , resume it can run normally,Why? > more detail message on Kylin_standalone_hbase.log -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3726) KylinSession should load spark properties from spark-defaults.conf
[
https://issues.apache.org/jira/browse/KYLIN-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui reassigned KYLIN-3726:
--
Assignee: Liu Shaohui
> KylinSession should load spark properties from spark-defaults.conf
> --
>
> Key: KYLIN-3726
> URL: https://issues.apache.org/jira/browse/KYLIN-3726
> Project: Kylin
> Issue Type: Sub-task
> Components: Storage - Parquet
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
>
> When testing parquet storage, the spark session job failed to be submit for
> no JAVA_HOME in executor env. This config is set in the spark default
> property file: spark-defaults.conf.
>
> {code}
> 2018-12-18,15:13:15,466 ERROR org.apache.spark.deploy.yarn.YarnAllocator:
> Failed to launch executor 6 on container
> container_e823_1541646991414_1025309_01_07
> java.util.NoSuchElementException: key not found: JAVA_HOME at
> scala.collection.MapLike$class.default(MapLike.scala:228) at
> scala.collection.AbstractMap.default(Map.scala:59) at
> scala.collection.mutable.HashMap.apply(HashMap.scala:65) at
> org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3$$anonfun$apply$3.apply(ExecutorRunnable.scala:286)
> at
> org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3$$anonfun$apply$3.apply(ExecutorRunnable.scala:275)
> at scala.Option.foreach(Option.scala:257) at
> org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3.apply(ExecutorRunnable.scala:275)
> at
> org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3.apply(ExecutorRunnable.scala:274)
> at scala.Option.foreach(Option.scala:257) at
> org.apache.spark.deploy.yarn.ExecutorRunnable.prepareEnvironment(ExecutorRunnable.scala:274)
> at
> org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:92)
> at
> org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:69)
> at
> org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$runAllocatedContainers$1$$anon$1.run(YarnAllocator.scala:556)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3726) KylinSession should load spark properties from spark-defaults.conf
Liu Shaohui created KYLIN-3726:
--
Summary: KylinSession should load spark properties from
spark-defaults.conf
Key: KYLIN-3726
URL: https://issues.apache.org/jira/browse/KYLIN-3726
Project: Kylin
Issue Type: Sub-task
Components: Storage - Parquet
Reporter: Liu Shaohui
When testing parquet storage, the spark session job failed to be submit for no
JAVA_HOME in executor env. This config is set in the spark default property
file: spark-defaults.conf.
{code}
2018-12-18,15:13:15,466 ERROR org.apache.spark.deploy.yarn.YarnAllocator:
Failed to launch executor 6 on container
container_e823_1541646991414_1025309_01_07
java.util.NoSuchElementException: key not found: JAVA_HOME at
scala.collection.MapLike$class.default(MapLike.scala:228) at
scala.collection.AbstractMap.default(Map.scala:59) at
scala.collection.mutable.HashMap.apply(HashMap.scala:65) at
org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3$$anonfun$apply$3.apply(ExecutorRunnable.scala:286)
at
org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3$$anonfun$apply$3.apply(ExecutorRunnable.scala:275)
at scala.Option.foreach(Option.scala:257) at
org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3.apply(ExecutorRunnable.scala:275)
at
org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3.apply(ExecutorRunnable.scala:274)
at scala.Option.foreach(Option.scala:257) at
org.apache.spark.deploy.yarn.ExecutorRunnable.prepareEnvironment(ExecutorRunnable.scala:274)
at
org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:92)
at
org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:69) at
org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$runAllocatedContainers$1$$anon$1.run(YarnAllocator.scala:556)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3495) Wrong datatype when using on max function on a empty double column
[
https://issues.apache.org/jira/browse/KYLIN-3495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-3495:
---
Summary: Wrong datatype when using on max function on a empty double column
(was: Wrong datatypy when using on max function on a empty double column)
> Wrong datatype when using on max function on a empty double column
> --
>
> Key: KYLIN-3495
> URL: https://issues.apache.org/jira/browse/KYLIN-3495
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Priority: Major
>
> SQL:
> {code:java}
> select
> count(*),sum(PER_BYTES_TIME_COST)/count(PER_BYTES_TIME_COST),max(PER_BYTES_TIME_COST),min(PER_BYTES_TIME_COST)
> from KYLIN_ONEBOX.HIVE_METRICS_JOB_DEV where KDAY_DATE >= '2018-07-01' and
> KDAY_DATE <= '2018-07-31' and PROJECT ='LEARN_KYLIN'{code}
>
> Exception:
> {code:java}
> NoSuchMethodException: SqlFunctions.greater(java.math.BigDecimal, double)
> while resolving method 'greater[class java.math.BigDecimal, double]' in class
> class org.apache.calcite.runtime.SqlFunctions
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218)
> at
> org.apache.kylin.rest.service.QueryService.execute(QueryService.java:940)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:670)
> at
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:188)
> at
> org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:505)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:464)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
> at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3495) Wrong datatypy when using on max function on a empty double column
[
https://issues.apache.org/jira/browse/KYLIN-3495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-3495:
---
Summary: Wrong datatypy when using on max function on a empty double column
(was: Wrong datatypo when using on max function on a empty column)
> Wrong datatypy when using on max function on a empty double column
> --
>
> Key: KYLIN-3495
> URL: https://issues.apache.org/jira/browse/KYLIN-3495
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Priority: Major
>
> SQL:
> {code:java}
> select
> count(*),sum(PER_BYTES_TIME_COST)/count(PER_BYTES_TIME_COST),max(PER_BYTES_TIME_COST),min(PER_BYTES_TIME_COST)
> from KYLIN_ONEBOX.HIVE_METRICS_JOB_DEV where KDAY_DATE >= '2018-07-01' and
> KDAY_DATE <= '2018-07-31' and PROJECT ='LEARN_KYLIN'{code}
>
> Exception:
> {code:java}
> NoSuchMethodException: SqlFunctions.greater(java.math.BigDecimal, double)
> while resolving method 'greater[class java.math.BigDecimal, double]' in class
> class org.apache.calcite.runtime.SqlFunctions
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218)
> at
> org.apache.kylin.rest.service.QueryService.execute(QueryService.java:940)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:670)
> at
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:188)
> at
> org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:505)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:464)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
> at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3495) Wrong datatypo when using on max function on a empty column
Liu Shaohui created KYLIN-3495:
--
Summary: Wrong datatypo when using on max function on a empty
column
Key: KYLIN-3495
URL: https://issues.apache.org/jira/browse/KYLIN-3495
Project: Kylin
Issue Type: Bug
Reporter: Liu Shaohui
SQL:
{code:java}
select
count(*),sum(PER_BYTES_TIME_COST)/count(PER_BYTES_TIME_COST),max(PER_BYTES_TIME_COST),min(PER_BYTES_TIME_COST)
from KYLIN_ONEBOX.HIVE_METRICS_JOB_DEV where KDAY_DATE >= '2018-07-01' and
KDAY_DATE <= '2018-07-31' and PROJECT ='LEARN_KYLIN'{code}
Exception:
{code:java}
NoSuchMethodException: SqlFunctions.greater(java.math.BigDecimal, double)
while resolving method 'greater[class java.math.BigDecimal, double]' in class
class org.apache.calcite.runtime.SqlFunctions
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
at
org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218)
at
org.apache.kylin.rest.service.QueryService.execute(QueryService.java:940)
at
org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:670)
at
org.apache.kylin.rest.service.QueryService.query(QueryService.java:188)
at
org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:505)
at
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:464)
at
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390)
at
org.apache.kylin.rest.controller.QueryController.query(QueryController.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3357) Sum of small int measure may be nagetive after KYLIN-2982
[ https://issues.apache.org/jira/browse/KYLIN-3357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16577768#comment-16577768 ] Liu Shaohui commented on KYLIN-3357: I don't think this workrounds is a good solution. It's hard to push all users to change data type. And in my opinion, kylin should support sum of small integer. > Sum of small int measure may be nagetive after KYLIN-2982 > - > > Key: KYLIN-3357 > URL: https://issues.apache.org/jira/browse/KYLIN-3357 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0 >Reporter: Liu Shaohui >Priority: Minor > > After KYLIN-2982, the sum of small int measure may be nagetive. > Same problem is reported in kylin user mail with title "negative result in > kylin 2.3.0" > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3357) Sum of small int measure may be nagetive after KYLIN-2982
Liu Shaohui created KYLIN-3357: -- Summary: Sum of small int measure may be nagetive after KYLIN-2982 Key: KYLIN-3357 URL: https://issues.apache.org/jira/browse/KYLIN-3357 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: v2.3.0 Reporter: Liu Shaohui After KYLIN-2982, the sum of small int measure may be nagetive. Same problem is reported in kylin user mail with title "negative result in kylin 2.3.0" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3156) Failed to delete meta path in SparkCubingByLayer
[
https://issues.apache.org/jira/browse/KYLIN-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-3156:
---
Component/s: Spark Engine
> Failed to delete meta path in SparkCubingByLayer
>
>
> Key: KYLIN-3156
> URL: https://issues.apache.org/jira/browse/KYLIN-3156
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
>Reporter: Liu Shaohui
>
> After KYLIN-2945, the meta url in SparkCubingByLayer will be a string of
> StorageURL not string with path@hdfs format. This will make the
> deleteHDFSMeta method failed in SparkCubingByLayer.
> {quote}
> 2018-01-08,11:51:50,903 ERROR org.apache.spark.deploy.yarn.ApplicationMaster:
> User class threw exception: java.lang.RuntimeException: error execute
> org.apache.kylin.engine.spark.SparkCubingByLayer
> java.lang.RuntimeException: error execute
> org.apache.kylin.engine.spark.SparkCubingByLayer
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:653)
> Caused by: java.lang.IllegalArgumentException: Cannot create FileSystem from
> URI: kylin_tst:kylin_metadata
> at org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:98)
> at
> org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:78)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer.deleteHDFSMeta(SparkCubingByLayer.java:484)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:207)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> ... 6 more
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 5: kylin_tst:kylin_metadata
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parse(URI.java:3048)
> at java.net.URI.(URI.java:588)
> at org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:96)
> ... 10 more
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KYLIN-3156) Failed to delete meta path in SparkCubingByLayer
Liu Shaohui created KYLIN-3156:
--
Summary: Failed to delete meta path in SparkCubingByLayer
Key: KYLIN-3156
URL: https://issues.apache.org/jira/browse/KYLIN-3156
Project: Kylin
Issue Type: Bug
Reporter: Liu Shaohui
After KYLIN-2945, the meta url in SparkCubingByLayer will be a string of
StorageURL not string with path@hdfs format. This will make the deleteHDFSMeta
method failed in SparkCubingByLayer.
{quote}
2018-01-08,11:51:50,903 ERROR org.apache.spark.deploy.yarn.ApplicationMaster:
User class threw exception: java.lang.RuntimeException: error execute
org.apache.kylin.engine.spark.SparkCubingByLayer
java.lang.RuntimeException: error execute
org.apache.kylin.engine.spark.SparkCubingByLayer
at
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:653)
Caused by: java.lang.IllegalArgumentException: Cannot create FileSystem from
URI: kylin_tst:kylin_metadata
at org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:98)
at
org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:78)
at
org.apache.kylin.engine.spark.SparkCubingByLayer.deleteHDFSMeta(SparkCubingByLayer.java:484)
at
org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:207)
at
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
... 6 more
Caused by: java.net.URISyntaxException: Illegal character in scheme name at
index 5: kylin_tst:kylin_metadata
at java.net.URI$Parser.fail(URI.java:2848)
at java.net.URI$Parser.checkChars(URI.java:3021)
at java.net.URI$Parser.parse(URI.java:3048)
at java.net.URI.(URI.java:588)
at org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:96)
... 10 more
{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2915) Exception during query on lookup table
[
https://issues.apache.org/jira/browse/KYLIN-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16235060#comment-16235060
]
Liu Shaohui commented on KYLIN-2915:
[~liyang.g...@gmail.com] In which commit this issue is fixed? Thx
> Exception during query on lookup table
> --
>
> Key: KYLIN-2915
> URL: https://issues.apache.org/jira/browse/KYLIN-2915
> Project: Kylin
> Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
> Fix For: v2.2.0
>
>
> {noformat}
> 2017-07-18 10:52:49,744 ERROR [Query
> 49c9601f-017f-44b7-b166-6991527f8903-2192] service.QueryService : Exception
> when execute sql
> java.sql.SQLException: Error while executing SQL "select
> "BUYER_ACCOUNT"."ACCOUNT_BUYER_LEVEL" as "c0"
> from
> "DEFAULT"."KYLIN_ACCOUNT" as "BUYER_ACCOUNT"
> group by
> "BUYER_ACCOUNT"."ACCOUNT_BUYER_LEVEL"
> order by
> "BUYER_ACCOUNT"."ACCOUNT_BUYER_LEVEL" ASC
> LIMIT 5": java.lang.Integer cannot be cast to java.lang.Long
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218)
> at
> org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
> at
> org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
> at
> org.apache.kylin.rest.service.QueryService.execute(QueryService.java:769)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:506)
> at
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:177)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:387)
> at
> org.apache.kylin.rest.controller2.QueryControllerV2.queryV2(QueryControllerV2.java:81)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:832)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:743)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
> at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:869)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
> at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at io.kyligence.kap.rest.spring.DoNothingFilter.doFilter(SourceFile:42)
> at
> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
> at
> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.Applic
[jira] [Issue Comment Deleted] (KYLIN-2846) Add a config of hbase namespace for cube storage
[
https://issues.apache.org/jira/browse/KYLIN-2846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated KYLIN-2846:
---
Comment: was deleted
(was: [~yaho]
{quote}
HTable htable = new HTable(hbaseConf,
getOptionValue(OPTION_HTABLE_NAME).toUpperCase());
{quote}
Fixed in KYLIN-2846-001-addium.patch.
{quote}
StorageCleanJobHbaseUtil
{quote}
Thanks for your reminding. I will fix it later.
)
> Add a config of hbase namespace for cube storage
>
>
> Key: KYLIN-2846
> URL: https://issues.apache.org/jira/browse/KYLIN-2846
> Project: Kylin
> Issue Type: New Feature
> Components: Storage - HBase
>Affects Versions: v2.1.0
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Labels: scope
> Fix For: v2.2.0
>
> Attachments: KYLIN-2846-001-addium.patch, KYLIN-2846-001.patch
>
>
> In multi-tenancy HBase cluster, namespace is important for quota management
> and permission control. So we add a global configuration of hbase namespace
> for cube storage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2846) Add a config of hbase namespace for cube storage
[
https://issues.apache.org/jira/browse/KYLIN-2846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169822#comment-16169822
]
Liu Shaohui commented on KYLIN-2846:
[~yaho]
{quote}
HTable htable = new HTable(hbaseConf,
getOptionValue(OPTION_HTABLE_NAME).toUpperCase());
{quote}
Fixed in KYLIN-2846-001-addium.patch.
{quote}
StorageCleanJobHbaseUtil
{quote}
Thanks for your reminding. I will fix it later.
> Add a config of hbase namespace for cube storage
>
>
> Key: KYLIN-2846
> URL: https://issues.apache.org/jira/browse/KYLIN-2846
> Project: Kylin
> Issue Type: New Feature
> Components: Storage - HBase
>Affects Versions: v2.1.0
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Labels: scope
> Fix For: v2.2.0
>
> Attachments: KYLIN-2846-001-addium.patch, KYLIN-2846-001.patch
>
>
> In multi-tenancy HBase cluster, namespace is important for quota management
> and permission control. So we add a global configuration of hbase namespace
> for cube storage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2846) Add a config of hbase namespace for cube storage
[
https://issues.apache.org/jira/browse/KYLIN-2846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169821#comment-16169821
]
Liu Shaohui commented on KYLIN-2846:
[~yaho]
{quote}
HTable htable = new HTable(hbaseConf,
getOptionValue(OPTION_HTABLE_NAME).toUpperCase());
{quote}
Fixed in KYLIN-2846-001-addium.patch.
{quote}
StorageCleanJobHbaseUtil
{quote}
Thanks for your reminding. I will fix it later.
> Add a config of hbase namespace for cube storage
>
>
> Key: KYLIN-2846
> URL: https://issues.apache.org/jira/browse/KYLIN-2846
> Project: Kylin
> Issue Type: New Feature
> Components: Storage - HBase
>Affects Versions: v2.1.0
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Labels: scope
> Fix For: v2.2.0
>
> Attachments: KYLIN-2846-001-addium.patch, KYLIN-2846-001.patch
>
>
> In multi-tenancy HBase cluster, namespace is important for quota management
> and permission control. So we add a global configuration of hbase namespace
> for cube storage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2846) Add a config of hbase namespace for cube storage
[ https://issues.apache.org/jira/browse/KYLIN-2846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated KYLIN-2846: --- Attachment: KYLIN-2846-001-addium.patch Addium for patch 001. There is a "toUpperCase" missed. > Add a config of hbase namespace for cube storage > > > Key: KYLIN-2846 > URL: https://issues.apache.org/jira/browse/KYLIN-2846 > Project: Kylin > Issue Type: New Feature > Components: Storage - HBase >Affects Versions: v2.1.0 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Fix For: v2.2.0 > > Attachments: KYLIN-2846-001-addium.patch, KYLIN-2846-001.patch > > > In multi-tenancy HBase cluster, namespace is important for quota management > and permission control. So we add a global configuration of hbase namespace > for cube storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
