[jira] [Commented] (KYLIN-1082) Hive dependencies should be add to tmpjars
[ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114875#comment-15114875 ] Zhong Yanghong commented on KYLIN-1082: --- For the first one, when convert cuboid to hfile, it may use hbase related jars. But I haven't tested it. Later I will tell you whether it's needed or not. For the second, I think I don't delete the replacement. > Hive dependencies should be add to tmpjars > -- > > Key: KYLIN-1082 > URL: https://issues.apache.org/jira/browse/KYLIN-1082 > Project: Kylin > Issue Type: Bug > Components: Environment , Job Engine >Reporter: liyang >Assignee: Zhong Yanghong > Labels: newbie > Fix For: v2.1, v1.3 > > Attachments: auto_hive_tmpjars_1_x_staging.patch, > auto_hive_tmpjars_2_x_staging.patch > > > Currently kylin assume all data nodes have hive deployment at exact same FS > location. However, a better position is to think hive as a client side app. > Then we need to ship hive jar with MR job every time. > This make deploy kylin a lot easier in cluster that does not have hive on all > data nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1366) Bind metadata version with release version
[ https://issues.apache.org/jira/browse/KYLIN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongbin ma updated KYLIN-1366: -- Affects Version/s: all > Bind metadata version with release version > -- > > Key: KYLIN-1366 > URL: https://issues.apache.org/jira/browse/KYLIN-1366 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Affects Versions: all >Reporter: hongbin ma >Assignee: hongbin ma > Labels: newbie > Fix For: v2.1 > > > Currently metadata does not have special property to identify its version, > even though we already versions of metadata in 1.x, 2.0 and 2.x. If each file > in the metadata is bound with version it will be easier for upgrading related > issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1366) Bind metadata version with release version
[ https://issues.apache.org/jira/browse/KYLIN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongbin ma updated KYLIN-1366: -- Fix Version/s: v2.1 > Bind metadata version with release version > -- > > Key: KYLIN-1366 > URL: https://issues.apache.org/jira/browse/KYLIN-1366 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Affects Versions: all >Reporter: hongbin ma >Assignee: hongbin ma > Labels: newbie > Fix For: v2.1 > > > Currently metadata does not have special property to identify its version, > even though we already versions of metadata in 1.x, 2.0 and 2.x. If each file > in the metadata is bound with version it will be easier for upgrading related > issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1366) Bind metadata version with release version
[ https://issues.apache.org/jira/browse/KYLIN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongbin ma updated KYLIN-1366: -- Labels: newbie (was: ) > Bind metadata version with release version > -- > > Key: KYLIN-1366 > URL: https://issues.apache.org/jira/browse/KYLIN-1366 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Affects Versions: all >Reporter: hongbin ma >Assignee: hongbin ma > Labels: newbie > Fix For: v2.1 > > > Currently metadata does not have special property to identify its version, > even though we already versions of metadata in 1.x, 2.0 and 2.x. If each file > in the metadata is bound with version it will be easier for upgrading related > issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1366) Bind metadata version with release version
[ https://issues.apache.org/jira/browse/KYLIN-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongbin ma updated KYLIN-1366: -- Component/s: Metadata > Bind metadata version with release version > -- > > Key: KYLIN-1366 > URL: https://issues.apache.org/jira/browse/KYLIN-1366 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: hongbin ma >Assignee: hongbin ma > > Currently metadata does not have special property to identify its version, > even though we already versions of metadata in 1.x, 2.0 and 2.x. If each file > in the metadata is bound with version it will be easier for upgrading related > issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1366) Bind metadata version with release version
hongbin ma created KYLIN-1366: - Summary: Bind metadata version with release version Key: KYLIN-1366 URL: https://issues.apache.org/jira/browse/KYLIN-1366 Project: Kylin Issue Type: Improvement Reporter: hongbin ma Assignee: hongbin ma Currently metadata does not have special property to identify its version, even though we already versions of metadata in 1.x, 2.0 and 2.x. If each file in the metadata is bound with version it will be easier for upgrading related issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1082) Hive dependencies should be add to tmpjars
[ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114872#comment-15114872 ] wangxianbin commented on KYLIN-1082: I think your implementation is great, just two questions, one is do we need this for hbase dependencies? another is why delete "colon-comma" replace code for DefaultMapRedClasspath? > Hive dependencies should be add to tmpjars > -- > > Key: KYLIN-1082 > URL: https://issues.apache.org/jira/browse/KYLIN-1082 > Project: Kylin > Issue Type: Bug > Components: Environment , Job Engine >Reporter: liyang >Assignee: Zhong Yanghong > Labels: newbie > Fix For: v2.1, v1.3 > > Attachments: auto_hive_tmpjars_1_x_staging.patch, > auto_hive_tmpjars_2_x_staging.patch > > > Currently kylin assume all data nodes have hive deployment at exact same FS > location. However, a better position is to think hive as a client side app. > Then we need to ship hive jar with MR job every time. > This make deploy kylin a lot easier in cluster that does not have hive on all > data nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1082) Hive dependencies should be add to tmpjars
[ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114873#comment-15114873 ] Zhong Yanghong commented on KYLIN-1082: --- Both ways appends related jars to the property "tmpjars". > Hive dependencies should be add to tmpjars > -- > > Key: KYLIN-1082 > URL: https://issues.apache.org/jira/browse/KYLIN-1082 > Project: Kylin > Issue Type: Bug > Components: Environment , Job Engine >Reporter: liyang >Assignee: Zhong Yanghong > Labels: newbie > Fix For: v2.1, v1.3 > > Attachments: auto_hive_tmpjars_1_x_staging.patch, > auto_hive_tmpjars_2_x_staging.patch > > > Currently kylin assume all data nodes have hive deployment at exact same FS > location. However, a better position is to think hive as a client side app. > Then we need to ship hive jar with MR job every time. > This make deploy kylin a lot easier in cluster that does not have hive on all > data nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1082) Hive dependencies should be add to tmpjars
[ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114871#comment-15114871 ] Zhong Yanghong commented on KYLIN-1082: --- In my implementation, there are two ways to distribute dependent jars to datanode. One is appending the related jars to the property "kylin.job.mr.lib.dir", the other is set the property "kylin.job.mr.lib.dir", and copy the related jars into this specified directory. Since for every machine running kylin, the Hive is supposed to be installed. Then there is a shell script called "find-hive-dependency.sh" will find the hive dependencies and set the property "kylin.hive.dependency". To avoid uploading too many useless jars which kylin jobs will not use, there is a filter inside "AbstractHadoopJob.java" to filter out only the jars to be used. > Hive dependencies should be add to tmpjars > -- > > Key: KYLIN-1082 > URL: https://issues.apache.org/jira/browse/KYLIN-1082 > Project: Kylin > Issue Type: Bug > Components: Environment , Job Engine >Reporter: liyang >Assignee: Zhong Yanghong > Labels: newbie > Fix For: v2.1, v1.3 > > Attachments: auto_hive_tmpjars_1_x_staging.patch, > auto_hive_tmpjars_2_x_staging.patch > > > Currently kylin assume all data nodes have hive deployment at exact same FS > location. However, a better position is to think hive as a client side app. > Then we need to ship hive jar with MR job every time. > This make deploy kylin a lot easier in cluster that does not have hive on all > data nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1082) Hive dependencies should be add to tmpjars
[ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114866#comment-15114866 ] wangxianbin commented on KYLIN-1082: hi yanghong, implementation seem like a little different from original agreement in KYLIN-1021, which is offer two way of deploying kylin env, and it can distinguish it by the property "kylin.job.mr.lib.dir", however, your implementation add hive dependencies into tmpjars arbitrarily > Hive dependencies should be add to tmpjars > -- > > Key: KYLIN-1082 > URL: https://issues.apache.org/jira/browse/KYLIN-1082 > Project: Kylin > Issue Type: Bug > Components: Environment , Job Engine >Reporter: liyang >Assignee: Zhong Yanghong > Labels: newbie > Fix For: v2.1, v1.3 > > Attachments: auto_hive_tmpjars_1_x_staging.patch, > auto_hive_tmpjars_2_x_staging.patch > > > Currently kylin assume all data nodes have hive deployment at exact same FS > location. However, a better position is to think hive as a client side app. > Then we need to ship hive jar with MR job every time. > This make deploy kylin a lot easier in cluster that does not have hive on all > data nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1082) Hive dependencies should be add to tmpjars
[ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114853#comment-15114853 ] wangxianbin commented on KYLIN-1082: sorry for getting involved so late, I have been quite busy in last two weeks > Hive dependencies should be add to tmpjars > -- > > Key: KYLIN-1082 > URL: https://issues.apache.org/jira/browse/KYLIN-1082 > Project: Kylin > Issue Type: Bug > Components: Environment , Job Engine >Reporter: liyang >Assignee: Zhong Yanghong > Labels: newbie > Fix For: v2.1, v1.3 > > Attachments: auto_hive_tmpjars_1_x_staging.patch, > auto_hive_tmpjars_2_x_staging.patch > > > Currently kylin assume all data nodes have hive deployment at exact same FS > location. However, a better position is to think hive as a client side app. > Then we need to ship hive jar with MR job every time. > This make deploy kylin a lot easier in cluster that does not have hive on all > data nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1365) Kylin ACL enhancement
Zhong,Jason created KYLIN-1365: -- Summary: Kylin ACL enhancement Key: KYLIN-1365 URL: https://issues.apache.org/jira/browse/KYLIN-1365 Project: Kylin Issue Type: Improvement Affects Versions: v2.0 Reporter: Zhong,Jason Assignee: Zhong,Jason Now we have Project,Cube to manage, we can grant permission or role to Project,Cube, Admin or owner can grant permission to others. For Project Instance, if permission is granted to Role, user who has the role can access the project. if permission is granted to User, user who is granted can access the project. if Project Permission is granted,the cubes under project will inherit the permission. like if you grant CUBE_QUERY permission on project to user,then user can query all Cube under this project. For Cube Instance if you grant cube permission to Role, user who has the role can access the cube(Query,Edit,Operation,Admin). if you grant cube permission to user, user can accesss the cube. System Admin can see all projects and cubes. other user can see Project he has permission or parent Project of cubes he has permission. For DataModel only who create it or System Admin can edit,clone,drop it. only Cube Query inherit from Project works now, will enable edit,and other operation permission inherit from Project on UI. will do more enhancement on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1178) Build dictionary out of job engine
[ https://issues.apache.org/jira/browse/KYLIN-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-1178: Description: Kylin build dictionary in job engine node, usually this is okay. But if there is some high cardinality dimentions, the JVM heap couldn't fit in all distinct values, then job engine instance will crash with OOM error. Need to enhance on this, move the dictionary building to another process or a hadoop node. Ideally only need modify "CreateDictionaryJob.java", move the dictionary building to a mapper task. was: Kylin build dictionary in job engine node, usually this is okay. But if there is some high cardinality dimentions, the JVM heap couldn't fit in all distinct values, then job engine instance will crash with OOM error. Need to enhance on this, move the dictionary building to another process or a hadoop node. > Build dictionary out of job engine > -- > > Key: KYLIN-1178 > URL: https://issues.apache.org/jira/browse/KYLIN-1178 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > > Kylin build dictionary in job engine node, usually this is okay. But if there > is some high cardinality dimentions, the JVM heap couldn't fit in all > distinct values, then job engine instance will crash with OOM error. > Need to enhance on this, move the dictionary building to another process or a > hadoop node. Ideally only need modify "CreateDictionaryJob.java", move the > dictionary building to a mapper task. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1082) Hive dependencies should be add to tmpjars
[ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-1082: Fix Version/s: v1.3 v2.1 Component/s: Job Engine Environment Patch applied on 1.x-staging and 2.x-staging branches; Thanks Yanghong! > Hive dependencies should be add to tmpjars > -- > > Key: KYLIN-1082 > URL: https://issues.apache.org/jira/browse/KYLIN-1082 > Project: Kylin > Issue Type: Bug > Components: Environment , Job Engine >Reporter: liyang >Assignee: Zhong Yanghong > Labels: newbie > Fix For: v2.1, v1.3 > > Attachments: auto_hive_tmpjars_1_x_staging.patch, > auto_hive_tmpjars_2_x_staging.patch > > > Currently kylin assume all data nodes have hive deployment at exact same FS > location. However, a better position is to think hive as a client side app. > Then we need to ship hive jar with MR job every time. > This make deploy kylin a lot easier in cluster that does not have hive on all > data nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1082) Hive dependencies should be add to tmpjars
[ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-1082: -- Attachment: auto_hive_tmpjars_2_x_staging.patch auto_hive_tmpjars_1_x_staging.patch Patches for branch 1.x-staging and 2.x-staging > Hive dependencies should be add to tmpjars > -- > > Key: KYLIN-1082 > URL: https://issues.apache.org/jira/browse/KYLIN-1082 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Zhong Yanghong > Labels: newbie > Attachments: auto_hive_tmpjars_1_x_staging.patch, > auto_hive_tmpjars_2_x_staging.patch > > > Currently kylin assume all data nodes have hive deployment at exact same FS > location. However, a better position is to think hive as a client side app. > Then we need to ship hive jar with MR job every time. > This make deploy kylin a lot easier in cluster that does not have hive on all > data nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1082) Hive dependencies should be add to tmpjars
[ https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-1082: -- Attachment: (was: 0001-For-achieving-automatically-upload-hive-related-jars.patch) > Hive dependencies should be add to tmpjars > -- > > Key: KYLIN-1082 > URL: https://issues.apache.org/jira/browse/KYLIN-1082 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: Zhong Yanghong > Labels: newbie > > Currently kylin assume all data nodes have hive deployment at exact same FS > location. However, a better position is to think hive as a client side app. > Then we need to ship hive jar with MR job every time. > This make deploy kylin a lot easier in cluster that does not have hive on all > data nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1363) Got "unary operator expected" error when executing "bin/kylin.sh"
[
https://issues.apache.org/jira/browse/KYLIN-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hao Chen updated KYLIN-1363:
Flags: Patch
> Got "unary operator expected" error when executing "bin/kylin.sh"
> -
>
> Key: KYLIN-1363
> URL: https://issues.apache.org/jira/browse/KYLIN-1363
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.0, v1.2
> Environment: [hdp-sandbox-2.2.4.2-2]
> LSB Version:
> :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
> Distributor ID: CentOS
> Description: CentOS release 6.6 (Final)
> Release: 6.6
> Codename: Final
>Reporter: Hao Chen
>
> h1. Problem
> {code}
> [root@sandbox apache-kylin-1.2-bin]# ./bin/kylin.sh
> KYLIN_HOME is set to /root/apache-kylin-1.2-bin
> ./bin/kylin.sh: line 25: [: ==: unary operator expected
> ./bin/kylin.sh: line 96: [: ==: unary operator expected
> usage: kylin.sh start or kylin.sh stop
> {code}
> h1. Root Cause
> http://stackoverflow.com/a/13618376
> v1.x:
> * https://github.com/apache/kylin/blob/v1.2-release/bin/kylin.sh#L25
> * https://github.com/apache/kylin/blob/v1.2-release/bin/kylin.sh#L96
> v2.x:
> * https://github.com/apache/kylin/blob/2.x-staging/build/bin/kylin.sh#L25
> * https://github.com/apache/kylin/blob/2.x-staging/build/bin/kylin.sh#L96
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KYLIN-1364) Job type always displays as BUILD in RestAPI results
Dong Li created KYLIN-1364:
--
Summary: Job type always displays as BUILD in RestAPI results
Key: KYLIN-1364
URL: https://issues.apache.org/jira/browse/KYLIN-1364
Project: Kylin
Issue Type: Bug
Components: Job Engine
Affects Versions: v1.2, v2.0
Reporter: Dong Li
Assignee: Dong Li
Priority: Minor
In restAPI result of /jobs, type is always BUILD, no matter it's
BUILD/MERGE/REFRESH
eg. {...type: "BUILD"...}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
