[jira] [Assigned] (KYLIN-3709) Pass the wrong parameter to addResource function, kylin can not load the configuration file hbase.hdfs.xml .
[
https://issues.apache.org/jira/browse/KYLIN-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI reassigned KYLIN-3709:
---
Assignee: Chao Long (was: Shaofeng SHI)
Assign to Chao for this issue, the previous change was made by him.
> Pass the wrong parameter to addResource function, kylin can not load the
> configuration file hbase.hdfs.xml .
>
>
> Key: KYLIN-3709
> URL: https://issues.apache.org/jira/browse/KYLIN-3709
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.5.1, v2.5.2
>Reporter: Hongtao He
>Assignee: Chao Long
>Priority: Critical
> Attachments:
> method-1-Pass-the-wrong-parameter-to-addResource-function-kyl.patch,
> method-2-Pass-the-wrong-parameter-to-addResource-function-kyl.patch
>
>
> When calling Configuration.addResource, KYLIN-3648 use a Path as the
> parameter instead of a string in Kylin's HBaseConnection.
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> hdfsConf.addResource(new Path(hdfsConfigFile));
> {code}
> Use a Path as the parameter of Configuration.addResource is better.
> Unfortunately, the parameter which passed to the addResource function is
> wrong. The addResource function only accepts absolute paths, but the
> parameter is just a filename. For example, the value of hdfsConfigFile is
> "hbase.hdfs.xml", so addResource function will not work . The end result is
> that kylin can not load the hbase configuration file hbase.hdfs.xml .
> There are two ways to fix this bug, and I think method 1 is better.
> Method-1.revert the code
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> hdfsConf.addResource(hdfsConfigFile);{code}
> Method-2.Get the absolute path of the configuration file
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> String hbaseHdfsConfigPath = System.getProperty("user.dir") + "/../conf/" +
> hdfsConfigFile;
> hdfsConf.addResource(new Path(hbaseHdfsConfigPath));
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3709) Pass the wrong parameter to addResource function, kylin can not load the configuration file hbase.hdfs.xml .
[
https://issues.apache.org/jira/browse/KYLIN-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709649#comment-16709649
]
Shaofeng SHI commented on KYLIN-3709:
-
Hi Hongtao, we only merged KYLIN-3648 on master branch; Kylin 2.5.x doesn't
have the code change of KYLIN-3648 , so 2.5.0 to 2.5.2 not affected, you can
double check it.
If "kylin.storage.hbase.cluster-hdfs-config-file" be set to an absolute path,
is there such problem?
> Pass the wrong parameter to addResource function, kylin can not load the
> configuration file hbase.hdfs.xml .
>
>
> Key: KYLIN-3709
> URL: https://issues.apache.org/jira/browse/KYLIN-3709
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.5.1, v2.5.2
>Reporter: Hongtao He
>Assignee: Shaofeng SHI
>Priority: Critical
> Attachments:
> method-1-Pass-the-wrong-parameter-to-addResource-function-kyl.patch,
> method-2-Pass-the-wrong-parameter-to-addResource-function-kyl.patch
>
>
> When calling Configuration.addResource, KYLIN-3648 use a Path as the
> parameter instead of a string in Kylin's HBaseConnection.
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> hdfsConf.addResource(new Path(hdfsConfigFile));
> {code}
> Use a Path as the parameter of Configuration.addResource is better.
> Unfortunately, the parameter which passed to the addResource function is
> wrong. The addResource function only accepts absolute paths, but the
> parameter is just a filename. For example, the value of hdfsConfigFile is
> "hbase.hdfs.xml", so addResource function will not work . The end result is
> that kylin can not load the hbase configuration file hbase.hdfs.xml .
> There are two ways to fix this bug, and I think method 1 is better.
> Method-1.revert the code
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> hdfsConf.addResource(hdfsConfigFile);{code}
> Method-2.Get the absolute path of the configuration file
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> String hbaseHdfsConfigPath = System.getProperty("user.dir") + "/../conf/" +
> hdfsConfigFile;
> hdfsConf.addResource(new Path(hbaseHdfsConfigPath));
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709638#comment-16709638
]
XiaoXiang Yu edited comment on KYLIN-3406 at 12/5/18 5:57 AM:
--
[5261]I have reproduced error as reporter describe.
When any single file of Kylin's metadata miss in HDFS, the "Monitor" Page will
failed to display all job info and kylin process will throw
FileNotFoundException as above.
The better way should be miss file should only affect specific task. And if
only execute output miss, task should be success, otherwise task should be
marked as failed.
was (Author: hit_lacus):
[5261]I have reproduced error as reporter describe.
When any single file of Kylin's metadata miss in HDFS, the "Monitor" Page will
failed to display all job info and kylin process will throw
FileNotFoundException as above.
The better way should be miss file should only affect specific item. And if
only execute output miss, task should be success, otherwise task should be
marked as failed.
> When the /execute_output/ metadata file sinking to HDFS is deleted, the
> monitoring page does not display any tasks.
> ---
>
> Key: KYLIN-3406
> URL: https://issues.apache.org/jira/browse/KYLIN-3406
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.3.1
>Reporter: yujie.sun
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: image-2018-12-05-13-22-42-456.png
>
>
> *{color:#24292e}Issue description:{color}*
> When a single file of Hbase metadata is greater than a certain value, the
> file is sunk into HDFS for storage. When files in execute_output/ are deleted
> in HDFS, the monitoring page will not display any tasks.
> *{color:#24292e}Reproduction step:{color}*
> {color:#24292e}1.Use Hbase as metadata storage{color}
> {color:#24292e}2.Modify the hbase configuration ,configure
> "hbase.client.keyvalue.maxsize" into a small value, such as 524288.{color}
> {color:#24292e}3.Create a new build task, after the build task finished,find
> the /execute_output hbase metadata path on HDFS,
> /\{working_dir}/\{metadata_url}/resources/execute_output/..., and delete one
> of the files.{color}
> {color:#24292e}4.At this time,{color:#33}the monitoring page will not
> display any tasks.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709638#comment-16709638
]
XiaoXiang Yu commented on KYLIN-3406:
-
[5261]I have reproduced error as reporter describe.
When any single file of Kylin's metadata miss in HDFS, the "Monitor" Page will
failed to display all job info and kylin process will throw
FileNotFoundException as above.
The better way should be miss file should only affect specific item. And if
only execute output miss, task should be success, otherwise task should be
marked as failed.
> When the /execute_output/ metadata file sinking to HDFS is deleted, the
> monitoring page does not display any tasks.
> ---
>
> Key: KYLIN-3406
> URL: https://issues.apache.org/jira/browse/KYLIN-3406
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.3.1
>Reporter: yujie.sun
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: image-2018-12-05-13-22-42-456.png
>
>
> *{color:#24292e}Issue description:{color}*
> When a single file of Hbase metadata is greater than a certain value, the
> file is sunk into HDFS for storage. When files in execute_output/ are deleted
> in HDFS, the monitoring page will not display any tasks.
> *{color:#24292e}Reproduction step:{color}*
> {color:#24292e}1.Use Hbase as metadata storage{color}
> {color:#24292e}2.Modify the hbase configuration ,configure
> "hbase.client.keyvalue.maxsize" into a small value, such as 524288.{color}
> {color:#24292e}3.Create a new build task, after the build task finished,find
> the /execute_output hbase metadata path on HDFS,
> /\{working_dir}/\{metadata_url}/resources/execute_output/..., and delete one
> of the files.{color}
> {color:#24292e}4.At this time,{color:#33}the monitoring page will not
> display any tasks.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
[
https://issues.apache.org/jira/browse/KYLIN-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709636#comment-16709636
]
ASF GitHub Bot commented on KYLIN-3695:
---
codecov-io commented on issue #371: KYLIN-3695, fix lose decimal scale value in
column type decimal(a, b)
URL: https://github.com/apache/kylin/pull/371#issuecomment-444367864
# [Codecov](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=h1) Report
> Merging [#371](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=desc)
into
[master](https://codecov.io/gh/apache/kylin/commit/be5df4a8bb6939c0c62c6135c4312fe3ad56775d?src=pr&el=desc)
will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #371 +/- ##
- Coverage 23.64% 23.63% -0.02%
+ Complexity 4838 4837 -1
Files 1131 1131
Lines 6819368193
Branches 9684 9684
- Hits 1612416116 -8
- Misses5050350509 +6
- Partials 1566 1568 +2
```
| [Impacted
Files](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=tree) | Coverage Δ
| Complexity Δ | |
|---|---|---|---|
|
[...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/371/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==)
| `72.97% <0%> (-5.41%)` | `6% <0%> (-1%)` | |
|
[.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/371/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=)
| `63.84% <0%> (-2.31%)` | `0% <0%> (ø)` | |
|
[...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/371/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh)
| `70.21% <0%> (-0.92%)` | `7% <0%> (ø)` | |
--
[Continue to review full report at
Codecov](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=continue).
> **Legend** - [Click here to learn
more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by
[Codecov](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=footer). Last
update
[be5df4a...a073222](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=lastupdated).
Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Error while creating hive table through Kylin build cube with mysql imported
> tables
> ---
>
> Key: KYLIN-3695
> URL: https://issues.apache.org/jira/browse/KYLIN-3695
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
>Reporter: Gaurav Rawat
>Assignee: XiaoXiang Yu
>Priority: Minor
> Attachments: image-2018-12-04-15-53-49-983.png
>
>
> HI I am trying to build a cube with Kylin, the data gets souced fine from
> sqoop but the next step for creating hive tables fails . Looking at the
> command being fired it looks weird as the create statement looks good to me .
> I think the issue is with DOUBLE types as when I remove the same the create
> statement works fine . Can someone please help .
> I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
> The errors logs with commands as as below, the table is a msql table which
> had columns with DOUBLE type
> Command
> {code:java}
> hive -e "USE default; DROP TABLE IF EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 (
> HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string
> ,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK
[GitHub] codecov-io commented on issue #371: KYLIN-3695, fix lose decimal scale value in column type decimal(a, b)
codecov-io commented on issue #371: KYLIN-3695, fix lose decimal scale value in column type decimal(a, b) URL: https://github.com/apache/kylin/pull/371#issuecomment-444367864 # [Codecov](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=h1) Report > Merging [#371](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=desc) into [master](https://codecov.io/gh/apache/kylin/commit/be5df4a8bb6939c0c62c6135c4312fe3ad56775d?src=pr&el=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #371 +/- ## - Coverage 23.64% 23.63% -0.02% + Complexity 4838 4837 -1 Files 1131 1131 Lines 6819368193 Branches 9684 9684 - Hits 1612416116 -8 - Misses5050350509 +6 - Partials 1566 1568 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/371/diff?src=pr&el=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==) | `72.97% <0%> (-5.41%)` | `6% <0%> (-1%)` | | | [.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/371/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=) | `63.84% <0%> (-2.31%)` | `0% <0%> (ø)` | | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/371/diff?src=pr&el=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `70.21% <0%> (-0.92%)` | `7% <0%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=footer). Last update [be5df4a...a073222](https://codecov.io/gh/apache/kylin/pull/371?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
[
https://issues.apache.org/jira/browse/KYLIN-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709630#comment-16709630
]
ASF subversion and git services commented on KYLIN-3695:
Commit 7f672b074a068d20779d33889f1cb63b0da172a7 in kylin's branch
refs/heads/master from Emiya0306
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=7f672b0 ]
KYLIN-3695, fix lose decimal scale value in column type decimal(a, b)
> Error while creating hive table through Kylin build cube with mysql imported
> tables
> ---
>
> Key: KYLIN-3695
> URL: https://issues.apache.org/jira/browse/KYLIN-3695
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
>Reporter: Gaurav Rawat
>Assignee: XiaoXiang Yu
>Priority: Minor
> Attachments: image-2018-12-04-15-53-49-983.png
>
>
> HI I am trying to build a cube with Kylin, the data gets souced fine from
> sqoop but the next step for creating hive tables fails . Looking at the
> command being fired it looks weird as the create statement looks good to me .
> I think the issue is with DOUBLE types as when I remove the same the create
> statement works fine . Can someone please help .
> I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
> The errors logs with commands as as below, the table is a msql table which
> had columns with DOUBLE type
> Command
> {code:java}
> hive -e "USE default; DROP TABLE IF EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 (
> HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string
> ,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK_STOCK_NM string
> ,STOCK_APPROACH string ,STOCK_STOCK_TYP string ,INVESTOR_ID string
> ,INVESTOR_NM string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string
> ,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22)
> ,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22) ) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY '|' STORED AS TEXTFILE LOCATION
> 's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368';
> ALTER TABLE
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET
> TBLPROPERTIES('auto.purge'='true'); " --hiveconf hive.merge.mapredfiles=false
> --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2
> --hiveconf hive.exec.compress.output=true --hiveconf
> hive.auto.convert.join.noconditionaltask=true --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false
> --hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf
> hive.stats.autogather=true{code}
> Error is as below
> {code:java}
> OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds
> MismatchedTokenException(334!=347) at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at
> org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3808)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1333) at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70) at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468) at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1316) at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1456) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1226) at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787) at
> org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at
> org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod
[jira] [Commented] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
[
https://issues.apache.org/jira/browse/KYLIN-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709629#comment-16709629
]
ASF GitHub Bot commented on KYLIN-3695:
---
luguosheng1314 closed pull request #371: KYLIN-3695, fix lose decimal scale
value in column type decimal(a, b)
URL: https://github.com/apache/kylin/pull/371
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/webapp/app/js/controllers/cubeMeasures.js
b/webapp/app/js/controllers/cubeMeasures.js
index 7beb528d6b..f1821dda87 100644
--- a/webapp/app/js/controllers/cubeMeasures.js
+++ b/webapp/app/js/controllers/cubeMeasures.js
@@ -465,10 +465,10 @@ KylinApp.controller('CubeMeasuresCtrl', function ($scope,
$modal,MetaModel,cubes
return 'bigint';
} else {
if(colType.indexOf('decimal') != -1) {
-var returnRegex = new
RegExp('(\\w+)(?:\\((\\w+?)(?:\\,(\\w+?))?\\))?')
-var returnValue = returnRegex.exec(colType)
-var precision = 19
-var scale = returnValue[3]
+var returnRegex = new
RegExp('(\\w+)(?:\\((\\w+?)(?:\\,(\\w+?))?\\))?');
+var returnValue = returnRegex.exec(colType);
+var precision = 19;
+var scale = returnValue[3] || 0;
return 'decimal(' + precision + ',' + scale + ')';
}else{
return colType;
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Error while creating hive table through Kylin build cube with mysql imported
> tables
> ---
>
> Key: KYLIN-3695
> URL: https://issues.apache.org/jira/browse/KYLIN-3695
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
>Reporter: Gaurav Rawat
>Assignee: XiaoXiang Yu
>Priority: Minor
> Attachments: image-2018-12-04-15-53-49-983.png
>
>
> HI I am trying to build a cube with Kylin, the data gets souced fine from
> sqoop but the next step for creating hive tables fails . Looking at the
> command being fired it looks weird as the create statement looks good to me .
> I think the issue is with DOUBLE types as when I remove the same the create
> statement works fine . Can someone please help .
> I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
> The errors logs with commands as as below, the table is a msql table which
> had columns with DOUBLE type
> Command
> {code:java}
> hive -e "USE default; DROP TABLE IF EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 (
> HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string
> ,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK_STOCK_NM string
> ,STOCK_APPROACH string ,STOCK_STOCK_TYP string ,INVESTOR_ID string
> ,INVESTOR_NM string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string
> ,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22)
> ,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22) ) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY '|' STORED AS TEXTFILE LOCATION
> 's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368';
> ALTER TABLE
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET
> TBLPROPERTIES('auto.purge'='true'); " --hiveconf hive.merge.mapredfiles=false
> --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2
> --hiveconf hive.exec.compress.output=true --hiveconf
> hive.auto.convert.join.noconditionaltask=true --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false
> --hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf
> hive.stats.autogather=true{code}
> Error is as below
> {code:java}
> OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds
> MismatchedTokenException(334!=347) at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at
> org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:380
[GitHub] luguosheng1314 closed pull request #371: KYLIN-3695, fix lose decimal scale value in column type decimal(a, b)
luguosheng1314 closed pull request #371: KYLIN-3695, fix lose decimal scale
value in column type decimal(a, b)
URL: https://github.com/apache/kylin/pull/371
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/webapp/app/js/controllers/cubeMeasures.js
b/webapp/app/js/controllers/cubeMeasures.js
index 7beb528d6b..f1821dda87 100644
--- a/webapp/app/js/controllers/cubeMeasures.js
+++ b/webapp/app/js/controllers/cubeMeasures.js
@@ -465,10 +465,10 @@ KylinApp.controller('CubeMeasuresCtrl', function ($scope,
$modal,MetaModel,cubes
return 'bigint';
} else {
if(colType.indexOf('decimal') != -1) {
-var returnRegex = new
RegExp('(\\w+)(?:\\((\\w+?)(?:\\,(\\w+?))?\\))?')
-var returnValue = returnRegex.exec(colType)
-var precision = 19
-var scale = returnValue[3]
+var returnRegex = new
RegExp('(\\w+)(?:\\((\\w+?)(?:\\,(\\w+?))?\\))?');
+var returnValue = returnRegex.exec(colType);
+var precision = 19;
+var scale = returnValue[3] || 0;
return 'decimal(' + precision + ',' + scale + ')';
}else{
return colType;
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
[
https://issues.apache.org/jira/browse/KYLIN-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709619#comment-16709619
]
ASF GitHub Bot commented on KYLIN-3695:
---
Emiya0306 opened a new pull request #371: KYLIN-3695, fix lose decimal scale
value in column type decimal(a, b)
URL: https://github.com/apache/kylin/pull/371
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Error while creating hive table through Kylin build cube with mysql imported
> tables
> ---
>
> Key: KYLIN-3695
> URL: https://issues.apache.org/jira/browse/KYLIN-3695
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
>Reporter: Gaurav Rawat
>Assignee: XiaoXiang Yu
>Priority: Minor
> Attachments: image-2018-12-04-15-53-49-983.png
>
>
> HI I am trying to build a cube with Kylin, the data gets souced fine from
> sqoop but the next step for creating hive tables fails . Looking at the
> command being fired it looks weird as the create statement looks good to me .
> I think the issue is with DOUBLE types as when I remove the same the create
> statement works fine . Can someone please help .
> I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
> The errors logs with commands as as below, the table is a msql table which
> had columns with DOUBLE type
> Command
> {code:java}
> hive -e "USE default; DROP TABLE IF EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 (
> HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string
> ,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK_STOCK_NM string
> ,STOCK_APPROACH string ,STOCK_STOCK_TYP string ,INVESTOR_ID string
> ,INVESTOR_NM string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string
> ,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22)
> ,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22) ) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY '|' STORED AS TEXTFILE LOCATION
> 's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368';
> ALTER TABLE
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET
> TBLPROPERTIES('auto.purge'='true'); " --hiveconf hive.merge.mapredfiles=false
> --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2
> --hiveconf hive.exec.compress.output=true --hiveconf
> hive.auto.convert.join.noconditionaltask=true --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false
> --hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf
> hive.stats.autogather=true{code}
> Error is as below
> {code:java}
> OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds
> MismatchedTokenException(334!=347) at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at
> org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3808)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1333) at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70) at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468) at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1316) at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1456) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1226) at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787) at
> org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at
> org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) at
> s
[jira] [Commented] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709621#comment-16709621
]
XiaoXiang Yu commented on KYLIN-3406:
-
{quote}KYLIN [ ERROR ] 12-05 11:01:10.223
org.apache.kylin.job.dao.ExecutableDao.getJobOutput(ExecutableDao.java:355)
from http-bio-7070-exec-1
> error get job output id:3b27b7df-aece-ba28-e986-3758fe8b9a85-10
java.io.IOException: Failed to read resource at
/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-10
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getInputStream(HBaseResourceStore.java:266)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getResourceImpl(HBaseResourceStore.java:287)
at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:175)
at
org.apache.kylin.job.dao.ExecutableDao.readJobOutputResource(ExecutableDao.java:219)
at org.apache.kylin.job.dao.ExecutableDao.getJobOutput(ExecutableDao.java:347)
at
org.apache.kylin.job.execution.ExecutableManager.getOutput(ExecutableManager.java:160)
at
org.apache.kylin.rest.service.JobService.getSingleJobInstance(JobService.java:505)
at org.apache.kylin.rest.service.JobService.getJobInstance(JobService.java:464)
at org.apache.kylin.rest.service.JobService.searchJobsV2(JobService.java:973)
at org.apache.kylin.rest.controller.JobController.list(JobController.java:86)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at
org.springframework.security.web.auth
[jira] [Commented] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
[
https://issues.apache.org/jira/browse/KYLIN-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709620#comment-16709620
]
ASF GitHub Bot commented on KYLIN-3695:
---
asfgit commented on issue #371: KYLIN-3695, fix lose decimal scale value in
column type decimal(a, b)
URL: https://github.com/apache/kylin/pull/371#issuecomment-444363598
Can one of the admins verify this patch?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Error while creating hive table through Kylin build cube with mysql imported
> tables
> ---
>
> Key: KYLIN-3695
> URL: https://issues.apache.org/jira/browse/KYLIN-3695
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
>Reporter: Gaurav Rawat
>Assignee: XiaoXiang Yu
>Priority: Minor
> Attachments: image-2018-12-04-15-53-49-983.png
>
>
> HI I am trying to build a cube with Kylin, the data gets souced fine from
> sqoop but the next step for creating hive tables fails . Looking at the
> command being fired it looks weird as the create statement looks good to me .
> I think the issue is with DOUBLE types as when I remove the same the create
> statement works fine . Can someone please help .
> I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
> The errors logs with commands as as below, the table is a msql table which
> had columns with DOUBLE type
> Command
> {code:java}
> hive -e "USE default; DROP TABLE IF EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 (
> HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string
> ,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK_STOCK_NM string
> ,STOCK_APPROACH string ,STOCK_STOCK_TYP string ,INVESTOR_ID string
> ,INVESTOR_NM string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string
> ,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22)
> ,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22) ) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY '|' STORED AS TEXTFILE LOCATION
> 's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368';
> ALTER TABLE
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET
> TBLPROPERTIES('auto.purge'='true'); " --hiveconf hive.merge.mapredfiles=false
> --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2
> --hiveconf hive.exec.compress.output=true --hiveconf
> hive.auto.convert.join.noconditionaltask=true --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false
> --hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf
> hive.stats.autogather=true{code}
> Error is as below
> {code:java}
> OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds
> MismatchedTokenException(334!=347) at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at
> org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3808)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1333) at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70) at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468) at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1316) at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1456) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1226) at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787) at
> org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at
> org.apache.hadoo
[GitHub] asfgit commented on issue #371: KYLIN-3695, fix lose decimal scale value in column type decimal(a, b)
asfgit commented on issue #371: KYLIN-3695, fix lose decimal scale value in column type decimal(a, b) URL: https://github.com/apache/kylin/pull/371#issuecomment-444363598 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Emiya0306 opened a new pull request #371: KYLIN-3695, fix lose decimal scale value in column type decimal(a, b)
Emiya0306 opened a new pull request #371: KYLIN-3695, fix lose decimal scale value in column type decimal(a, b) URL: https://github.com/apache/kylin/pull/371 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3406:
Attachment: image-2018-12-05-13-22-42-456.png
> When the /execute_output/ metadata file sinking to HDFS is deleted, the
> monitoring page does not display any tasks.
> ---
>
> Key: KYLIN-3406
> URL: https://issues.apache.org/jira/browse/KYLIN-3406
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.3.1
>Reporter: yujie.sun
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: image-2018-12-05-13-22-42-456.png
>
>
> *{color:#24292e}Issue description:{color}*
> When a single file of Hbase metadata is greater than a certain value, the
> file is sunk into HDFS for storage. When files in execute_output/ are deleted
> in HDFS, the monitoring page will not display any tasks.
> *{color:#24292e}Reproduction step:{color}*
> {color:#24292e}1.Use Hbase as metadata storage{color}
> {color:#24292e}2.Modify the hbase configuration ,configure
> "hbase.client.keyvalue.maxsize" into a small value, such as 524288.{color}
> {color:#24292e}3.Create a new build task, after the build task finished,find
> the /execute_output hbase metadata path on HDFS,
> /\{working_dir}/\{metadata_url}/resources/execute_output/..., and delete one
> of the files.{color}
> {color:#24292e}4.At this time,{color:#33}the monitoring page will not
> display any tasks.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709615#comment-16709615
]
XiaoXiang Yu commented on KYLIN-3406:
-
!image-2018-12-05-13-22-42-456.png!
> When the /execute_output/ metadata file sinking to HDFS is deleted, the
> monitoring page does not display any tasks.
> ---
>
> Key: KYLIN-3406
> URL: https://issues.apache.org/jira/browse/KYLIN-3406
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.3.1
>Reporter: yujie.sun
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: image-2018-12-05-13-22-42-456.png
>
>
> *{color:#24292e}Issue description:{color}*
> When a single file of Hbase metadata is greater than a certain value, the
> file is sunk into HDFS for storage. When files in execute_output/ are deleted
> in HDFS, the monitoring page will not display any tasks.
> *{color:#24292e}Reproduction step:{color}*
> {color:#24292e}1.Use Hbase as metadata storage{color}
> {color:#24292e}2.Modify the hbase configuration ,configure
> "hbase.client.keyvalue.maxsize" into a small value, such as 524288.{color}
> {color:#24292e}3.Create a new build task, after the build task finished,find
> the /execute_output hbase metadata path on HDFS,
> /\{working_dir}/\{metadata_url}/resources/execute_output/..., and delete one
> of the files.{color}
> {color:#24292e}4.At this time,{color:#33}the monitoring page will not
> display any tasks.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3690) New streaming backend implementation
[ https://issues.apache.org/jira/browse/KYLIN-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709553#comment-16709553 ] Ma Gang commented on KYLIN-3690: Hi Hubert, The PR should be ready next week, because I did some refactor work after merge the existing code to the master, and I need further test after the refactoring. Sorry it comes a little bit late. > New streaming backend implementation > > > Key: KYLIN-3690 > URL: https://issues.apache.org/jira/browse/KYLIN-3690 > Project: Kylin > Issue Type: Sub-task >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > > New streaming backend implementation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides
shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides URL: https://github.com/apache/kylin/pull/348#discussion_r238900904 ## File path: website/_docs/howto/howto_upgrade.md ## @@ -1,112 +1,146 @@ --- layout: docs -title: Upgrade From Old Versions +title: Upgrade Kylin From Old Versions categories: howto permalink: /docs/howto/howto_upgrade.html since: v1.5.1 --- -Running as a Hadoop client, Apache Kylin's metadata and Cube data are persistended in Hadoop (HBase and HDFS), so the upgrade is relatively easy and user does not need worry about data loss. The upgrade can be performed in the following steps: +Running as a Hadoop client, Apache Kylin's metadata and Cube data are persistent in Hadoop (HBase and HDFS), so the upgrade is relatively easy and user does not need worry about data loss. The upgrade can be performed in the following steps: Review comment: "persistent" is incorrect in this context, it is adjective, while here we need "be persisted" which is a passive voice; (a typo in the previous version). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides
shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides URL: https://github.com/apache/kylin/pull/348#discussion_r238900533 ## File path: website/_docs/howto/howto_upgrade.md ## @@ -1,112 +1,146 @@ --- layout: docs -title: Upgrade From Old Versions +title: Upgrade Kylin From Old Versions Review comment: Of course it is "Kylin", this looks redundant ... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides
shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides URL: https://github.com/apache/kylin/pull/348#discussion_r238900295 ## File path: website/_data/docs.yml ## @@ -78,6 +77,10 @@ - howto/howto_use_cli - howto/howto_ldap_and_sso - howto/howto_use_beeline - - howto/howto_update_coprocessor + - howto/howto_upgrade_coprocessor Review comment: This change will change the URL endpoint. We hope the URL can be kept as there are many external references. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides
shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides URL: https://github.com/apache/kylin/pull/348#discussion_r238900055 ## File path: website/download/index.md ## @@ -20,7 +20,7 @@ You can verify the download by following these [procedures](https://www.apache.o v2.5.0 - This is a major release after 2.4, with 96 bug fixes and enhancement. Check the [v2.5.0 release announcement](/blog/2018/09/20/release-v2.5.0/) and the release notes. -- [Release notes](/docs/release_notes.html) and [upgrade guide](/docs/howto/howto_upgrade.html) +- [Kylin Release Notes](/docs/releases/kylin_release_notes.html) and [Upgrade Kylin From Old Versions](/docs/howto/howto_upgrade.html) Review comment: I think the "upgrade guide" in lower case is more friendly than "Upgrade Kylin From Old Versions" in hump... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides
shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides URL: https://github.com/apache/kylin/pull/348#discussion_r238899806 ## File path: website/download/index.cn.md ## @@ -21,7 +21,7 @@ title: 下载 v2.5.0 - 这是2.4版本后的一个主要发布版本，包含了96 个以及各种改进。关于具体内容请查看[v2.5.0 正式发布](/blog/2018/09/20/release-v2.5.0/). -- [发布说明](/docs/release_notes.html) and [升级指南](/docs/howto/howto_upgrade.html) +- [发布声明](/cn/docs/releases/kylin_release_notes.html) and [Kylin 升级](/cn/docs/howto/howto_upgrade.html) Review comment: We don't need to provide multiple language for release notes... This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides
shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides URL: https://github.com/apache/kylin/pull/348#discussion_r238899445 ## File path: website/_docs/index.cn.md ## @@ -9,21 +9,31 @@ permalink: /cn/docs/index.html > Extreme OLAP Engine for Big Data -Apache Kylin™是一个开源的分布式分析引擎，提供Hadoop之上的SQL查询接口及多维分析（OLAP）能力以支持超大规模数据，最初由eBay Inc.开发并贡献至开源社区。 +Apache Kylin™ 是一个开源 OLAP（On-Line Analytics Processing）分析引擎，针对数据库与亿万级记录提供亚秒级查询响应能力。2014 年由 eBay Inc.开发并贡献至开源社区。2015 年毕业于 Apache 顶级项目并迅速被全球数千家公司与组织应用于数据分析业务，同时也在 2015 与 2016 年赢得 InfoWorld Bossie Awards 年度最佳大数据开源工具。 Review comment: "Online" is a word, not "On-Line", please check the terms: https://en.wikipedia.org/wiki/Online_analytical_processing This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides
shaofengshi commented on a change in pull request #348: added release notes, modified kylin and coprocessor upgrade guides URL: https://github.com/apache/kylin/pull/348#discussion_r238899442 ## File path: website/_docs/index.cn.md ## @@ -9,21 +9,31 @@ permalink: /cn/docs/index.html > Extreme OLAP Engine for Big Data -Apache Kylin™是一个开源的分布式分析引擎，提供Hadoop之上的SQL查询接口及多维分析（OLAP）能力以支持超大规模数据，最初由eBay Inc.开发并贡献至开源社区。 +Apache Kylin™ 是一个开源 OLAP（On-Line Analytics Processing）分析引擎，针对数据库与亿万级记录提供亚秒级查询响应能力。2014 年由 eBay Inc.开发并贡献至开源社区。2015 年毕业于 Apache 顶级项目并迅速被全球数千家公司与组织应用于数据分析业务，同时也在 2015 与 2016 年赢得 InfoWorld Bossie Awards 年度最佳大数据开源工具。 Review comment: "Online" is a word, not "On-Line", please check the terms: https://en.wikipedia.org/wiki/Online_analytical_processing This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3686) Top_N metric code requires cube storage type to be ID_SHARDED_HBASE, but the Web UI defaults to ID_HBASE and provides no safeguards against storage type mismatch
[
https://issues.apache.org/jira/browse/KYLIN-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709162#comment-16709162
]
Vsevolod Ostapenko commented on KYLIN-3686:
---
Hi Chao,
the "kylin.storage.default" parameter is not set in kylin.properties in our
environment, so it does default ti ID_HBASE, as I understand.
As far as I can see, the fix for KYLIN-3636 changes only cube defaults to be
ID_SHARDED_HBASE.
However, it does not address the misalignment and lack of safety checks between
cube storage type and implied pre-requisites of Top_N metric. It's still
possible to load cube with ID_HBASE from JSON and define Topn_N metric and get
failing cube build with no clear explanations for the failure reasons.
Thanks,
Vsevolod.
> Top_N metric code requires cube storage type to be ID_SHARDED_HBASE, but the
> Web UI defaults to ID_HBASE and provides no safeguards against storage type
> mismatch
> -
>
> Key: KYLIN-3686
> URL: https://issues.apache.org/jira/browse/KYLIN-3686
> Project: Kylin
> Issue Type: Improvement
> Components: Measure - TopN, Metadata, Web
>Affects Versions: v2.5.0
> Environment: HDP 2.5.6, Kylin 2.5
>Reporter: Vsevolod Ostapenko
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.6.0
>
>
> When new cube is defined via Kylin 2.5 UI, the default cube storage type is
> set to 0 (ID_HBASE).
> Top_N metric support is currently hard coded to expect cube storage type 2
> (ID_SHARDED_HBASE), and it *_does not_* check if the cube storage type is the
> "sharded HBASE".
> UI provides no safeguards either to prevent a user from defining a cube with
> Top_N metric that would blow up on the cube building stage with a perplexing
> stack trace like the following:
> {quote}2018-10-22 16:15:50,388 ERROR [main]
> org.apache.kylin.engine.mr.KylinMapper:
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> {quote}
> Please, either:
> – modify Top_N code to support all cube storage types (not only
> ID_SHARDED_HBASE),
> or
> – modify Top_N code to perform explicit check for cube storage type and
> raise descriptive exception, when cube storage is not the one that is
> expected. Plus update the UI to prevent the user from creating cube
> definitions that are incompatible with the storage type compatible with Top_N
> measure
> PS: NDCCuboidBuilder,java contains the following line:
> {quote}int offset = RowConstants.ROWKEY_SHARDID_LEN +
> RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId{quote}
> If cube storage type is not ID_SHARDED_HBASE, offset is calculated
> incorrectly, which leads to ArrayIndexOutOfBounds exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3686) Top_N metric code requires cube storage type to be ID_SHARDED_HBASE, but the Web UI defaults to ID_HBASE and provides no safeguards against storage type mismatch
[
https://issues.apache.org/jira/browse/KYLIN-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708623#comment-16708623
]
Chao Long edited comment on KYLIN-3686 at 12/4/18 12:21 PM:
Hi Vsevolod, The front-end will get the value of parameter
"kylin.storage.default"(default ID_SHARDED_HBASE=2) from the kylin.properties
and will use it as the cube's storage type. Can you check this parameter in
your kylin.properties?
Another situation is the front-end does not get this value for some unknown
reason, it will pass a null value to the back-end, and the back-end will use
ID_HBASE as the default value of storage type.
If the latter, KYLIN-3636 has fixed it and will release soon in kylin-2.5.2.
was (Author: wayne0101):
Hi Vsevolod, The front-end will get the value of parameter
"kylin.storage.default"(default ID_SHARDED_HBASE=2) from the kylin.properties
and will use it as the cube's storage type. Can you check this parameter in
your kylin.properties?
Another situation is the front-end does not get this value for some unknown
reason, it will pass a null value to the back-end, and the back-end will use
ID_HBASE as the default value of storage type.
If the latter, [KYLIN-3636|https://issues.apache.org/jira/browse/KYLIN-3636]
has fixed it and will release soon.
> Top_N metric code requires cube storage type to be ID_SHARDED_HBASE, but the
> Web UI defaults to ID_HBASE and provides no safeguards against storage type
> mismatch
> -
>
> Key: KYLIN-3686
> URL: https://issues.apache.org/jira/browse/KYLIN-3686
> Project: Kylin
> Issue Type: Improvement
> Components: Measure - TopN, Metadata, Web
>Affects Versions: v2.5.0
> Environment: HDP 2.5.6, Kylin 2.5
>Reporter: Vsevolod Ostapenko
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.6.0
>
>
> When new cube is defined via Kylin 2.5 UI, the default cube storage type is
> set to 0 (ID_HBASE).
> Top_N metric support is currently hard coded to expect cube storage type 2
> (ID_SHARDED_HBASE), and it *_does not_* check if the cube storage type is the
> "sharded HBASE".
> UI provides no safeguards either to prevent a user from defining a cube with
> Top_N metric that would blow up on the cube building stage with a perplexing
> stack trace like the following:
> {quote}2018-10-22 16:15:50,388 ERROR [main]
> org.apache.kylin.engine.mr.KylinMapper:
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> {quote}
> Please, either:
> – modify Top_N code to support all cube storage types (not only
> ID_SHARDED_HBASE),
> or
> – modify Top_N code to perform explicit check for cube storage type and
> raise descriptive exception, when cube storage is not the one that is
> expected. Plus update the UI to prevent the user from creating cube
> definitions that are incompatible with the storage type compatible with Top_N
> measure
> PS: NDCCuboidBuilder,java contains the following line:
> {quote}int offset = RowConstants.ROWKEY_SHARDID_LEN +
> RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId{quote}
> If cube storage type is not ID_SHARDED_HBASE, offset is calculated
> incorrectly, which leads to ArrayIndexOutOfBounds exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3686) Top_N metric code requires cube storage type to be ID_SHARDED_HBASE, but the Web UI defaults to ID_HBASE and provides no safeguards against storage type mismatch
[
https://issues.apache.org/jira/browse/KYLIN-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708623#comment-16708623
]
Chao Long commented on KYLIN-3686:
--
Hi Vsevolod, The front-end will get the value of parameter
"kylin.storage.default"(default ID_SHARDED_HBASE=2) from the kylin.properties
and will use it as the cube's storage type. Can you check this parameter in
your kylin.properties?
Another situation is the front-end does not get this value for some unknown
reason, it will pass a null value to the back-end, and the back-end will use
ID_HBASE as the default value of storage type.
If the latter, [KYLIN-3636|https://issues.apache.org/jira/browse/KYLIN-3636]
has fixed it and will release soon.
> Top_N metric code requires cube storage type to be ID_SHARDED_HBASE, but the
> Web UI defaults to ID_HBASE and provides no safeguards against storage type
> mismatch
> -
>
> Key: KYLIN-3686
> URL: https://issues.apache.org/jira/browse/KYLIN-3686
> Project: Kylin
> Issue Type: Improvement
> Components: Measure - TopN, Metadata, Web
>Affects Versions: v2.5.0
> Environment: HDP 2.5.6, Kylin 2.5
>Reporter: Vsevolod Ostapenko
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.6.0
>
>
> When new cube is defined via Kylin 2.5 UI, the default cube storage type is
> set to 0 (ID_HBASE).
> Top_N metric support is currently hard coded to expect cube storage type 2
> (ID_SHARDED_HBASE), and it *_does not_* check if the cube storage type is the
> "sharded HBASE".
> UI provides no safeguards either to prevent a user from defining a cube with
> Top_N metric that would blow up on the cube building stage with a perplexing
> stack trace like the following:
> {quote}2018-10-22 16:15:50,388 ERROR [main]
> org.apache.kylin.engine.mr.KylinMapper:
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> {quote}
> Please, either:
> – modify Top_N code to support all cube storage types (not only
> ID_SHARDED_HBASE),
> or
> – modify Top_N code to perform explicit check for cube storage type and
> raise descriptive exception, when cube storage is not the one that is
> expected. Plus update the UI to prevent the user from creating cube
> definitions that are incompatible with the storage type compatible with Top_N
> measure
> PS: NDCCuboidBuilder,java contains the following line:
> {quote}int offset = RowConstants.ROWKEY_SHARDID_LEN +
> RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId{quote}
> If cube storage type is not ID_SHARDED_HBASE, offset is calculated
> incorrectly, which leads to ArrayIndexOutOfBounds exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3690) New streaming backend implementation
[
https://issues.apache.org/jira/browse/KYLIN-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708544#comment-16708544
]
Hubert STEFANI commented on KYLIN-3690:
---
Hi Ma,
is there any rescheduling of the PR ?
We feel so impatient to help you on making it production ready !
Hu{color:#FF}bert{color}
> New streaming backend implementation
>
>
> Key: KYLIN-3690
> URL: https://issues.apache.org/jira/browse/KYLIN-3690
> Project: Kylin
> Issue Type: Sub-task
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
>
> New streaming backend implementation
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu reassigned KYLIN-3406:
---
Assignee: XiaoXiang Yu
> When the /execute_output/ metadata file sinking to HDFS is deleted, the
> monitoring page does not display any tasks.
> ---
>
> Key: KYLIN-3406
> URL: https://issues.apache.org/jira/browse/KYLIN-3406
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.3.1
>Reporter: yujie.sun
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
>
> *{color:#24292e}Issue description:{color}*
> When a single file of Hbase metadata is greater than a certain value, the
> file is sunk into HDFS for storage. When files in execute_output/ are deleted
> in HDFS, the monitoring page will not display any tasks.
> *{color:#24292e}Reproduction step:{color}*
> {color:#24292e}1.Use Hbase as metadata storage{color}
> {color:#24292e}2.Modify the hbase configuration ,configure
> "hbase.client.keyvalue.maxsize" into a small value, such as 524288.{color}
> {color:#24292e}3.Create a new build task, after the build task finished,find
> the /execute_output hbase metadata path on HDFS,
> /\{working_dir}/\{metadata_url}/resources/execute_output/..., and delete one
> of the files.{color}
> {color:#24292e}4.At this time,{color:#33}the monitoring page will not
> display any tasks.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
[
https://issues.apache.org/jira/browse/KYLIN-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708395#comment-16708395
]
XiaoXiang Yu edited comment on KYLIN-3695 at 12/4/18 9:04 AM:
--
[~gauravbrills] I debug in my test env.
I found when using decimal data type with empty precision to build a cube. The
front end(JavaScript) will use 'undefined' for null value and send it to
back-end to validate cube metadata. It clear that decimal(19, undefined) do
not match a valid regex pattern, so back end return error message to front end
as my above message.
Source code at
org.apache.kylin.metadata.datatype.DataType.[https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/metadata/datatype/DataType.java]
I think it has no need to fix it because the root cause is clear.
The following is part of preload send by front end.
{
"name":"SUM_INTEREST_SCORE2",
"function":{
"expression":"SUM",
"returntype":"decimal(19,undefined)",
"parameter":
{ "type":"column", "value":"USERACTION.INTEREST_SCORE2"
}
}
}
was (Author: hit_lacus):
[~gauravbrills] I debug in my test env.
I found when using decimal data type with empty precision to build a cube. The
front end(JavaScript) will use 'undefined' for null value and send it to
back-end to validate cube metadata. It clear that decimal(19, undefined) do
not match a valid regex pattern, so back end return error message to front end
as my above message.
Source code at
org.apache.kylin.metadata.datatype.DataType.[https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/metadata/datatype/DataType.java]
I think it has no need to fix it because the root cause is clear.
The following is part of preload send by backend.
{
"name":"SUM_INTEREST_SCORE2",
"function":{
"expression":"SUM",
"returntype":"decimal(19,undefined)",
"parameter":
{ "type":"column", "value":"USERACTION.INTEREST_SCORE2"
}
}
}
> Error while creating hive table through Kylin build cube with mysql imported
> tables
> ---
>
> Key: KYLIN-3695
> URL: https://issues.apache.org/jira/browse/KYLIN-3695
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
>Reporter: Gaurav Rawat
>Assignee: XiaoXiang Yu
>Priority: Minor
> Attachments: image-2018-12-04-15-53-49-983.png
>
>
> HI I am trying to build a cube with Kylin, the data gets souced fine from
> sqoop but the next step for creating hive tables fails . Looking at the
> command being fired it looks weird as the create statement looks good to me .
> I think the issue is with DOUBLE types as when I remove the same the create
> statement works fine . Can someone please help .
> I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
> The errors logs with commands as as below, the table is a msql table which
> had columns with DOUBLE type
> Command
> {code:java}
> hive -e "USE default; DROP TABLE IF EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 (
> HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string
> ,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK_STOCK_NM string
> ,STOCK_APPROACH string ,STOCK_STOCK_TYP string ,INVESTOR_ID string
> ,INVESTOR_NM string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string
> ,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22)
> ,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22) ) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY '|' STORED AS TEXTFILE LOCATION
> 's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368';
> ALTER TABLE
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET
> TBLPROPERTIES('auto.purge'='true'); " --hiveconf hive.merge.mapredfiles=false
> --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2
> --hiveconf hive.exec.compress.output=true --hiveconf
> hive.auto.convert.join.noconditionaltask=true --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false
> --hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf
> hive.stats.autogather=true{code}
> Error is as below
> {code:java}
> OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds
> MismatchedTokenException(334!=347) at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime
[jira] [Commented] (KYLIN-3617) Reduce number of visiting metastore for job scheduler
[ https://issues.apache.org/jira/browse/KYLIN-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708421#comment-16708421 ] XiaoXiang Yu commented on KYLIN-3617: - I will try to verfiy these patch and merge to master branch. > Reduce number of visiting metastore for job scheduler > - > > Key: KYLIN-3617 > URL: https://issues.apache.org/jira/browse/KYLIN-3617 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.4.1 >Reporter: nichunen >Assignee: nichunen >Priority: Major > Fix For: v2.6.0 > > > For KYLIN-3470 introduced cache for jobs' metadata, it's also can be used in > job scheduler to reduce the pressure on metastore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
[
https://issues.apache.org/jira/browse/KYLIN-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708395#comment-16708395
]
XiaoXiang Yu edited comment on KYLIN-3695 at 12/4/18 8:55 AM:
--
[~gauravbrills] I debug in my test env.
I found when using decimal data type with empty precision to build a cube. The
front end(JavaScript) will use 'undefined' for null value and send it to
back-end to validate cube metadata. It clear that decimal(19, undefined) do
not match a valid regex pattern, so back end return error message to front end
as my above message.
Source code at
org.apache.kylin.metadata.datatype.DataType.[https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/metadata/datatype/DataType.java]
I think it has no need to fix it because the root cause is clear.
The following is part of preload send by backend.
{
"name":"SUM_INTEREST_SCORE2",
"function":{
"expression":"SUM",
"returntype":"decimal(19,undefined)",
"parameter":
{ "type":"column", "value":"USERACTION.INTEREST_SCORE2"
}
}
}
was (Author: hit_lacus):
[~gauravbrills] I debug in my test env.
I found when using decimal data type with empty precision to build a cube. The
front end(JavaScript) will use 'undefined' for null value and send it to
back-end to validate cube metadata. It clear that decimal(19, undefined) do
not match a valid regex pattern, so back end return error message to front end
as my above message.
Source code at
org.apache.kylin.metadata.datatype.DataType.[https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/metadata/datatype/DataType.java]
I think it has no need to fix it because the root cause is clear.
{
"name":"SUM_INTEREST_SCORE2",
"function":{
"expression":"SUM",
"returntype":"decimal(19,undefined)",
"parameter":{
"type":"column",
"value":"USERACTION.INTEREST_SCORE2"
}
}
}
> Error while creating hive table through Kylin build cube with mysql imported
> tables
> ---
>
> Key: KYLIN-3695
> URL: https://issues.apache.org/jira/browse/KYLIN-3695
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
>Reporter: Gaurav Rawat
>Assignee: XiaoXiang Yu
>Priority: Minor
> Attachments: image-2018-12-04-15-53-49-983.png
>
>
> HI I am trying to build a cube with Kylin, the data gets souced fine from
> sqoop but the next step for creating hive tables fails . Looking at the
> command being fired it looks weird as the create statement looks good to me .
> I think the issue is with DOUBLE types as when I remove the same the create
> statement works fine . Can someone please help .
> I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
> The errors logs with commands as as below, the table is a msql table which
> had columns with DOUBLE type
> Command
> {code:java}
> hive -e "USE default; DROP TABLE IF EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 (
> HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string
> ,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK_STOCK_NM string
> ,STOCK_APPROACH string ,STOCK_STOCK_TYP string ,INVESTOR_ID string
> ,INVESTOR_NM string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string
> ,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22)
> ,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22) ) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY '|' STORED AS TEXTFILE LOCATION
> 's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368';
> ALTER TABLE
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET
> TBLPROPERTIES('auto.purge'='true'); " --hiveconf hive.merge.mapredfiles=false
> --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2
> --hiveconf hive.exec.compress.output=true --hiveconf
> hive.auto.convert.join.noconditionaltask=true --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false
> --hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf
> hive.stats.autogather=true{code}
> Error is as below
> {code:java}
> OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds
> MismatchedTokenException(334!=347) at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at
> org.apach
[jira] [Updated] (KYLIN-3709) Pass the wrong parameter to addResource function, kylin can not load the configuration file hbase.hdfs.xml .
[
https://issues.apache.org/jira/browse/KYLIN-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongtao He updated KYLIN-3709:
--
Description:
When calling Configuration.addResource, KYLIN-3648 use a Path as the parameter
instead of a string in Kylin's HBaseConnection.
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
hdfsConf.addResource(new Path(hdfsConfigFile));
{code}
Use a Path as the parameter of Configuration.addResource is better.
Unfortunately, the parameter which passed to the addResource function is
wrong. The addResource function only accepts absolute paths, but the parameter
is just a filename. For example, the value of
hdfsConfigFile is "hbase.hdfs.xml", so addResource function will not work . The
end result is that kylin can not load the hbase configuration file
hbase.hdfs.xml .
There are two ways to fix this bug, and I think method 1 is better.
Method-1.revert the code
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
hdfsConf.addResource(hdfsConfigFile);{code}
Method-2.Get the absolute path of the configuration file
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
String hbaseHdfsConfigPath = System.getProperty("user.dir") + "/../conf/" +
hdfsConfigFile;
hdfsConf.addResource(new Path(hbaseHdfsConfigPath));
{code}
was:
When calling Configuration.addResource,
[KYLIN-3648|https://jira.apache.org/jira/browse/KYLIN-3648] use a Path as the
parameter instead of a string in Kylin's HBaseConnection.
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
hdfsConf.addResource(new Path(hdfsConfigFile));
{code}
Use a Path as the parameter of Configuration.addResource is better.
Unfortunately, the parameter which passed to the addResource function is
wrong. The addResource function only accepts absolute paths, but the parameter
is just a filename. For example, the value of
hdfsConfigFile is "hbase.hdfs.xml", so addResource function will not work . The
end result is that kylin can not load the hbase configuration file
hbase.hdfs.xml .
There are two ways to fix this bug, and I think method 1 is better.
Method-1.revert the code
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
hdfsConf.addResource(hdfsConfigFile);{code}
Method-2.Get the absolute path of the configuration file
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
String hbaseHdfsConfigPath = System.getProperty("user.dir") + "/../conf/" +
hdfsConfigFile;
hdfsConf.addResource(new Path(hbaseHdfsConfigPath));
{code}
> Pass the wrong parameter to addResource function, kylin can not load the
> configuration file hbase.hdfs.xml .
>
>
> Key: KYLIN-3709
> URL: https://issues.apache.org/jira/browse/KYLIN-3709
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.5.1, v2.5.2
>Reporter: Hongtao He
>Assignee: Shaofeng SHI
>Priority: Critical
> Attachments:
> method-1-Pass-the-wrong-parameter-to-addResource-function-kyl.patch,
> method-2-Pass-the-wrong-parameter-to-addResource-function-kyl.patch
>
>
> When calling Configuration.addResource, KYLIN-3648 use a Path as the
> parameter instead of a string in Kylin's HBaseConnection.
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> hdfsConf.addResource(new Path(hdfsConfigFile));
> {code}
> Use a Path as the parameter of Configuration.addResource is better.
> Unfortunately, the parameter which passed to the addResource function is
> wrong. The addResource function only accepts absolute paths, but the
> parameter is just a filename. For example
[jira] [Comment Edited] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
[
https://issues.apache.org/jira/browse/KYLIN-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708395#comment-16708395
]
XiaoXiang Yu edited comment on KYLIN-3695 at 12/4/18 8:46 AM:
--
[~gauravbrills] I debug in my test env.
I found when using decimal data type with empty precision to build a cube. The
front end(JavaScript) will use 'undefined' for null value and send it to
back-end to validate cube metadata. It clear that decimal(19, undefined) do
not match a valid regex pattern, so back end return error message to front end
as my above message.
Source code at
org.apache.kylin.metadata.datatype.DataType.[https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/metadata/datatype/DataType.java]
I think it has no need to fix it because the root cause is clear.
{
"name":"SUM_INTEREST_SCORE2",
"function":{
"expression":"SUM",
"returntype":"decimal(19,undefined)",
"parameter":{
"type":"column",
"value":"USERACTION.INTEREST_SCORE2"
}
}
}
was (Author: hit_lacus):
[~gauravbrills] I debug in my test env.
I found when using decimal data type with empty precision to build a cube. The
front end(JavaScript) will use 'undefined' for null value and send it to
back-end to validate cube metadata. It clear that decimal(19, undefined) do
not match a valid regex pattern, so back end return error message to front end
as my above message.
Source code at
org.apache.kylin.metadata.datatype.DataType.[https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/metadata/datatype/DataType.java]
I think it has no need to fix it because the root cause is clear.
{quote}{
"name":"SUM_INTEREST_SCORE2",
"function":{
"expression":"SUM",
"returntype":"decimal(19,undefined)",
"parameter":
Unknown macro: \{ "type"}
}
}
{quote}
> Error while creating hive table through Kylin build cube with mysql imported
> tables
> ---
>
> Key: KYLIN-3695
> URL: https://issues.apache.org/jira/browse/KYLIN-3695
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
>Reporter: Gaurav Rawat
>Assignee: XiaoXiang Yu
>Priority: Minor
> Attachments: image-2018-12-04-15-53-49-983.png
>
>
> HI I am trying to build a cube with Kylin, the data gets souced fine from
> sqoop but the next step for creating hive tables fails . Looking at the
> command being fired it looks weird as the create statement looks good to me .
> I think the issue is with DOUBLE types as when I remove the same the create
> statement works fine . Can someone please help .
> I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
> The errors logs with commands as as below, the table is a msql table which
> had columns with DOUBLE type
> Command
> {code:java}
> hive -e "USE default; DROP TABLE IF EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 (
> HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string
> ,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK_STOCK_NM string
> ,STOCK_APPROACH string ,STOCK_STOCK_TYP string ,INVESTOR_ID string
> ,INVESTOR_NM string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string
> ,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22)
> ,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22) ) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY '|' STORED AS TEXTFILE LOCATION
> 's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368';
> ALTER TABLE
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET
> TBLPROPERTIES('auto.purge'='true'); " --hiveconf hive.merge.mapredfiles=false
> --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2
> --hiveconf hive.exec.compress.output=true --hiveconf
> hive.auto.convert.join.noconditionaltask=true --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false
> --hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf
> hive.stats.autogather=true{code}
> Error is as below
> {code:java}
> OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds
> MismatchedTokenException(334!=347) at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at
> org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179)
> at
> org.a
[jira] [Updated] (KYLIN-3709) Pass the wrong parameter to addResource function, kylin can not load the configuration file hbase.hdfs.xml .
[
https://issues.apache.org/jira/browse/KYLIN-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongtao He updated KYLIN-3709:
--
Description:
When calling Configuration.addResource, KYLIN-3648 use a Path as the parameter
instead of a string in Kylin's HBaseConnection.
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
hdfsConf.addResource(new Path(hdfsConfigFile));
{code}
Use a Path as the parameter of Configuration.addResource is better.
Unfortunately, the parameter which passed to the addResource function is
wrong. The addResource function only accepts absolute paths, but the parameter
is just a filename. For example, the value of hdfsConfigFile is
"hbase.hdfs.xml", so addResource function will not work . The end result is
that kylin can not load the hbase configuration file hbase.hdfs.xml .
There are two ways to fix this bug, and I think method 1 is better.
Method-1.revert the code
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
hdfsConf.addResource(hdfsConfigFile);{code}
Method-2.Get the absolute path of the configuration file
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
String hbaseHdfsConfigPath = System.getProperty("user.dir") + "/../conf/" +
hdfsConfigFile;
hdfsConf.addResource(new Path(hbaseHdfsConfigPath));
{code}
was:
When calling Configuration.addResource, KYLIN-3648 use a Path as the parameter
instead of a string in Kylin's HBaseConnection.
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
hdfsConf.addResource(new Path(hdfsConfigFile));
{code}
Use a Path as the parameter of Configuration.addResource is better.
Unfortunately, the parameter which passed to the addResource function is
wrong. The addResource function only accepts absolute paths, but the parameter
is just a filename. For example, the value of
hdfsConfigFile is "hbase.hdfs.xml", so addResource function will not work . The
end result is that kylin can not load the hbase configuration file
hbase.hdfs.xml .
There are two ways to fix this bug, and I think method 1 is better.
Method-1.revert the code
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
hdfsConf.addResource(hdfsConfigFile);{code}
Method-2.Get the absolute path of the configuration file
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
String hbaseHdfsConfigPath = System.getProperty("user.dir") + "/../conf/" +
hdfsConfigFile;
hdfsConf.addResource(new Path(hbaseHdfsConfigPath));
{code}
> Pass the wrong parameter to addResource function, kylin can not load the
> configuration file hbase.hdfs.xml .
>
>
> Key: KYLIN-3709
> URL: https://issues.apache.org/jira/browse/KYLIN-3709
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.5.1, v2.5.2
>Reporter: Hongtao He
>Assignee: Shaofeng SHI
>Priority: Critical
> Attachments:
> method-1-Pass-the-wrong-parameter-to-addResource-function-kyl.patch,
> method-2-Pass-the-wrong-parameter-to-addResource-function-kyl.patch
>
>
> When calling Configuration.addResource, KYLIN-3648 use a Path as the
> parameter instead of a string in Kylin's HBaseConnection.
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> hdfsConf.addResource(new Path(hdfsConfigFile));
> {code}
> Use a Path as the parameter of Configuration.addResource is better.
> Unfortunately, the parameter which passed to the addResource function is
> wrong. The addResource function only accepts absolute paths, but the
> parameter is just a filename. For example, the value of hdfsConfigFile is
> "hbase.hdfs.xml", s
[jira] [Assigned] (KYLIN-3680) Spark cubing failed with JDBC resource
[ https://issues.apache.org/jira/browse/KYLIN-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long reassigned KYLIN-3680: Assignee: Chao Long (was: Zhixiong Chen) > Spark cubing failed with JDBC resource > -- > > Key: KYLIN-3680 > URL: https://issues.apache.org/jira/browse/KYLIN-3680 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.4.0, v2.4.1, v2.5.0, v2.5.1 >Reporter: Zhixiong Chen >Assignee: Chao Long >Priority: Major > Fix For: v2.6.0 > > > I find in JdbcHiveMRInput.java , the system save flat table with TextFile. > However in SparkCubingByLayer.java, the system getFlatTableStorageFormat is > SequenceFile .It read the flat table as SequenceFile and failed. > I workaround with add property > "kylin.source.hive.flat-table-storage-format=TEXTFILE" in cube. > But I think maybe we could do it better. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
[
https://issues.apache.org/jira/browse/KYLIN-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708395#comment-16708395
]
XiaoXiang Yu edited comment on KYLIN-3695 at 12/4/18 8:45 AM:
--
[~gauravbrills] I debug in my test env.
I found when using decimal data type with empty precision to build a cube. The
front end(JavaScript) will use 'undefined' for null value and send it to
back-end to validate cube metadata. It clear that decimal(19, undefined) do
not match a valid regex pattern, so back end return error message to front end
as my above message.
Source code at
org.apache.kylin.metadata.datatype.DataType.[https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/metadata/datatype/DataType.java]
I think it has no need to fix it because the root cause is clear.
{quote}{
"name":"SUM_INTEREST_SCORE2",
"function":{
"expression":"SUM",
"returntype":"decimal(19,undefined)",
"parameter":
Unknown macro: \{ "type"}
}
}
{quote}
was (Author: hit_lacus):
[~gauravbrills] I debug in my test env.
I found when using decimal data type with empty precision to build a cube. The
front end(JavaScript) will use 'undefined' for null value and send it to
back-end to validate cube metadata. It clear that decimal(19, undefined) do
not match a valid regex pattern, so back end return error message to front end
as my above message.
I think it has no need to fix it because the root cause is clear.
{quote}{
"name":"SUM_INTEREST_SCORE2",
"function":{
"expression":"SUM",
"returntype":"decimal(19,undefined)",
"parameter":{
"type":"column",
"value":"USERACTION.INTEREST_SCORE2"
}
}
}{quote}
> Error while creating hive table through Kylin build cube with mysql imported
> tables
> ---
>
> Key: KYLIN-3695
> URL: https://issues.apache.org/jira/browse/KYLIN-3695
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
>Reporter: Gaurav Rawat
>Assignee: XiaoXiang Yu
>Priority: Minor
> Attachments: image-2018-12-04-15-53-49-983.png
>
>
> HI I am trying to build a cube with Kylin, the data gets souced fine from
> sqoop but the next step for creating hive tables fails . Looking at the
> command being fired it looks weird as the create statement looks good to me .
> I think the issue is with DOUBLE types as when I remove the same the create
> statement works fine . Can someone please help .
> I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
> The errors logs with commands as as below, the table is a msql table which
> had columns with DOUBLE type
> Command
> {code:java}
> hive -e "USE default; DROP TABLE IF EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 (
> HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string
> ,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK_STOCK_NM string
> ,STOCK_APPROACH string ,STOCK_STOCK_TYP string ,INVESTOR_ID string
> ,INVESTOR_NM string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string
> ,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22)
> ,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22) ) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY '|' STORED AS TEXTFILE LOCATION
> 's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368';
> ALTER TABLE
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET
> TBLPROPERTIES('auto.purge'='true'); " --hiveconf hive.merge.mapredfiles=false
> --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2
> --hiveconf hive.exec.compress.output=true --hiveconf
> hive.auto.convert.join.noconditionaltask=true --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false
> --hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf
> hive.stats.autogather=true{code}
> Error is as below
> {code:java}
> OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds
> MismatchedTokenException(334!=347) at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at
> org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3808)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382)
> at
> or
[jira] [Commented] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
[
https://issues.apache.org/jira/browse/KYLIN-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708395#comment-16708395
]
XiaoXiang Yu commented on KYLIN-3695:
-
[~gauravbrills] I debug in my test env.
I found when using decimal data type with empty precision to build a cube. The
front end(JavaScript) will use 'undefined' for null value and send it to
back-end to validate cube metadata. It clear that decimal(19, undefined) do
not match a valid regex pattern, so back end return error message to front end
as my above message.
I think it has no need to fix it because the root cause is clear.
{quote}{
"name":"SUM_INTEREST_SCORE2",
"function":{
"expression":"SUM",
"returntype":"decimal(19,undefined)",
"parameter":{
"type":"column",
"value":"USERACTION.INTEREST_SCORE2"
}
}
}{quote}
> Error while creating hive table through Kylin build cube with mysql imported
> tables
> ---
>
> Key: KYLIN-3695
> URL: https://issues.apache.org/jira/browse/KYLIN-3695
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
>Reporter: Gaurav Rawat
>Assignee: XiaoXiang Yu
>Priority: Minor
> Attachments: image-2018-12-04-15-53-49-983.png
>
>
> HI I am trying to build a cube with Kylin, the data gets souced fine from
> sqoop but the next step for creating hive tables fails . Looking at the
> command being fired it looks weird as the create statement looks good to me .
> I think the issue is with DOUBLE types as when I remove the same the create
> statement works fine . Can someone please help .
> I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
> The errors logs with commands as as below, the table is a msql table which
> had columns with DOUBLE type
> Command
> {code:java}
> hive -e "USE default; DROP TABLE IF EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 (
> HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string
> ,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK_STOCK_NM string
> ,STOCK_APPROACH string ,STOCK_STOCK_TYP string ,INVESTOR_ID string
> ,INVESTOR_NM string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string
> ,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22)
> ,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22) ) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY '|' STORED AS TEXTFILE LOCATION
> 's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368';
> ALTER TABLE
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET
> TBLPROPERTIES('auto.purge'='true'); " --hiveconf hive.merge.mapredfiles=false
> --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2
> --hiveconf hive.exec.compress.output=true --hiveconf
> hive.auto.convert.join.noconditionaltask=true --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false
> --hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf
> hive.stats.autogather=true{code}
> Error is as below
> {code:java}
> OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds
> MismatchedTokenException(334!=347) at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at
> org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3808)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1333) at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70) at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468) at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1316) at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1456) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1226) at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:3
[jira] [Created] (KYLIN-3709) Pass the wrong parameter to addResource function, kylin can not load the configuration file hbase.hdfs.xml .
Hongtao He created KYLIN-3709:
-
Summary: Pass the wrong parameter to addResource function, kylin
can not load the configuration file hbase.hdfs.xml .
Key: KYLIN-3709
URL: https://issues.apache.org/jira/browse/KYLIN-3709
Project: Kylin
Issue Type: Bug
Components: Job Engine
Affects Versions: v2.5.2, v2.5.1
Reporter: Hongtao He
Assignee: Shaofeng SHI
Attachments:
method-1-Pass-the-wrong-parameter-to-addResource-function-kyl.patch,
method-2-Pass-the-wrong-parameter-to-addResource-function-kyl.patch
When calling Configuration.addResource,
[KYLIN-3648|https://jira.apache.org/jira/browse/KYLIN-3648] use a Path as the
parameter instead of a string in Kylin's HBaseConnection.
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
hdfsConf.addResource(new Path(hdfsConfigFile));
{code}
Use a Path as the parameter of Configuration.addResource is better.
Unfortunately, the parameter which passed to the addResource function is
wrong. The addResource function only accepts absolute paths, but the parameter
is just a filename. For example, the value of
hdfsConfigFile is "hbase.hdfs.xml", so addResource function will not work . The
end result is that kylin can not load the hbase configuration file
hbase.hdfs.xml .
There are two ways to fix this bug, and I think method 1 is better.
Method-1.revert the code
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
hdfsConf.addResource(hdfsConfigFile);{code}
Method-2.Get the absolute path of the configuration file
{code:java}
String hdfsConfigFile =
KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
return;
}
Configuration hdfsConf = new Configuration(false);
String hbaseHdfsConfigPath = System.getProperty("user.dir") + "/../conf/" +
hdfsConfigFile;
hdfsConf.addResource(new Path(hbaseHdfsConfigPath));
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3126) Query fails with "Error while compiling generated Java code" when equality condition is used, and works when equivalent IN clause is specified
[
https://issues.apache.org/jira/browse/KYLIN-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3126:
Fix Version/s: (was: v2.6.0)
> Query fails with "Error while compiling generated Java code" when equality
> condition is used, and works when equivalent IN clause is specified
> --
>
> Key: KYLIN-3126
> URL: https://issues.apache.org/jira/browse/KYLIN-3126
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.2.0
> Environment: HDP 2.5.6, Kylin 2.2.0, sample cube
>Reporter: Vsevolod Ostapenko
>Priority: Major
>
> The following query fails with "Error while compiling generated Java code",
> when equality condition is used {{(d0.year_beg_dt = '2012-01-01')}} and works
> when IN clause is used {{(d0.year_beg_dt in ('2012-01-01'))}}
> {code:sql}
> select
> d2.country,
> count(f.item_count) items_ttl
> from
> kylin_sales f
> join
> kylin_cal_dt d0
> on
> f.part_dt = d0.cal_dt
> join
> kylin_account d1
> on
> f.buyer_id = d1.account_id
> join
> kylin_country d2
> on
> d1.account_country = d2.country
> where
> d0.year_beg_dt = '2012-01-01' -- blows up
> -- d0.year_beg_dt in ('2012-01-01') -- works
> and
> d2.country in ('US', 'JP')
> group by
> d2.country
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3290) Avoid calling Class#newInstance
[ https://issues.apache.org/jira/browse/KYLIN-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3290. - Resolution: Fixed > Avoid calling Class#newInstance > --- > > Key: KYLIN-3290 > URL: https://issues.apache.org/jira/browse/KYLIN-3290 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Labels: jdk > Fix For: v2.6.0 > > > Class#newInstance is deprecated starting in Java 9 - > https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw > undeclared checked exceptions. > The suggested replacement is getDeclaredConstructor().newInstance(), which > wraps the checked exceptions in InvocationException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
[
https://issues.apache.org/jira/browse/KYLIN-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3695:
Attachment: image-2018-12-04-15-53-49-983.png
> Error while creating hive table through Kylin build cube with mysql imported
> tables
> ---
>
> Key: KYLIN-3695
> URL: https://issues.apache.org/jira/browse/KYLIN-3695
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
>Reporter: Gaurav Rawat
>Assignee: XiaoXiang Yu
>Priority: Minor
> Attachments: image-2018-12-04-15-53-49-983.png
>
>
> HI I am trying to build a cube with Kylin, the data gets souced fine from
> sqoop but the next step for creating hive tables fails . Looking at the
> command being fired it looks weird as the create statement looks good to me .
> I think the issue is with DOUBLE types as when I remove the same the create
> statement works fine . Can someone please help .
> I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
> The errors logs with commands as as below, the table is a msql table which
> had columns with DOUBLE type
> Command
> {code:java}
> hive -e "USE default; DROP TABLE IF EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 (
> HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string
> ,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK_STOCK_NM string
> ,STOCK_APPROACH string ,STOCK_STOCK_TYP string ,INVESTOR_ID string
> ,INVESTOR_NM string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string
> ,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22)
> ,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22) ) ROW FORMAT DELIMITED FIELDS
> TERMINATED BY '|' STORED AS TEXTFILE LOCATION
> 's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368';
> ALTER TABLE
> kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET
> TBLPROPERTIES('auto.purge'='true'); " --hiveconf hive.merge.mapredfiles=false
> --hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2
> --hiveconf hive.exec.compress.output=true --hiveconf
> hive.auto.convert.join.noconditionaltask=true --hiveconf
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false
> --hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf
> hive.stats.autogather=true{code}
> Error is as below
> {code:java}
> OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds
> MismatchedTokenException(334!=347) at
> org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
> at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at
> org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3808)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382)
> at
> org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1333) at
> org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77) at
> org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70) at
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468) at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1316) at
> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1456) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236) at
> org.apache.hadoop.hive.ql.Driver.run(Driver.java:1226) at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787) at
> org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at
> org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
> org.apache.hadoop.util.RunJar.main(RunJar.java:136) FAILED: ParseException
> line 15:42 mismatched input '(' expecting ) near 'double' in c
