[jira] [Commented] (KYLIN-3887) Query with decimal sum measure of double complied failed after KYLIN-3703
[
https://issues.apache.org/jira/browse/KYLIN-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839060#comment-16839060
]
Temple Zhou commented on KYLIN-3887:
Before KYLIN-2982 update_sum_measure_default_return_type,
cubeMeasures.js:
{code:javascript}
switch($scope.newMeasure.function.expression){
case "SUM":
if(colType==="tinyint"||colType==="smallint"||colType==="int"||colType==="bigint"||colType==="integer"){
$scope.newMeasure.function.returntype= 'bigint';
}else{
if(colType.indexOf('decimal')!=-1||colType==="double"||colType==="float"){
$scope.newMeasure.function.returntype= 'decimal(19,4)';
}else{
$scope.newMeasure.function.returntype= 'decimal(14,0)';
}
}
break;
{code}
this method will set the return type to decimal when your column type is double.
So, the temporary solution is to reedit the cube via WEB UI(new version), and
rebuild the cube.
> Query with decimal sum measure of double complied failed after KYLIN-3703
> -
>
> Key: KYLIN-3887
> URL: https://issues.apache.org/jira/browse/KYLIN-3887
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: nichunen
>Priority: Major
> Fix For: Future, v3.0.0
>
> Attachments: image-2019-05-14-11-19-05-514.png
>
>
> After KYLIN-3703, Query with decimal sum measure of double complied failed.
> {code:java}
> Caused by: org.codehaus.commons.compiler.CompileException:
> Line 112, Column 42: Cannot cast "java.math.BigDecimal" to
> "java.lang.Double"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] nichunen merged pull request #639: Add unit tests three
nichunen merged pull request #639: Add unit tests three URL: https://github.com/apache/kylin/pull/639 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-3887) Query with decimal sum measure of double complied failed after KYLIN-3703
[
https://issues.apache.org/jira/browse/KYLIN-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Temple Zhou updated KYLIN-3887:
---
Attachment: image-2019-05-14-11-19-05-514.png
> Query with decimal sum measure of double complied failed after KYLIN-3703
> -
>
> Key: KYLIN-3887
> URL: https://issues.apache.org/jira/browse/KYLIN-3887
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: nichunen
>Priority: Major
> Fix For: Future, v3.0.0
>
> Attachments: image-2019-05-14-11-19-05-514.png
>
>
> After KYLIN-3703, Query with decimal sum measure of double complied failed.
> {code:java}
> Caused by: org.codehaus.commons.compiler.CompileException:
> Line 112, Column 42: Cannot cast "java.math.BigDecimal" to
> "java.lang.Double"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3983) Add extra metadata for measure
[ https://issues.apache.org/jira/browse/KYLIN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839030#comment-16839030 ] ASF subversion and git services commented on KYLIN-3983: Commit 5ebcee696d29dc0bfb584e1cde72570a56e2a11a in kylin's branch refs/heads/dynamic-measure from yuzhang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=5ebcee6 ] KYLIN-3983 Add extra metadata for measure. Add test case and example/test_case_data for MeasureManager. Add measure metadata in sample cube > Add extra metadata for measure > -- > > Key: KYLIN-3983 > URL: https://issues.apache.org/jira/browse/KYLIN-3983 > Project: Kylin > Issue Type: Sub-task > Components: Metadata >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Minor > > Just like CubeDesc and CubeInstance, we need to add extra metadata for > measure to persist some runtime data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3983) Add extra metadata for measure
[ https://issues.apache.org/jira/browse/KYLIN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839029#comment-16839029 ] ASF GitHub Bot commented on KYLIN-3983: --- nichunen commented on pull request #638: KYLIN-3983 Add extra metadata for measure. Add test case and example/… URL: https://github.com/apache/kylin/pull/638 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add extra metadata for measure > -- > > Key: KYLIN-3983 > URL: https://issues.apache.org/jira/browse/KYLIN-3983 > Project: Kylin > Issue Type: Sub-task > Components: Metadata >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Minor > > Just like CubeDesc and CubeInstance, we need to add extra metadata for > measure to persist some runtime data. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] nichunen merged pull request #638: KYLIN-3983 Add extra metadata for measure. Add test case and example/…
nichunen merged pull request #638: KYLIN-3983 Add extra metadata for measure. Add test case and example/… URL: https://github.com/apache/kylin/pull/638 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3887) Query with decimal sum measure of double complied failed after KYLIN-3703
[
https://issues.apache.org/jira/browse/KYLIN-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839002#comment-16839002
]
Temple Zhou commented on KYLIN-3887:
[~liushaohui] [~nichunen]
I will review my patch soon.
> Query with decimal sum measure of double complied failed after KYLIN-3703
> -
>
> Key: KYLIN-3887
> URL: https://issues.apache.org/jira/browse/KYLIN-3887
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: nichunen
>Priority: Major
> Fix For: Future, v3.0.0
>
>
> After KYLIN-3703, Query with decimal sum measure of double complied failed.
> {code:java}
> Caused by: org.codehaus.commons.compiler.CompileException:
> Line 112, Column 42: Cannot cast "java.math.BigDecimal" to
> "java.lang.Double"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] codecov-io commented on issue #639: Add unit tests three
codecov-io commented on issue #639: Add unit tests three URL: https://github.com/apache/kylin/pull/639#issuecomment-491922284 # [Codecov](https://codecov.io/gh/apache/kylin/pull/639?src=pr=h1) Report > Merging [#639](https://codecov.io/gh/apache/kylin/pull/639?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/c39b238fb2f4999f5484a7d2fa0b55245db5534c?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/kylin/pull/639?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#639 +/- ## === + Coverage 25.48% 25.5% +0.01% - Complexity 58925900 +8 === Files 13791379 Lines 81871 81871 Branches 11479 11479 === + Hits 20863 20878 +15 + Misses58986 58964 -22 - Partials 20222029 +7 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/639?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/639/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==) | `72.97% <0%> (-5.41%)` | `6% <0%> (-1%)` | | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/639/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `69.3% <0%> (-0.92%)` | `7% <0%> (ø)` | | | [...apache/kylin/engine/mr/common/HadoopCmdOutput.java](https://codecov.io/gh/apache/kylin/pull/639/diff?src=pr=tree#diff-ZW5naW5lLW1yL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvbXIvY29tbW9uL0hhZG9vcENtZE91dHB1dC5qYXZh) | `14.7% <0%> (+14.7%)` | `4% <0%> (+4%)` | :arrow_up: | | [...ylin/engine/mr/common/DefaultX509TrustManager.java](https://codecov.io/gh/apache/kylin/pull/639/diff?src=pr=tree#diff-ZW5naW5lLW1yL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvbXIvY29tbW9uL0RlZmF1bHRYNTA5VHJ1c3RNYW5hZ2VyLmphdmE=) | `35.71% <0%> (+35.71%)` | `5% <0%> (+5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/639?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/639?src=pr=footer). Last update [c39b238...4c5f7f1](https://codecov.io/gh/apache/kylin/pull/639?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] coveralls commented on issue #639: Add unit tests three
coveralls commented on issue #639: Add unit tests three URL: https://github.com/apache/kylin/pull/639#issuecomment-491921501 ## Pull Request Test Coverage Report for [Build 4490](https://coveralls.io/builds/23356161) * **0** of **0** changed or added relevant lines in **0** files are covered. * **6** unchanged lines in **2** files lost coverage. * Overall coverage increased (+**0.03%**) to **27.984%** --- | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/23356161/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/23356161/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L439) | 5 | 77.81% | | Totals | [](https://coveralls.io/builds/23356161) | | :-- | --: | | Change from base [Build 4487](https://coveralls.io/builds/23340471): | 0.03% | | Covered Lines: | 22911 | | Relevant Lines: | 81871 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] asfgit commented on issue #639: Add unit tests three
asfgit commented on issue #639: Add unit tests three URL: https://github.com/apache/kylin/pull/639#issuecomment-491899719 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] TheRealHaui opened a new pull request #639: Add unit tests three
TheRealHaui opened a new pull request #639: Add unit tests three URL: https://github.com/apache/kylin/pull/639 Added additional Unit Tests to increase code coverage. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-2620) Check for "ORDER BY LIMIT" clause when rewrite SUM query as TOPN
[
https://issues.apache.org/jira/browse/KYLIN-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838674#comment-16838674
]
Na Zhai commented on KYLIN-2620:
Use this SQL test: {code:sql}SELECT SELLER_ID, SUM(PRICE) FROM KYLIN_SALES
group by SELLER_ID
order by SUM(PRICE), seller_id DESC limit 10;{code} It did not hit the top_n
measure.
!sql.png!
!cube_desc.png!
> Check for "ORDER BY LIMIT" clause when rewrite SUM query as TOPN
>
>
> Key: KYLIN-2620
> URL: https://issues.apache.org/jira/browse/KYLIN-2620
> Project: Kylin
> Issue Type: Bug
> Components: Measure - TopN
>Reporter: Lin Tingmao
>Assignee: Chao Long
>Priority: Major
> Fix For: Future, v3.0.0
>
> Attachments: cube_desc.png, sql.png
>
>
> When running the following query
> select sum(measure) from table group by col_id
> if there exists TOPN(measure, group by col_id) measure,
> TopNMeasureType.isTopNCompatibleSum()will pass, so the SUM is rewritten
> to TOPN. This confuses the user since they may expect a accurate result for
> every distinct value of group by column(s).
> Kylin should check if "ORDER BY col_id LIMIT topncapacity" is present in the
> query to determine whether to rewrite.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-2620) Check for "ORDER BY LIMIT" clause when rewrite SUM query as TOPN
[ https://issues.apache.org/jira/browse/KYLIN-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Zhai updated KYLIN-2620: --- Attachment: cube_desc.png > Check for "ORDER BY LIMIT" clause when rewrite SUM query as TOPN > > > Key: KYLIN-2620 > URL: https://issues.apache.org/jira/browse/KYLIN-2620 > Project: Kylin > Issue Type: Bug > Components: Measure - TopN >Reporter: Lin Tingmao >Assignee: Chao Long >Priority: Major > Fix For: Future, v3.0.0 > > Attachments: cube_desc.png, sql.png > > > When running the following query > select sum(measure) from table group by col_id > if there exists TOPN(measure, group by col_id) measure, > TopNMeasureType.isTopNCompatibleSum()will pass, so the SUM is rewritten > to TOPN. This confuses the user since they may expect a accurate result for > every distinct value of group by column(s). > Kylin should check if "ORDER BY col_id LIMIT topncapacity" is present in the > query to determine whether to rewrite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2620) Check for "ORDER BY LIMIT" clause when rewrite SUM query as TOPN
[ https://issues.apache.org/jira/browse/KYLIN-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Zhai updated KYLIN-2620: --- Attachment: sql.png > Check for "ORDER BY LIMIT" clause when rewrite SUM query as TOPN > > > Key: KYLIN-2620 > URL: https://issues.apache.org/jira/browse/KYLIN-2620 > Project: Kylin > Issue Type: Bug > Components: Measure - TopN >Reporter: Lin Tingmao >Assignee: Chao Long >Priority: Major > Fix For: Future, v3.0.0 > > Attachments: cube_desc.png, sql.png > > > When running the following query > select sum(measure) from table group by col_id > if there exists TOPN(measure, group by col_id) measure, > TopNMeasureType.isTopNCompatibleSum()will pass, so the SUM is rewritten > to TOPN. This confuses the user since they may expect a accurate result for > every distinct value of group by column(s). > Kylin should check if "ORDER BY col_id LIMIT topncapacity" is present in the > query to determine whether to rewrite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (KYLIN-2620) Check for "ORDER BY LIMIT" clause when rewrite SUM query as TOPN
[ https://issues.apache.org/jira/browse/KYLIN-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen reopened KYLIN-2620: - > Check for "ORDER BY LIMIT" clause when rewrite SUM query as TOPN > > > Key: KYLIN-2620 > URL: https://issues.apache.org/jira/browse/KYLIN-2620 > Project: Kylin > Issue Type: Bug > Components: Measure - TopN >Reporter: Lin Tingmao >Assignee: Chao Long >Priority: Major > Fix For: Future, v3.0.0 > > > When running the following query > select sum(measure) from table group by col_id > if there exists TOPN(measure, group by col_id) measure, > TopNMeasureType.isTopNCompatibleSum()will pass, so the SUM is rewritten > to TOPN. This confuses the user since they may expect a accurate result for > every distinct value of group by column(s). > Kylin should check if "ORDER BY col_id LIMIT topncapacity" is present in the > query to determine whether to rewrite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3965) When using DriverManager - No suitable driver found for jdbc:kylin://
[ https://issues.apache.org/jira/browse/KYLIN-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838431#comment-16838431 ] ASF subversion and git services commented on KYLIN-3965: Commit fd7ca25f869419034f3cd52381db71bebaaf9153 in kylin's branch refs/heads/2.6.x-hadoop3.1 from alexandr.sidorchuk [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=fd7ca25 ] KYLIN-3965 JDBC fix. - Driver manager compability. - Add guava to uber-jar for standalone usage. > When using DriverManager - No suitable driver found for jdbc:kylin:// > - > > Key: KYLIN-3965 > URL: https://issues.apache.org/jira/browse/KYLIN-3965 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v2.6.1 >Reporter: Alexander >Assignee: Alexander >Priority: Minor > Fix For: v2.6.2 > > Attachments: KYLIN-3965.master.001.patch > > > Caused by: java.sql.SQLException: No suitable driver found for jdbc:kylin:// > > This is because META-INF/services/java.sql.Driver got incorrect name > org.apache.calcite.avatica.remote.Driver -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3926) Should set sourceRecordCount when updating statistics
[ https://issues.apache.org/jira/browse/KYLIN-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838434#comment-16838434 ] ASF subversion and git services commented on KYLIN-3926: Commit edab859ee51128f94d4e192abc56dc1171c468a9 in kylin's branch refs/heads/2.6.x-hadoop3.1 from Chunen Ni [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=edab859 ] KYLIN-3926 Code review > Should set sourceRecordCount when updating statistics > - > > Key: KYLIN-3926 > URL: https://issues.apache.org/jira/browse/KYLIN-3926 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v2.6.2 > > > Currently there're 3 parts to write cuboids statistics: > * SaveStatisticsStep > * MergeStatisticsStep > * MergeStatisticsWithOldStep > With feature [KYLIN-3453], the sourceRecordCount is introduced. However, only > *SaveStatisticsStep* set this value. The other two should also be updated > with this value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3968) Customized precision doesn't work in web
[ https://issues.apache.org/jira/browse/KYLIN-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838430#comment-16838430 ] ASF subversion and git services commented on KYLIN-3968: Commit 20b3721a88e52df447ca640236526d545145da11 in kylin's branch refs/heads/2.6.x-hadoop3.1 from majie [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=20b3721 ] KYLIN-3968 Customized precision doesn't work in web > Customized precision doesn't work in web > > > Key: KYLIN-3968 > URL: https://issues.apache.org/jira/browse/KYLIN-3968 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: Jack >Priority: Minor > Fix For: v2.6.2 > > > In the cubeMeasures.js, It will withdraw precision and scale by using Regular > Expression. The scale parameter is ok, but precision use the magic number 19. > So we fixed it, In cubeMeasures.js, around line 469: > “var precision = 19;” --> "var precision = returnValue[2] || 0;" > and we test ok including building cube and querying when the column is > decimal(38,18). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3518) Coprocessor reports NPE when execute a query on HBase 2.0
[
https://issues.apache.org/jira/browse/KYLIN-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838437#comment-16838437
]
ASF subversion and git services commented on KYLIN-3518:
Commit 0e02c256b568fc332d27ff1ab4c3ff397d3a8386 in kylin's branch
refs/heads/2.6.x-hadoop3.1 from Cheng Wang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=0e02c25 ]
KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT
KYLIN-3518 Fix Coprocessor NPE problem on hbase 2
Signed-off-by: shaofengshi
KYLIN-2565 Add cdh60 profile
Signed-off-by: shaofengshi
> Coprocessor reports NPE when execute a query on HBase 2.0
> -
>
> Key: KYLIN-3518
> URL: https://issues.apache.org/jira/browse/KYLIN-3518
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.5.0
>
>
> On HDP 3.0, build a cube and then run a simple count query, NPE occurred:
>
> {code:java}
> 2018-08-28 01:30:16,969 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 9 ms, use 81000 ms as timeout
> for coprocessor
> 2018-08-28 01:30:16,983 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseEndpointRPC:141 : Serialized scanRequestBytes 522 bytes,
> rawScanBytesString 44 bytes
> 2018-08-28 01:30:16,984 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseEndpointRPC:143 : The scan 67b41fc6 for segment
> kylin_sales_cube_clone[2012010100_2013010100] is as below with 1
> separate raw scans, shard part of start/end key is set to 0
> 2018-08-28 01:30:16,991 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseRPC:288 : Visiting hbase table KYLIN_5Q088VO5I0: cuboid require
> post aggregation, from 0 to 16384 Start:
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00) Stop:
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\xFF\xFF\xFF\x00
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\xFF\xFF\xFF\x00), No Fuzzy Key
> 2018-08-28 01:30:16,991 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseEndpointRPC:148 : Submitting rpc to 1 shards starting from shard
> 0, scan range count 1
> 2018-08-28 01:30:17,010 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> common.KylinConfig:332 : Loading kylin-defaults.properties from
> file:/root/shaofengshi/apache-kylin-2.5.0-SNAPSHOT-bin/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.5.0-SNAPSHOT.jar!/kylin-defaults.properties
> 2018-08-28 01:30:17,033 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> common.KylinConfig:291 : KYLIN_CONF property was not set, will seek
> KYLIN_HOME env variable
> 2018-08-28 01:30:17,051 INFO [pool-14-thread-1] hbase.HBaseConnection:110 :
> Creating coprocessor thread pool with max of 2048, core of 2048
> 2018-08-28 01:30:17,094 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> gtrecord.SequentialCubeTupleIterator:73 : Using SortedIteratorMergerWithLimit
> to merge segment results
> 2018-08-28 01:30:17,097 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> enumerator.OLAPEnumerator:117 : return TupleIterator...
> 2018-08-28 01:30:21,607 INFO [kylin-coproc--pool9-t1]
> client.RpcRetryingCallerImpl:134 : Call exception, tries=6, retries=6,
> started=4410 ms ago, cancelled=false, msg=java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:468)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: java.lang.NullPointerException
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:253)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:)
> at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8032)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2426)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2408)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42010)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> ... 3 more
> , details=row '' on table 'KYLIN_5Q088VO5I0' at
> region=KYLIN_5Q088VO5I0,,1535417272444.27b82cb4702db4557a98b9a7e60b7692.,
> hostname=ignite03.com,16020,1534313612401, seqNum=2
> 2018-08-28 01:30:25,633 INFO [kylin-coproc--pool9-t1]
> client.RpcRetryingCallerImpl:134 : Call
[jira] [Commented] (KYLIN-2565) Support Hadoop 3.0
[ https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838438#comment-16838438 ] ASF subversion and git services commented on KYLIN-2565: Commit 0e02c256b568fc332d27ff1ab4c3ff397d3a8386 in kylin's branch refs/heads/2.6.x-hadoop3.1 from Cheng Wang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=0e02c25 ] KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT KYLIN-3518 Fix Coprocessor NPE problem on hbase 2 Signed-off-by: shaofengshi KYLIN-2565 Add cdh60 profile Signed-off-by: shaofengshi > Support Hadoop 3.0 > -- > > Key: KYLIN-2565 > URL: https://issues.apache.org/jira/browse/KYLIN-2565 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: Wang Cheng >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.5.0 > > > Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below > is the Hadoop3.0 components requirements: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2565) Support Hadoop 3.0
[ https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838440#comment-16838440 ] ASF subversion and git services commented on KYLIN-2565: Commit 8fecd3b2c69a9e17f02b539a6270ef4f13f1b665 in kylin's branch refs/heads/2.6.x-hadoop3.1 from Chunen Ni [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=8fecd3b ] KYLIN-2565 Fix use of HBaseAdmin > Support Hadoop 3.0 > -- > > Key: KYLIN-2565 > URL: https://issues.apache.org/jira/browse/KYLIN-2565 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: Wang Cheng >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.5.0 > > > Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below > is the Hadoop3.0 components requirements: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3926) Should set sourceRecordCount when updating statistics
[ https://issues.apache.org/jira/browse/KYLIN-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838433#comment-16838433 ] ASF subversion and git services commented on KYLIN-3926: Commit dff3a3c3c9dd2802e44de8273b785339ce232441 in kylin's branch refs/heads/2.6.x-hadoop3.1 from kyotoYaho [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=dff3a3c ] KYLIN-3926 set sourceRecordCount when updating statistics > Should set sourceRecordCount when updating statistics > - > > Key: KYLIN-3926 > URL: https://issues.apache.org/jira/browse/KYLIN-3926 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v2.6.2 > > > Currently there're 3 parts to write cuboids statistics: > * SaveStatisticsStep > * MergeStatisticsStep > * MergeStatisticsWithOldStep > With feature [KYLIN-3453], the sourceRecordCount is introduced. However, only > *SaveStatisticsStep* set this value. The other two should also be updated > with this value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3934) sqoop import param '--null-string' result in null value become blank string in hive table
[
https://issues.apache.org/jira/browse/KYLIN-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838435#comment-16838435
]
ASF subversion and git services commented on KYLIN-3934:
Commit 53123d3e5250041203f1c815ed93533bd03a7d2b in kylin's branch
refs/heads/2.6.x-hadoop3.1 from wanghao
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=53123d3 ]
KYLIN-3934 add config for sqoop config null-string and null-non-string
> sqoop import param '--null-string' result in null value become blank string
> in hive table
> -
>
> Key: KYLIN-3934
> URL: https://issues.apache.org/jira/browse/KYLIN-3934
> Project: Kylin
> Issue Type: Bug
> Components: Others
>Affects Versions: v2.6.0
>Reporter: wanghao
>Priority: Major
> Fix For: v2.6.2
>
>
> when column value from jdbc is null, sqoop will write it into hive table with
> blank string.
> eg
> jdbc:
> A | B
> 1 | 1
> 2 | 2
> a | null
>
> hive table:
> A | B
> 1 | 1
> 2 | 2
> a |
> because of this, when I use count(distinct B), it return 3, not 2, and it can
> lead to other problems
>
>
> {code:java}
> String cmd = String.format(Locale.ROOT,
> "%s/bin/sqoop import" + generateSqoopConfigArgString()
> + "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND
> \\$CONDITIONS\" "
> + "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
> + "--fields-terminated-by '%s' --num-mappers %d",
> sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql,
> jobWorkingDir, hiveTable,
> splitColumn, bquery, filedDelimiter, mapperNum);
> {code}
> the param '–null=string' should be '
> \\N' instead of blank string ''.
> I resolved this problem by replace the param. But it needs be configured in
> kylin.properties
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3995) config data type may make oom
[ https://issues.apache.org/jira/browse/KYLIN-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838432#comment-16838432 ] ASF subversion and git services commented on KYLIN-3995: Commit 85c7388f2f92d907e03c537cf49ea2b42e61b94e in kylin's branch refs/heads/2.6.x-hadoop3.1 from Zhixiong Chen [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=85c7388 ] KYLIN-3995 config data type may make oom > config data type may make oom > - > > Key: KYLIN-3995 > URL: https://issues.apache.org/jira/browse/KYLIN-3995 > Project: Kylin > Issue Type: Bug > Components: Others >Reporter: Zhixiong Chen >Assignee: Zhixiong Chen >Priority: Major > Fix For: v2.6.2 > > > getTrieDictionaryForestMaxTrieSizeMB() data type is int > however in TrieDictionaryForestBuilder "maxTrieTreeSizeMB * 1024 * 1024" > it could over integer.max -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2565) Support Hadoop 3.0
[ https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838436#comment-16838436 ] ASF subversion and git services commented on KYLIN-2565: Commit 0e02c256b568fc332d27ff1ab4c3ff397d3a8386 in kylin's branch refs/heads/2.6.x-hadoop3.1 from Cheng Wang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=0e02c25 ] KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT KYLIN-3518 Fix Coprocessor NPE problem on hbase 2 Signed-off-by: shaofengshi KYLIN-2565 Add cdh60 profile Signed-off-by: shaofengshi > Support Hadoop 3.0 > -- > > Key: KYLIN-2565 > URL: https://issues.apache.org/jira/browse/KYLIN-2565 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: Wang Cheng >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.5.0 > > > Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below > is the Hadoop3.0 components requirements: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3537) Use Spark to build Cube on Yarn faild at Setp8 on HDP3.0
[ https://issues.apache.org/jira/browse/KYLIN-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838439#comment-16838439 ] ASF subversion and git services commented on KYLIN-3537: Commit 977d30e99fb2657d512d2141cd29c5937b9340ed in kylin's branch refs/heads/2.6.x-hadoop3.1 from alexandr.sidorchuk [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=977d30e ] KYLIN-3537 Use Spark to build Cube on Yarn faild at Setp8 on HDP3.0 add HBase dependicies to avoid java.lang.NoClassDefFoundError Signed-off-by: shaofengshi > Use Spark to build Cube on Yarn faild at Setp8 on HDP3.0 > > > Key: KYLIN-3537 > URL: https://issues.apache.org/jira/browse/KYLIN-3537 > Project: Kylin > Issue Type: Bug > Environment: HDP3.0 >Reporter: Lijun Cao >Assignee: Alexander >Priority: Major > Fix For: v2.6.1 > > Attachments: KYLIN-3537.master.001.patch > > > *The log on Yarn:* > 18/09/05 03:34:00 INFO scheduler.DAGScheduler: Job 0 failed: > saveAsNewAPIHadoopDataset at SparkCubeHFile.java:227, took 13.734642 s > 18/09/05 03:34:00 ERROR yarn.ApplicationMaster: User class threw exception: > java.lang.RuntimeException: error execute > org.apache.kylin.storage.hbase.steps.SparkCubeHFile > java.lang.RuntimeException: error execute > org.apache.kylin.storage.hbase.steps.SparkCubeHFile > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) > at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:636) > Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: > Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in > stage 1.0 (TID 14, ignite02.com, executor 2): java.lang.NoClassDefFoundError: > org/apache/hadoop/hbase/io/hfile/HFileWriterImpl > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.createRecordWriter(HFileOutputFormat2.java:209) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.getRecordWriter(HFileOutputFormat2.java:181) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1119) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Driver stacktrace: > at > org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802) > at scala.Option.foreach(Option.scala:257) > at > org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605) > at > org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594) > at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) > at > org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1928) > at
[jira] [Updated] (KYLIN-3842) kylinProperties.js Unable to get the public configuration of the first line in the front end
[
https://issues.apache.org/jira/browse/KYLIN-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-3842:
Fix Version/s: (was: v2.6.2)
v3.0.0
Future
> kylinProperties.js Unable to get the public configuration of the first line
> in the front end
>
>
> Key: KYLIN-3842
> URL: https://issues.apache.org/jira/browse/KYLIN-3842
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Affects Versions: v2.5.2
>Reporter: Yuzhang QIU
>Assignee: Yuzhang QIU
>Priority: Minor
> Fix For: Future, v3.0.0
>
> Attachments: KYLIN-3842.master.001.patch
>
>
> Hi dear team:
> I'm developing OLAP Platform based on Kylin2.5.2. During my work, I found
> that kylinProperties.js:37(getProperty(name)) can't get the property of the
> first line in the '_config' which initialized through /admin/public_config.
> For example, the public config is
> 'kylin.restclient.connection.default-max-per-route=20\nkylin.restclient.connection.max-total=200\nkylin.engine.default=2\nkylin.storage.default=2\n
> kylin.web.hive-limit=20\nkylin.web.help.length=4\n'. I expected to get 20
> but got '' when I want to get config by key
> 'kylin.restclient.connection.default-max-per-route'. This problem caused by
> 'var keyIndex = _config.indexOf('\n' + name + '=');'(at
> kylinProperties.js:37) return -1 for those names before which don't have an
> \n(at the first line).
> Then, I debug the AdminService.java, KylinConfig.java and found that the
> KylinConfig.java:517(around this line, in method
> exportToString(Collection propertyKeys)) build the public config
> string with a char '\n' after each property, which cause the first property
> don't has '\n' before it.
> Those are what I found, which will cause problem for developers.
> How do you think?
> Best regard
> yuzhang
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3887) Query with decimal sum measure of double complied failed after KYLIN-3703
[
https://issues.apache.org/jira/browse/KYLIN-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen reassigned KYLIN-3887:
---
Assignee: nichunen
> Query with decimal sum measure of double complied failed after KYLIN-3703
> -
>
> Key: KYLIN-3887
> URL: https://issues.apache.org/jira/browse/KYLIN-3887
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: nichunen
>Priority: Major
> Fix For: Future, v3.0.0
>
>
> After KYLIN-3703, Query with decimal sum measure of double complied failed.
> {code:java}
> Caused by: org.codehaus.commons.compiler.CompileException:
> Line 112, Column 42: Cannot cast "java.math.BigDecimal" to
> "java.lang.Double"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3887) Query with decimal sum measure of double complied failed after KYLIN-3703
[
https://issues.apache.org/jira/browse/KYLIN-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-3887:
Fix Version/s: Future
> Query with decimal sum measure of double complied failed after KYLIN-3703
> -
>
> Key: KYLIN-3887
> URL: https://issues.apache.org/jira/browse/KYLIN-3887
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Priority: Major
> Fix For: Future, v3.0.0
>
>
> After KYLIN-3703, Query with decimal sum measure of double complied failed.
> {code:java}
> Caused by: org.codehaus.commons.compiler.CompileException:
> Line 112, Column 42: Cannot cast "java.math.BigDecimal" to
> "java.lang.Double"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3887) Query with decimal sum measure of double complied failed after KYLIN-3703
[
https://issues.apache.org/jira/browse/KYLIN-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-3887:
Fix Version/s: (was: v2.6.2)
v3.0.0
> Query with decimal sum measure of double complied failed after KYLIN-3703
> -
>
> Key: KYLIN-3887
> URL: https://issues.apache.org/jira/browse/KYLIN-3887
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Priority: Major
> Fix For: v3.0.0
>
>
> After KYLIN-3703, Query with decimal sum measure of double complied failed.
> {code:java}
> Caused by: org.codehaus.commons.compiler.CompileException:
> Line 112, Column 42: Cannot cast "java.math.BigDecimal" to
> "java.lang.Double"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3887) Query with decimal sum measure of double complied failed after KYLIN-3703
[
https://issues.apache.org/jira/browse/KYLIN-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838342#comment-16838342
]
nichunen commented on KYLIN-3887:
-
[~temple.zhou] Hi, any comment?
> Query with decimal sum measure of double complied failed after KYLIN-3703
> -
>
> Key: KYLIN-3887
> URL: https://issues.apache.org/jira/browse/KYLIN-3887
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Priority: Major
> Fix For: v2.6.2
>
>
> After KYLIN-3703, Query with decimal sum measure of double complied failed.
> {code:java}
> Caused by: org.codehaus.commons.compiler.CompileException:
> Line 112, Column 42: Cannot cast "java.math.BigDecimal" to
> "java.lang.Double"{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3892) Set cubing job priority
[ https://issues.apache.org/jira/browse/KYLIN-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-3892. - Resolution: Fixed > Set cubing job priority > --- > > Key: KYLIN-3892 > URL: https://issues.apache.org/jira/browse/KYLIN-3892 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Affects Versions: v2.4.0, v2.5.0, v2.6.0 >Reporter: Temple Zhou >Assignee: Temple Zhou >Priority: Minor > Fix For: v2.6.2 > > > The cubing job with high priority will be delayed when there are too many > tasks running. > So I want to set the job priority for the important cubing jobs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3893) Cube build failed for wrong row key column description
[
https://issues.apache.org/jira/browse/KYLIN-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-3893:
Fix Version/s: (was: v2.6.2)
Future
> Cube build failed for wrong row key column description
> --
>
> Key: KYLIN-3893
> URL: https://issues.apache.org/jira/browse/KYLIN-3893
> Project: Kylin
> Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: Future
>
>
> User created wrong RowKeyColDesc, eg,
> RowKeyColDesc\{column=MYSQL_FEEDBACK_USER_AUDIT.DATE,
> encoding=integer:undefined}
> which cause the cube build forever.
>
> {code:java}
> org.apache.kylin.engine.mr.exception.HadoopShellException:
> java.lang.NumberFormatException: For input string: "undefined" at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:580) at
> java.lang.Integer.parseInt(Integer.java:615) at
> org.apache.kylin.dimension.IntegerDimEnc$Factory.createDimensionEncoding(IntegerDimEnc.java:65)
> at
> org.apache.kylin.dimension.DimensionEncodingFactory.create(DimensionEncodingFactory.java:65)
> at org.apache.kylin.cube.kv.CubeDimEncMap.get(CubeDimEncMap.java:74)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMapFromRowCount(CubeStatsReader.java:206)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMap(CubeStatsReader.java:170)
> at
> org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:102)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92) at
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)result code:2 at
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3917) Add max segment merge span to cleanup intermediate data of cube building
[ https://issues.apache.org/jira/browse/KYLIN-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-3917: Fix Version/s: (was: v2.6.2) Future > Add max segment merge span to cleanup intermediate data of cube building > - > > Key: KYLIN-3917 > URL: https://issues.apache.org/jira/browse/KYLIN-3917 > Project: Kylin > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Major > Fix For: Future > > > Currently the intermediate data of building cube can not be deleted for it > maybe used for later cubing merging. But it result in double space used in > HDFS. > In actual scenario, we only need month-level segment span in maximum. > So if a span of segment is larger than a month, we think it don't need be > merged and the intermediate data can be deleted. > So we can add a config kylin.cube.max-segment-merge.span, default is -1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3930) ArrayIndexOutOfBoundsException when building
[
https://issues.apache.org/jira/browse/KYLIN-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen resolved KYLIN-3930.
-
Resolution: Fixed
> ArrayIndexOutOfBoundsException when building
>
>
> Key: KYLIN-3930
> URL: https://issues.apache.org/jira/browse/KYLIN-3930
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: all
>Reporter: Jacky Woo
>Priority: Major
> Fix For: v2.6.2
>
> Attachments: KYLIN-3930.master.01.patch
>
>
> h2. ArrayIndexOutOfBoundsException when building.
> I hive a cube building error with kylin-2.5.0:
> {code:java}
> 2019-03-31 02:45:18,460 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}
> I checked the code of "NDCuboidBuilder.buildKeyInternal" method
> {code:java}
> private void buildKeyInternal(Cuboid parentCuboid, Cuboid childCuboid,
> ByteArray[] splitBuffers, ByteArray newKeyBodyBuf) {
> RowKeyEncoder rowkeyEncoder =
> rowKeyEncoderProvider.getRowkeyEncoder(childCuboid);
> // rowkey columns
> long mask = Long.highestOneBit(parentCuboid.getId());
> long parentCuboidId = parentCuboid.getId();
> long childCuboidId = childCuboid.getId();
> long parentCuboidIdActualLength = (long)Long.SIZE -
> Long.numberOfLeadingZeros(parentCuboid.getId());
> int index = rowKeySplitter.getBodySplitOffset(); // skip shard and
> cuboidId
> int offset = RowConstants.ROWKEY_SHARDID_LEN +
> RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId
> for (int i = 0; i < parentCuboidIdActualLength; i++) {
> if ((mask & parentCuboidId) > 0) {// if the this bit position
> equals
> // 1
> if ((mask & childCuboidId) > 0) {// if the child cuboid has
> this
> // column
> System.arraycopy(splitBuffers[index].array(),
> splitBuffers[index].offset(), newKeyBodyBuf.array(), offset,
> splitBuffers[index].length());
> offset += splitBuffers[index].length();
> }
> index++;
> }
> mask = mask >> 1;
> }
> rowkeyEncoder.fillHeader(newKeyBodyBuf.array());
> }
> {code}
> Found that "offset = SHARDID_LEN + CUBOIDID_LEN" , which is wrong when cube
> is not sharding. In my case my cube's storage type is 0, which means it is
> not sharding.
> So, I set offset according to cube sharding, like below:
> {code:java}
> int offset = rowKeySplitter.getHeaderLength(); // skip shard and cuboidId
> {code}
> After modifying building succeeds in my environment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3934) sqoop import param '--null-string' result in null value become blank string in hive table
[
https://issues.apache.org/jira/browse/KYLIN-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen resolved KYLIN-3934.
-
Resolution: Fixed
> sqoop import param '--null-string' result in null value become blank string
> in hive table
> -
>
> Key: KYLIN-3934
> URL: https://issues.apache.org/jira/browse/KYLIN-3934
> Project: Kylin
> Issue Type: Bug
> Components: Others
>Affects Versions: v2.6.0
>Reporter: wanghao
>Priority: Major
> Fix For: v2.6.2
>
>
> when column value from jdbc is null, sqoop will write it into hive table with
> blank string.
> eg
> jdbc:
> A | B
> 1 | 1
> 2 | 2
> a | null
>
> hive table:
> A | B
> 1 | 1
> 2 | 2
> a |
> because of this, when I use count(distinct B), it return 3, not 2, and it can
> lead to other problems
>
>
> {code:java}
> String cmd = String.format(Locale.ROOT,
> "%s/bin/sqoop import" + generateSqoopConfigArgString()
> + "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND
> \\$CONDITIONS\" "
> + "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
> + "--fields-terminated-by '%s' --num-mappers %d",
> sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql,
> jobWorkingDir, hiveTable,
> splitColumn, bquery, filedDelimiter, mapperNum);
> {code}
> the param '–null=string' should be '
> \\N' instead of blank string ''.
> I resolved this problem by replace the param. But it needs be configured in
> kylin.properties
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3936) MR/Spark task will still run after the job is stopped.
[ https://issues.apache.org/jira/browse/KYLIN-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-3936. - Resolution: Fixed > MR/Spark task will still run after the job is stopped. > -- > > Key: KYLIN-3936 > URL: https://issues.apache.org/jira/browse/KYLIN-3936 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: v2.6.2 > > > The command "pause" only sets status of the job to "stopped" and does not > reset the status of the subtask. > So, In SparkExecutable, we can't get the real status of the running task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3943) system-cube.sh has some problems
[ https://issues.apache.org/jira/browse/KYLIN-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-3943. - Resolution: Fixed > system-cube.sh has some problems > > > Key: KYLIN-3943 > URL: https://issues.apache.org/jira/browse/KYLIN-3943 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > Attachments: image-2019-04-09-18-50-35-111.png, > image-2019-04-09-23-07-19-229.png > > > 1, The name of system cube should not hard code. the suffix should get from > config. > !image-2019-04-09-18-50-35-111.png! > 2, server port should not hard code > !image-2019-04-09-23-07-19-229.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
