[jira] [Created] (KYLIN-3828) ArrayIndexOutOfBoundsException thrown when build a streaming cube with empty data in its first dimension
nichunen created KYLIN-3828:
---
Summary: ArrayIndexOutOfBoundsException thrown when build a
streaming cube with empty data in its first dimension
Key: KYLIN-3828
URL: https://issues.apache.org/jira/browse/KYLIN-3828
Project: Kylin
Issue Type: Bug
Components: NRT Streaming
Affects Versions: v2.6.0
Reporter: nichunen
Assignee: nichunen
Fix For: Future
The root cause is
in org.apache.kylin.source.kafka.hadoop.KafkaFlatTableMapper#doMap
{code:java}
public void doMap(LongWritable key, BytesWritable value, Context context)
throws IOException, InterruptedException {
ByteBuffer buffer = ByteBuffer.wrap(value.getBytes(), 0,
value.getLength());
StreamingMessageRow row = streamingParser.parse(buffer).get(0);
if (row == null) {
throw new IllegalArgumentException("");
}
data = StringUtil.join(row.getData(), delimiter);
// output this row to value
outValue.set(Bytes.toBytes(data));
context.write(outKey, outValue);
}
{code}
Method _StringUtil.join_ is used to join a collection of string to a single
string with delimiter. But _org.apache.kylin.common.util.StringUtil#join_ will
ignore the leading empty strings in the colletion, source code is:
{code:java}
public static String join(Iterable parts, String separator) {
StringBuilder buf = new StringBuilder();
for (String p : parts) {
if (buf.length() > 0)
buf.append(separator);
buf.append(p);
}
return buf.toString();
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nichunen opened a new pull request #482: KYLIN-3828 Fix heading empty string be ignored in StringUtil.join
nichunen opened a new pull request #482: KYLIN-3828 Fix heading empty string be ignored in StringUtil.join URL: https://github.com/apache/kylin/pull/482 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #482: KYLIN-3828 Fix heading empty string be ignored in StringUtil.join
asfgit commented on issue #482: KYLIN-3828 Fix heading empty string be ignored in StringUtil.join URL: https://github.com/apache/kylin/pull/482#issuecomment-466910470 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3828) ArrayIndexOutOfBoundsException thrown when build a streaming cube with empty data in its first dimension
[
https://issues.apache.org/jira/browse/KYLIN-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776620#comment-16776620
]
ASF GitHub Bot commented on KYLIN-3828:
---
nichunen commented on pull request #482: KYLIN-3828 Fix heading empty string be
ignored in StringUtil.join
URL: https://github.com/apache/kylin/pull/482
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> ArrayIndexOutOfBoundsException thrown when build a streaming cube with empty
> data in its first dimension
>
>
> Key: KYLIN-3828
> URL: https://issues.apache.org/jira/browse/KYLIN-3828
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.0
>Reporter: nichunen
>Assignee: nichunen
>Priority: Major
> Fix For: Future
>
>
> The root cause is
> in org.apache.kylin.source.kafka.hadoop.KafkaFlatTableMapper#doMap
> {code:java}
> public void doMap(LongWritable key, BytesWritable value, Context context)
> throws IOException, InterruptedException {
> ByteBuffer buffer = ByteBuffer.wrap(value.getBytes(), 0,
> value.getLength());
> StreamingMessageRow row = streamingParser.parse(buffer).get(0);
> if (row == null) {
> throw new IllegalArgumentException("");
> }
> data = StringUtil.join(row.getData(), delimiter);
> // output this row to value
> outValue.set(Bytes.toBytes(data));
> context.write(outKey, outValue);
> }
> {code}
> Method _StringUtil.join_ is used to join a collection of string to a single
> string with delimiter. But _org.apache.kylin.common.util.StringUtil#join_
> will ignore the leading empty strings in the colletion, source code is:
> {code:java}
> public static String join(Iterable parts, String separator) {
> StringBuilder buf = new StringBuilder();
> for (String p : parts) {
> if (buf.length() > 0)
> buf.append(separator);
> buf.append(p);
> }
> return buf.toString();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3826) MergeCuboidJob only uploads necessary segment's dictionary
[ https://issues.apache.org/jira/browse/KYLIN-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3826: --- Assignee: Shaofeng SHI > MergeCuboidJob only uploads necessary segment's dictionary > -- > > Key: KYLIN-3826 > URL: https://issues.apache.org/jira/browse/KYLIN-3826 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.6.1 > > > On yesterday's Kylin meetup, Zhang Wei mentioned that the "MergeCuboidJob" > will upload all segment's metadata, which will take extra long time when the > segment number is large. While this is unnecessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3826) MergeCuboidJob only uploads necessary segment's dictionary
[ https://issues.apache.org/jira/browse/KYLIN-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3826. - Resolution: Fixed > MergeCuboidJob only uploads necessary segment's dictionary > -- > > Key: KYLIN-3826 > URL: https://issues.apache.org/jira/browse/KYLIN-3826 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.6.1 > > > On yesterday's Kylin meetup, Zhang Wei mentioned that the "MergeCuboidJob" > will upload all segment's metadata, which will take extra long time when the > segment number is large. While this is unnecessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io commented on issue #482: KYLIN-3828 Fix heading empty string be ignored in StringUtil.join
codecov-io commented on issue #482: KYLIN-3828 Fix heading empty string be ignored in StringUtil.join URL: https://github.com/apache/kylin/pull/482#issuecomment-466919471 # [Codecov](https://codecov.io/gh/apache/kylin/pull/482?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@500f102`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `54.54%`. [](https://codecov.io/gh/apache/kylin/pull/482?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master #482 +/- ## = Coverage ? 24.48% Complexity? 4964 = Files ? 1144 Lines ?69428 Branches ? 9898 = Hits ?16996 Misses?50714 Partials ? 1718 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/482?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../java/org/apache/kylin/common/util/StringUtil.java](https://codecov.io/gh/apache/kylin/pull/482/diff?src=pr&el=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi91dGlsL1N0cmluZ1V0aWwuamF2YQ==) | `15.05% <54.54%> (ø)` | `5 <0> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/482?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/482?src=pr&el=footer). Last update [500f102...5bd3e9c](https://codecov.io/gh/apache/kylin/pull/482?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] coveralls commented on issue #482: KYLIN-3828 Fix heading empty string be ignored in StringUtil.join
coveralls commented on issue #482: KYLIN-3828 Fix heading empty string be ignored in StringUtil.join URL: https://github.com/apache/kylin/pull/482#issuecomment-466920942 ## Pull Request Test Coverage Report for [Build 4162](https://coveralls.io/builds/21825581) * **10** of **11** **(90.91%)** changed or added relevant lines in **1** file are covered. * **3** unchanged lines in **1** file lost coverage. * Overall coverage increased (+**0.02%**) to **26.96%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-common/src/main/java/org/apache/kylin/common/util/StringUtil.java](https://coveralls.io/builds/21825581/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Futil%2FStringUtil.java#L81) | 10 | 11 | 90.91% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/21825581/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449) | 3 | 78.42% | | Totals | [](https://coveralls.io/builds/21825581) | | :-- | --: | | Change from base [Build 4161](https://coveralls.io/builds/21821945): | 0.02% | | Covered Lines: | 18718 | | Relevant Lines: | 69428 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3816) Current CI doesn't cover the case of streaming table join lookup table
[ https://issues.apache.org/jira/browse/KYLIN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776645#comment-16776645 ] ASF GitHub Bot commented on KYLIN-3816: --- Wayne1c commented on pull request #483: KYLIN-3816 Make CI cover streaming table join case URL: https://github.com/apache/kylin/pull/483 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Current CI doesn't cover the case of streaming table join lookup table > --- > > Key: KYLIN-3816 > URL: https://issues.apache.org/jira/browse/KYLIN-3816 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Wayne1c opened a new pull request #483: KYLIN-3816 Make CI cover streaming table join case
Wayne1c opened a new pull request #483: KYLIN-3816 Make CI cover streaming table join case URL: https://github.com/apache/kylin/pull/483 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #483: KYLIN-3816 Make CI cover streaming table join case
asfgit commented on issue #483: KYLIN-3816 Make CI cover streaming table join case URL: https://github.com/apache/kylin/pull/483#issuecomment-466924299 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] coveralls commented on issue #483: KYLIN-3816 Make CI cover streaming table join case
coveralls commented on issue #483: KYLIN-3816 Make CI cover streaming table join case URL: https://github.com/apache/kylin/pull/483#issuecomment-466934743 ## Pull Request Test Coverage Report for [Build 4163](https://coveralls.io/builds/21826094) * **0** of **1** **(0.0%)** changed or added relevant line in **1** file are covered. * **3** unchanged lines in **1** file lost coverage. * Overall coverage increased (+**0.006%**) to **26.945%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [source-kafka/src/main/java/org/apache/kylin/source/kafka/KafkaSource.java](https://coveralls.io/builds/21826094/source?filename=source-kafka%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fkafka%2FKafkaSource.java#L255) | 0 | 1 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/21826094/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449) | 3 | 78.42% | | Totals | [](https://coveralls.io/builds/21826094) | | :-- | --: | | Change from base [Build 4161](https://coveralls.io/builds/21821945): | 0.006% | | Covered Lines: | 18705 | | Relevant Lines: | 69420 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #483: KYLIN-3816 Make CI cover streaming table join case
codecov-io commented on issue #483: KYLIN-3816 Make CI cover streaming table join case URL: https://github.com/apache/kylin/pull/483#issuecomment-466935858 # [Codecov](https://codecov.io/gh/apache/kylin/pull/483?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@500f102`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `0%`. [](https://codecov.io/gh/apache/kylin/pull/483?src=pr&el=tree) ```diff @@Coverage Diff@@ ## master #483 +/- ## = Coverage ? 24.46% Complexity? 4960 = Files ? 1144 Lines ?69420 Branches ? 9895 = Hits ?16986 Misses?50719 Partials ? 1715 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/483?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ava/org/apache/kylin/source/kafka/KafkaSource.java](https://codecov.io/gh/apache/kylin/pull/483/diff?src=pr&el=tree#diff-c291cmNlLWthZmthL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9zb3VyY2Uva2Fma2EvS2Fma2FTb3VyY2UuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/483?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/483?src=pr&el=footer). Last update [500f102...27875dc](https://codecov.io/gh/apache/kylin/pull/483?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3322) TopN requires a SUM to work
[ https://issues.apache.org/jira/browse/KYLIN-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776848#comment-16776848 ] KANG-SEN LU commented on KYLIN-3322: Hi, Shaofeng: Thanks for your response. I have two points to add. # What if I already put SUM(X) in a separated cube, why do I have to add SUM(X) into second cube while I am defining TOPN(X) in the second cube. If it is just redundant metadata data, I will not complain about the extra human effort. I am worried if the kylin may not be able to find the right cube to compute SUM(X), because now there are two cubes both are, supposed, equally qualified to answer the query. It will create more challenge to the cost evaluation function to kylin. # My experiment seems to suggest that when SUM(X) not group by B was issued, the cost evaluation function sent the query to the cube containing both TOPN(SUM(X)) and SUM(X) and, more importantly, it goes after TOPN(SUM(X)), then perform SUM(X), that takes more than 20 seconds in my test case. If it goes after SUM(X) directly, it took less than 0.2 second. I think how kylin try to accomplish SUM(X) in a cube containing both TOPN(SUM(X)) and SUM(X) may not be correct. That is the main reason I am against this decision that in a cube containing TOPN(SUM(X)), one must also configure SUM(X) # > TopN requires a SUM to work > --- > > Key: KYLIN-3322 > URL: https://issues.apache.org/jira/browse/KYLIN-3322 > Project: Kylin > Issue Type: Bug > Components: Measure - TopN >Reporter: liyang >Assignee: Na Zhai >Priority: Major > > Currently if user creates a measure of TopN seller by sum of price, it is > required that user also creates a measure of SUM(price). Otherwise, NPE will > be thrown at query time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit commented on issue #484: KYLIN-3820 Add a curator-based scheduler
asfgit commented on issue #484: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/484#issuecomment-467015219 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Wayne1c opened a new pull request #484: KYLIN-3820 Add a curator-based scheduler
Wayne1c opened a new pull request #484: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/484 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3820) Add a curator-based scheduler
[ https://issues.apache.org/jira/browse/KYLIN-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776894#comment-16776894 ] ASF GitHub Bot commented on KYLIN-3820: --- Wayne1c commented on pull request #484: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/484 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add a curator-based scheduler > - > > Key: KYLIN-3820 > URL: https://issues.apache.org/jira/browse/KYLIN-3820 > Project: Kylin > Issue Type: Improvement >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3827) kylin 某一字段必须使用trim才可以查出来
[
https://issues.apache.org/jira/browse/KYLIN-3827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776910#comment-16776910
]
Shaofeng SHI commented on KYLIN-3827:
-
Seems be related with KYLIN-3705. But I need the detailed kylin.log when
executing the queries. Please provide the log, or upgrade to v2.5.2 and then
rebuild the segments.
> kylin 某一字段必须使用trim才可以查出来
>
>
> Key: KYLIN-3827
> URL: https://issues.apache.org/jira/browse/KYLIN-3827
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.5.1
>Reporter: litewood
>Priority: Major
>
> SELECT pt ,pageid from st.ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1 where pt =
> '20190114' and pageid='用户标签文档' GROUP BY pt,pageid;
> 当前sql 使用hive查询可以查出来结果。
> 但是使用kylin接口查询，除去一个近视去重选用全局字段的cube能查询出来，其他cube 必须trim(pageid) 或者使用like
> 关键字才能查出来。下面帖出三个cube信息。
>
> {color:#FF}可以查出来的cube info （近视去重）{color}
> {
> "uuid": "ad8cbc65-dd6c-8c12-7d18-2f625cedf7ea",
> "last_modified": 1550719049143,
> "version": "2.5.1.20500",
> "name": "c_pageview",
> "is_draft": false,
> "model_name": "m_bury_point_analysis",
> "description": "c_pageview",
> "null_string": null,
> "dimensions": [
> \{
> "name": "APPTYPE",
> "table": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1",
> "column": "APPTYPE",
> "derived": null
> },
> \{
> "name": "PLATFORM",
> "table": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1",
> "column": "PLATFORM",
> "derived": null
> },
> \{
> "name": "UTM_SOURCE",
> "table": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1",
> "column": "UTM_SOURCE",
> "derived": null
> },
> \{
> "name": "ADCODE",
> "table": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1",
> "column": "ADCODE",
> "derived": null
> },
> \{
> "name": "ADDITIONTYPE",
> "table": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1",
> "column": "ADDITIONTYPE",
> "derived": null
> },
> \{
> "name": "ADDITIONVALUE",
> "table": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1",
> "column": "ADDITIONVALUE",
> "derived": null
> },
> \{
> "name": "APPVERSION",
> "table": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1",
> "column": "APPVERSION",
> "derived": null
> },
> \{
> "name": "CATEGORYID",
> "table": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1",
> "column": "CATEGORYID",
> "derived": null
> },
> \{
> "name": "CITYCODE",
> "table": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1",
> "column": "CITYCODE",
> "derived": null
> },
> \{
> "name": "FLAGVALUE",
> "table": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1",
> "column": "FLAGVALUE",
> "derived": null
> },
> \{
> "name": "PAGEID",
> "table": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1",
> "column": "PAGEID",
> "derived": null
> },
> \{
> "name": "H_DATE",
> "table": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1",
> "column": "H_DATE",
> "derived": null
> },
> \{
> "name": "PT",
> "table": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1",
> "column": "PT",
> "derived": null
> },
> \{
> "name": "WEEKYTH_CNT_CN",
> "table": "DIM_PUB_DATE",
> "column": null,
> "derived": [
> "WEEKYTH_CNT_CN"
> ]
> },
> \{
> "name": "WEEKYTH_START_DATA_CN",
> "table": "DIM_PUB_DATE",
> "column": null,
> "derived": [
> "WEEKYTH_START_DATA_CN"
> ]
> },
> \{
> "name": "WEEKYTH_END_DATA_CN",
> "table": "DIM_PUB_DATE",
> "column": null,
> "derived": [
> "WEEKYTH_END_DATA_CN"
> ]
> }
> ],
> "measures": [
> \{
> "name": "PV",
> "function": {
> "expression": "COUNT",
> "parameter": {
> "type": "constant",
> "value": "1"
> },
> "returntype": "bigint"
> }
> },
> \{
> "name": "UV",
> "function": {
> "expression": "COUNT_DISTINCT",
> "parameter": {
> "type": "column",
> "value": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1.USERGUID"
> },
> "returntype": "hllc(16)"
> }
> },
> \{
> "name": "DUV",
> "function": {
> "expression": "COUNT_DISTINCT",
> "parameter": {
> "type": "column",
> "value": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1.SSID"
> },
> "returntype": "hllc(16)"
> }
> }
> ],
> "dictionaries": [],
> "rowkey": \{
> "rowkey_columns": [
> {
> "column": "ST_T_BIKE_YUKON_PRO_PAGEVIEW_CLEAN1.PT",
> "encoding": "dict",
> "encoding_version": 1,
> "isShardBy": false
> },
> \{
> "column": "S
[jira] [Commented] (KYLIN-3828) ArrayIndexOutOfBoundsException thrown when build a streaming cube with empty data in its first dimension
[
https://issues.apache.org/jira/browse/KYLIN-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776916#comment-16776916
]
ASF subversion and git services commented on KYLIN-3828:
Commit fb34122243372cb229b43eee29ca59f933d0ada3 in kylin's branch
refs/heads/master from nichunen
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=fb34122 ]
KYLIN-3828 Fix heading empty string be ignored in StringUtil.join
> ArrayIndexOutOfBoundsException thrown when build a streaming cube with empty
> data in its first dimension
>
>
> Key: KYLIN-3828
> URL: https://issues.apache.org/jira/browse/KYLIN-3828
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.0
>Reporter: nichunen
>Assignee: nichunen
>Priority: Major
> Fix For: Future
>
>
> The root cause is
> in org.apache.kylin.source.kafka.hadoop.KafkaFlatTableMapper#doMap
> {code:java}
> public void doMap(LongWritable key, BytesWritable value, Context context)
> throws IOException, InterruptedException {
> ByteBuffer buffer = ByteBuffer.wrap(value.getBytes(), 0,
> value.getLength());
> StreamingMessageRow row = streamingParser.parse(buffer).get(0);
> if (row == null) {
> throw new IllegalArgumentException("");
> }
> data = StringUtil.join(row.getData(), delimiter);
> // output this row to value
> outValue.set(Bytes.toBytes(data));
> context.write(outKey, outValue);
> }
> {code}
> Method _StringUtil.join_ is used to join a collection of string to a single
> string with delimiter. But _org.apache.kylin.common.util.StringUtil#join_
> will ignore the leading empty strings in the colletion, source code is:
> {code:java}
> public static String join(Iterable parts, String separator) {
> StringBuilder buf = new StringBuilder();
> for (String p : parts) {
> if (buf.length() > 0)
> buf.append(separator);
> buf.append(p);
> }
> return buf.toString();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] shaofengshi merged pull request #482: KYLIN-3828 Fix heading empty string be ignored in StringUtil.join
shaofengshi merged pull request #482: KYLIN-3828 Fix heading empty string be ignored in StringUtil.join URL: https://github.com/apache/kylin/pull/482 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3828) ArrayIndexOutOfBoundsException thrown when build a streaming cube with empty data in its first dimension
[
https://issues.apache.org/jira/browse/KYLIN-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776915#comment-16776915
]
ASF GitHub Bot commented on KYLIN-3828:
---
shaofengshi commented on pull request #482: KYLIN-3828 Fix heading empty string
be ignored in StringUtil.join
URL: https://github.com/apache/kylin/pull/482
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> ArrayIndexOutOfBoundsException thrown when build a streaming cube with empty
> data in its first dimension
>
>
> Key: KYLIN-3828
> URL: https://issues.apache.org/jira/browse/KYLIN-3828
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.0
>Reporter: nichunen
>Assignee: nichunen
>Priority: Major
> Fix For: Future
>
>
> The root cause is
> in org.apache.kylin.source.kafka.hadoop.KafkaFlatTableMapper#doMap
> {code:java}
> public void doMap(LongWritable key, BytesWritable value, Context context)
> throws IOException, InterruptedException {
> ByteBuffer buffer = ByteBuffer.wrap(value.getBytes(), 0,
> value.getLength());
> StreamingMessageRow row = streamingParser.parse(buffer).get(0);
> if (row == null) {
> throw new IllegalArgumentException("");
> }
> data = StringUtil.join(row.getData(), delimiter);
> // output this row to value
> outValue.set(Bytes.toBytes(data));
> context.write(outKey, outValue);
> }
> {code}
> Method _StringUtil.join_ is used to join a collection of string to a single
> string with delimiter. But _org.apache.kylin.common.util.StringUtil#join_
> will ignore the leading empty strings in the colletion, source code is:
> {code:java}
> public static String join(Iterable parts, String separator) {
> StringBuilder buf = new StringBuilder();
> for (String p : parts) {
> if (buf.length() > 0)
> buf.append(separator);
> buf.append(p);
> }
> return buf.toString();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3828) ArrayIndexOutOfBoundsException thrown when build a streaming cube with empty data in its first dimension
[
https://issues.apache.org/jira/browse/KYLIN-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3828:
Fix Version/s: (was: Future)
v2.6.1
> ArrayIndexOutOfBoundsException thrown when build a streaming cube with empty
> data in its first dimension
>
>
> Key: KYLIN-3828
> URL: https://issues.apache.org/jira/browse/KYLIN-3828
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.0
>Reporter: nichunen
>Assignee: nichunen
>Priority: Major
> Fix For: v2.6.1
>
>
> The root cause is
> in org.apache.kylin.source.kafka.hadoop.KafkaFlatTableMapper#doMap
> {code:java}
> public void doMap(LongWritable key, BytesWritable value, Context context)
> throws IOException, InterruptedException {
> ByteBuffer buffer = ByteBuffer.wrap(value.getBytes(), 0,
> value.getLength());
> StreamingMessageRow row = streamingParser.parse(buffer).get(0);
> if (row == null) {
> throw new IllegalArgumentException("");
> }
> data = StringUtil.join(row.getData(), delimiter);
> // output this row to value
> outValue.set(Bytes.toBytes(data));
> context.write(outKey, outValue);
> }
> {code}
> Method _StringUtil.join_ is used to join a collection of string to a single
> string with delimiter. But _org.apache.kylin.common.util.StringUtil#join_
> will ignore the leading empty strings in the colletion, source code is:
> {code:java}
> public static String join(Iterable parts, String separator) {
> StringBuilder buf = new StringBuilder();
> for (String p : parts) {
> if (buf.length() > 0)
> buf.append(separator);
> buf.append(p);
> }
> return buf.toString();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] coveralls commented on issue #484: KYLIN-3820 Add a curator-based scheduler
coveralls commented on issue #484: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/484#issuecomment-467030756 ## Pull Request Test Coverage Report for [Build 4164](https://coveralls.io/builds/21831279) * **114** of **177** **(64.41%)** changed or added relevant lines in **4** files are covered. * **3** unchanged lines in **1** file lost coverage. * Overall coverage increased (+**0.1%**) to **27.044%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/21831279/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L366) | 0 | 4 | 0.0% | [core-job/src/main/java/org/apache/kylin/job/impl/curator/CuratorLeaderSelector.java](https://coveralls.io/builds/21831279/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fcurator%2FCuratorLeaderSelector.java#L56) | 30 | 43 | 69.77% | [core-common/src/main/java/org/apache/kylin/common/ServerMode.java](https://coveralls.io/builds/21831279/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FServerMode.java#L21) | 0 | 17 | 0.0% | [core-job/src/main/java/org/apache/kylin/job/impl/curator/CuratorScheduler.java](https://coveralls.io/builds/21831279/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fcurator%2FCuratorScheduler.java#L87) | 84 | 113 | 74.34% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/21831279/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449) | 3 | 78.42% | | Totals | [](https://coveralls.io/builds/21831279) | | :-- | --: | | Change from base [Build 4161](https://coveralls.io/builds/21821945): | 0.1% | | Covered Lines: | 18822 | | Relevant Lines: | 69597 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3782) ZookeeperDistributedLock can't acquir lock on windows because wrong Path
[
https://issues.apache.org/jira/browse/KYLIN-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776943#comment-16776943
]
ASF subversion and git services commented on KYLIN-3782:
Commit 8e717518642b36bea5b35a45d7f1995c0f662c73 in kylin's branch
refs/heads/master from hailin huang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=8e71751 ]
KYLIN-3782 ZookeeperDistributedLock can't acquir lock on windows
> ZookeeperDistributedLock can't acquir lock on windows because wrong Path
>
>
> Key: KYLIN-3782
> URL: https://issues.apache.org/jira/browse/KYLIN-3782
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Affects Versions: v2.6.0
>Reporter: hailin.huang
>Assignee: hailin.huang
>Priority: Major
> Fix For: v2.6.1
>
>
> In my windows env, when I run kylin, Kylin service can't start. In class
> ZookeeperDistributedLock, I found kylin use File(path).getCanonicalPath() to
> norm windows path, however this will get a path like '
> C:\kylin\kylin_metadata1\job_engine\global_job_engine_lock', for
> zookeeper,Path must start with / character.
> {color:red}return new File(path).toURI().getPath(){color} might be better!!!
> Below is stackTrace:
> Caused by: java.lang.IllegalStateException: Error while
> 832820@UC-20180829ZLJB trying to lock
> C:\kylin\kylin_metadata1\job_engine\global_job_engine_lock
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:153)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:170)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lockJobEngine(ZookeeperDistributedLock.java:318)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperJobLock.lockJobEngine(ZookeeperJobLock.java:81)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler.init(DefaultScheduler.java:152)
> at
> org.apache.kylin.rest.service.JobService.afterPropertiesSet(JobService.java:126)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
> ... 60 more
> caused by: java.lang.IllegalArgumentException: Path must start with /
> character
> at org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:54)
> at org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:37)
> at org.apache.curator.utils.ZKPaths.fixForNamespace(ZKPaths.java:63)
> at
> org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:82)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:579)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:434)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:149)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3782) ZookeeperDistributedLock can't acquir lock on windows because wrong Path
[
https://issues.apache.org/jira/browse/KYLIN-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776949#comment-16776949
]
ASF subversion and git services commented on KYLIN-3782:
Commit 23c8e985cb4a005f563c68e4a33c8a74a20cdfed in kylin's branch
refs/heads/2.6.x from hailin huang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=23c8e98 ]
KYLIN-3782 ZookeeperDistributedLock can't acquir lock on windows
> ZookeeperDistributedLock can't acquir lock on windows because wrong Path
>
>
> Key: KYLIN-3782
> URL: https://issues.apache.org/jira/browse/KYLIN-3782
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Affects Versions: v2.6.0
>Reporter: hailin.huang
>Assignee: hailin.huang
>Priority: Major
> Fix For: v2.6.1
>
>
> In my windows env, when I run kylin, Kylin service can't start. In class
> ZookeeperDistributedLock, I found kylin use File(path).getCanonicalPath() to
> norm windows path, however this will get a path like '
> C:\kylin\kylin_metadata1\job_engine\global_job_engine_lock', for
> zookeeper,Path must start with / character.
> {color:red}return new File(path).toURI().getPath(){color} might be better!!!
> Below is stackTrace:
> Caused by: java.lang.IllegalStateException: Error while
> 832820@UC-20180829ZLJB trying to lock
> C:\kylin\kylin_metadata1\job_engine\global_job_engine_lock
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:153)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:170)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lockJobEngine(ZookeeperDistributedLock.java:318)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperJobLock.lockJobEngine(ZookeeperJobLock.java:81)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler.init(DefaultScheduler.java:152)
> at
> org.apache.kylin.rest.service.JobService.afterPropertiesSet(JobService.java:126)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
> ... 60 more
> caused by: java.lang.IllegalArgumentException: Path must start with /
> character
> at org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:54)
> at org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:37)
> at org.apache.curator.utils.ZKPaths.fixForNamespace(ZKPaths.java:63)
> at
> org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:82)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:579)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:434)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:149)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3826) MergeCuboidJob only uploads necessary segment's dictionary
[ https://issues.apache.org/jira/browse/KYLIN-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776947#comment-16776947 ] ASF subversion and git services commented on KYLIN-3826: Commit 4b3a20be3cefd540a2a300bf7f01f213d6a591b7 in kylin's branch refs/heads/2.6.x from Shao Feng Shi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=4b3a20b ] KYLIN-3826 MergeCuboidJob only uploads necessary segment's dictionary > MergeCuboidJob only uploads necessary segment's dictionary > -- > > Key: KYLIN-3826 > URL: https://issues.apache.org/jira/browse/KYLIN-3826 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.6.1 > > > On yesterday's Kylin meetup, Zhang Wei mentioned that the "MergeCuboidJob" > will upload all segment's metadata, which will take extra long time when the > segment number is large. While this is unnecessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3828) ArrayIndexOutOfBoundsException thrown when build a streaming cube with empty data in its first dimension
[
https://issues.apache.org/jira/browse/KYLIN-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776956#comment-16776956
]
ASF subversion and git services commented on KYLIN-3828:
Commit d6819b3e70033888b42ab0e5585c191898f8afe9 in kylin's branch
refs/heads/2.5.x from nichunen
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=d6819b3 ]
KYLIN-3828 Fix heading empty string be ignored in StringUtil.join
> ArrayIndexOutOfBoundsException thrown when build a streaming cube with empty
> data in its first dimension
>
>
> Key: KYLIN-3828
> URL: https://issues.apache.org/jira/browse/KYLIN-3828
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.0
>Reporter: nichunen
>Assignee: nichunen
>Priority: Major
> Fix For: v2.6.1
>
>
> The root cause is
> in org.apache.kylin.source.kafka.hadoop.KafkaFlatTableMapper#doMap
> {code:java}
> public void doMap(LongWritable key, BytesWritable value, Context context)
> throws IOException, InterruptedException {
> ByteBuffer buffer = ByteBuffer.wrap(value.getBytes(), 0,
> value.getLength());
> StreamingMessageRow row = streamingParser.parse(buffer).get(0);
> if (row == null) {
> throw new IllegalArgumentException("");
> }
> data = StringUtil.join(row.getData(), delimiter);
> // output this row to value
> outValue.set(Bytes.toBytes(data));
> context.write(outKey, outValue);
> }
> {code}
> Method _StringUtil.join_ is used to join a collection of string to a single
> string with delimiter. But _org.apache.kylin.common.util.StringUtil#join_
> will ignore the leading empty strings in the colletion, source code is:
> {code:java}
> public static String join(Iterable parts, String separator) {
> StringBuilder buf = new StringBuilder();
> for (String p : parts) {
> if (buf.length() > 0)
> buf.append(separator);
> buf.append(p);
> }
> return buf.toString();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3826) MergeCuboidJob only uploads necessary segment's dictionary
[ https://issues.apache.org/jira/browse/KYLIN-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776967#comment-16776967 ] ASF subversion and git services commented on KYLIN-3826: Commit 7a6bf9871d422289cb642bb1caaefef85c9c1604 in kylin's branch refs/heads/2.4.x from Shao Feng Shi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=7a6bf98 ] KYLIN-3826 MergeCuboidJob only uploads necessary segment's dictionary > MergeCuboidJob only uploads necessary segment's dictionary > -- > > Key: KYLIN-3826 > URL: https://issues.apache.org/jira/browse/KYLIN-3826 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.6.1 > > > On yesterday's Kylin meetup, Zhang Wei mentioned that the "MergeCuboidJob" > will upload all segment's metadata, which will take extra long time when the > segment number is large. While this is unnecessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3782) ZookeeperDistributedLock can't acquir lock on windows because wrong Path
[
https://issues.apache.org/jira/browse/KYLIN-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI resolved KYLIN-3782.
-
Resolution: Fixed
> ZookeeperDistributedLock can't acquir lock on windows because wrong Path
>
>
> Key: KYLIN-3782
> URL: https://issues.apache.org/jira/browse/KYLIN-3782
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Affects Versions: v2.6.0
>Reporter: hailin.huang
>Assignee: hailin.huang
>Priority: Major
> Fix For: v2.6.1
>
>
> In my windows env, when I run kylin, Kylin service can't start. In class
> ZookeeperDistributedLock, I found kylin use File(path).getCanonicalPath() to
> norm windows path, however this will get a path like '
> C:\kylin\kylin_metadata1\job_engine\global_job_engine_lock', for
> zookeeper,Path must start with / character.
> {color:red}return new File(path).toURI().getPath(){color} might be better!!!
> Below is stackTrace:
> Caused by: java.lang.IllegalStateException: Error while
> 832820@UC-20180829ZLJB trying to lock
> C:\kylin\kylin_metadata1\job_engine\global_job_engine_lock
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:153)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:170)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lockJobEngine(ZookeeperDistributedLock.java:318)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperJobLock.lockJobEngine(ZookeeperJobLock.java:81)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler.init(DefaultScheduler.java:152)
> at
> org.apache.kylin.rest.service.JobService.afterPropertiesSet(JobService.java:126)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
> ... 60 more
> caused by: java.lang.IllegalArgumentException: Path must start with /
> character
> at org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:54)
> at org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:37)
> at org.apache.curator.utils.ZKPaths.fixForNamespace(ZKPaths.java:63)
> at
> org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:82)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:579)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:434)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:149)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] shaofengshi commented on issue #477: bug fixed for KYLIN-3782
shaofengshi commented on issue #477: bug fixed for KYLIN-3782 URL: https://github.com/apache/kylin/pull/477#issuecomment-467040007 Merged with patch, close the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shaofengshi closed pull request #477: bug fixed for KYLIN-3782
shaofengshi closed pull request #477: bug fixed for KYLIN-3782 URL: https://github.com/apache/kylin/pull/477 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3782) ZookeeperDistributedLock can't acquir lock on windows because wrong Path
[
https://issues.apache.org/jira/browse/KYLIN-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776968#comment-16776968
]
ASF GitHub Bot commented on KYLIN-3782:
---
shaofengshi commented on pull request #477: bug fixed for KYLIN-3782
URL: https://github.com/apache/kylin/pull/477
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> ZookeeperDistributedLock can't acquir lock on windows because wrong Path
>
>
> Key: KYLIN-3782
> URL: https://issues.apache.org/jira/browse/KYLIN-3782
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Affects Versions: v2.6.0
>Reporter: hailin.huang
>Assignee: hailin.huang
>Priority: Major
> Fix For: v2.6.1
>
>
> In my windows env, when I run kylin, Kylin service can't start. In class
> ZookeeperDistributedLock, I found kylin use File(path).getCanonicalPath() to
> norm windows path, however this will get a path like '
> C:\kylin\kylin_metadata1\job_engine\global_job_engine_lock', for
> zookeeper,Path must start with / character.
> {color:red}return new File(path).toURI().getPath(){color} might be better!!!
> Below is stackTrace:
> Caused by: java.lang.IllegalStateException: Error while
> 832820@UC-20180829ZLJB trying to lock
> C:\kylin\kylin_metadata1\job_engine\global_job_engine_lock
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:153)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:170)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lockJobEngine(ZookeeperDistributedLock.java:318)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperJobLock.lockJobEngine(ZookeeperJobLock.java:81)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler.init(DefaultScheduler.java:152)
> at
> org.apache.kylin.rest.service.JobService.afterPropertiesSet(JobService.java:126)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
> ... 60 more
> caused by: java.lang.IllegalArgumentException: Path must start with /
> character
> at org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:54)
> at org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:37)
> at org.apache.curator.utils.ZKPaths.fixForNamespace(ZKPaths.java:63)
> at
> org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:82)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:579)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:434)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
> at
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:149)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3828) ArrayIndexOutOfBoundsException thrown when build a streaming cube with empty data in its first dimension
[
https://issues.apache.org/jira/browse/KYLIN-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776948#comment-16776948
]
ASF subversion and git services commented on KYLIN-3828:
Commit 9a3ec734d1ccaf81a9f6df819961d7b9f541b7df in kylin's branch
refs/heads/2.6.x from nichunen
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=9a3ec73 ]
KYLIN-3828 Fix heading empty string be ignored in StringUtil.join
> ArrayIndexOutOfBoundsException thrown when build a streaming cube with empty
> data in its first dimension
>
>
> Key: KYLIN-3828
> URL: https://issues.apache.org/jira/browse/KYLIN-3828
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
>Affects Versions: v2.6.0
>Reporter: nichunen
>Assignee: nichunen
>Priority: Major
> Fix For: v2.6.1
>
>
> The root cause is
> in org.apache.kylin.source.kafka.hadoop.KafkaFlatTableMapper#doMap
> {code:java}
> public void doMap(LongWritable key, BytesWritable value, Context context)
> throws IOException, InterruptedException {
> ByteBuffer buffer = ByteBuffer.wrap(value.getBytes(), 0,
> value.getLength());
> StreamingMessageRow row = streamingParser.parse(buffer).get(0);
> if (row == null) {
> throw new IllegalArgumentException("");
> }
> data = StringUtil.join(row.getData(), delimiter);
> // output this row to value
> outValue.set(Bytes.toBytes(data));
> context.write(outKey, outValue);
> }
> {code}
> Method _StringUtil.join_ is used to join a collection of string to a single
> string with delimiter. But _org.apache.kylin.common.util.StringUtil#join_
> will ignore the leading empty strings in the colletion, source code is:
> {code:java}
> public static String join(Iterable parts, String separator) {
> StringBuilder buf = new StringBuilder();
> for (String p : parts) {
> if (buf.length() > 0)
> buf.append(separator);
> buf.append(p);
> }
> return buf.toString();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3826) MergeCuboidJob only uploads necessary segment's dictionary
[ https://issues.apache.org/jira/browse/KYLIN-3826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776955#comment-16776955 ] ASF subversion and git services commented on KYLIN-3826: Commit 7a731c8d8832e7292cc9b79de4dbda98fd52c0ff in kylin's branch refs/heads/2.5.x from Shao Feng Shi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=7a731c8 ] KYLIN-3826 MergeCuboidJob only uploads necessary segment's dictionary > MergeCuboidJob only uploads necessary segment's dictionary > -- > > Key: KYLIN-3826 > URL: https://issues.apache.org/jira/browse/KYLIN-3826 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.6.1 > > > On yesterday's Kylin meetup, Zhang Wei mentioned that the "MergeCuboidJob" > will upload all segment's metadata, which will take extra long time when the > segment number is large. While this is unnecessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3829) Chinese document on the kylin official site about sample cube got the sql wrong
王汝鹏 created KYLIN-3829: -- Summary: Chinese document on the kylin official site about sample cube got the sql wrong Key: KYLIN-3829 URL: https://issues.apache.org/jira/browse/KYLIN-3829 Project: Kylin Issue Type: Improvement Components: Documentation, Website Affects Versions: v2.6.0 Reporter: 王汝鹏 As we can see, the sql in the [Chinese document about the quick start with sample cube|[http://kylin.apache.org/cn/docs/tutorial/kylin_sample.html]|http://kylin.apache.org/cn/docs/tutorial/kylin_sample.html],] misses a piece which should be "select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales group by part_dt order by part_dt". But the English version is correct. With the helping of the front end code checking tool, you can see the right sql, so it's just a front end display issue and I think it's easy to fix. :) I've tested by chrome, firefox and edge and got the same problem, so it's not the problem of brower . It could be confusing for the beginner though it's easy to resolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3814) Add pause interval for job retry
[ https://issues.apache.org/jira/browse/KYLIN-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 王汝鹏 updated KYLIN-3814: --- Attachment: (was: kylin.jpg) > Add pause interval for job retry > > > Key: KYLIN-3814 > URL: https://issues.apache.org/jira/browse/KYLIN-3814 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Blocker > > When there is an exception occurred while the job was running, like network > shake or ZK service shake. The job will be retried. The configuration > "kylin.job.retry" will handle this. > But sometimes we need to pause a little time before the retry job start. The > purpose is to wait for the network shake or ZK shake recovered, otherwise the > retry make no sense. > The design is to add a configuration "kylin.job.retry-interval", the default > is 3 milliseconds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3814) Add pause interval for job retry
[ https://issues.apache.org/jira/browse/KYLIN-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 王汝鹏 updated KYLIN-3814: --- Attachment: kylin.jpg > Add pause interval for job retry > > > Key: KYLIN-3814 > URL: https://issues.apache.org/jira/browse/KYLIN-3814 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Blocker > Attachments: kylin.jpg > > > When there is an exception occurred while the job was running, like network > shake or ZK service shake. The job will be retried. The configuration > "kylin.job.retry" will handle this. > But sometimes we need to pause a little time before the retry job start. The > purpose is to wait for the network shake or ZK shake recovered, otherwise the > retry make no sense. > The design is to add a configuration "kylin.job.retry-interval", the default > is 3 milliseconds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3814) Add pause interval for job retry
[ https://issues.apache.org/jira/browse/KYLIN-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3814: Priority: Major (was: Blocker) Fix Version/s: v3.0.0 > Add pause interval for job retry > > > Key: KYLIN-3814 > URL: https://issues.apache.org/jira/browse/KYLIN-3814 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Major > Fix For: v3.0.0 > > > When there is an exception occurred while the job was running, like network > shake or ZK service shake. The job will be retried. The configuration > "kylin.job.retry" will handle this. > But sometimes we need to pause a little time before the retry job start. The > purpose is to wait for the network shake or ZK shake recovered, otherwise the > retry make no sense. > The design is to add a configuration "kylin.job.retry-interval", the default > is 3 milliseconds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3814) Add pause interval for job retry
[ https://issues.apache.org/jira/browse/KYLIN-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777467#comment-16777467 ] 王汝鹏 commented on KYLIN-3814: I'm so sorry for my wrong operation about deleting the link. Here is the link [This issue links to "GitHub Pull Request #470 (Web Link)" [ 146226 ]|[https://github.com/apache/kylin/pull/470]] > Add pause interval for job retry > > > Key: KYLIN-3814 > URL: https://issues.apache.org/jira/browse/KYLIN-3814 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Major > Fix For: v3.0.0 > > > When there is an exception occurred while the job was running, like network > shake or ZK service shake. The job will be retried. The configuration > "kylin.job.retry" will handle this. > But sometimes we need to pause a little time before the retry job start. The > purpose is to wait for the network shake or ZK shake recovered, otherwise the > retry make no sense. > The design is to add a configuration "kylin.job.retry-interval", the default > is 3 milliseconds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3829) Chinese document on the kylin official site about sample cube got the sql wrong
[ https://issues.apache.org/jira/browse/KYLIN-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 王汝鹏 updated KYLIN-3829: --- Attachment: kylin.jpg > Chinese document on the kylin official site about sample cube got the sql > wrong > --- > > Key: KYLIN-3829 > URL: https://issues.apache.org/jira/browse/KYLIN-3829 > Project: Kylin > Issue Type: Improvement > Components: Documentation, Website >Affects Versions: v2.6.0 >Reporter: 王汝鹏 >Priority: Trivial > Labels: beginner, easyfix > Attachments: kylin.jpg > > > As we can see, the sql in the [Chinese document about the quick start with > sample > cube|[http://kylin.apache.org/cn/docs/tutorial/kylin_sample.html]|http://kylin.apache.org/cn/docs/tutorial/kylin_sample.html],] > misses a piece which should be "select part_dt, sum(price) as total_selled, > count(distinct seller_id) as sellers from kylin_sales group by part_dt order > by part_dt". But the English version is correct. With the helping of the > front end code checking tool, you can see the right sql, so it's just a front > end display issue and I think it's easy to fix. :) > I've tested by chrome, firefox and edge and got the same problem, so it's not > the problem of brower . > It could be confusing for the beginner though it's easy to resolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3814) Add pause interval for job retry
[ https://issues.apache.org/jira/browse/KYLIN-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3814. - Resolution: Fixed > Add pause interval for job retry > > > Key: KYLIN-3814 > URL: https://issues.apache.org/jira/browse/KYLIN-3814 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Major > Fix For: v3.0.0 > > > When there is an exception occurred while the job was running, like network > shake or ZK service shake. The job will be retried. The configuration > "kylin.job.retry" will handle this. > But sometimes we need to pause a little time before the retry job start. The > purpose is to wait for the network shake or ZK shake recovered, otherwise the > retry make no sense. > The design is to add a configuration "kylin.job.retry-interval", the default > is 3 milliseconds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3829) Chinese document on the kylin official site about sample cube got the sql wrong
[ https://issues.apache.org/jira/browse/KYLIN-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 王汝鹏 updated KYLIN-3829: --- Description: As we can see, the sql in the [Chinese document about the quick start with sample cube|[http://kylin.apache.org/cn/docs/tutorial/kylin_sample.html]] misses a piece which should be "select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales group by part_dt order by part_dt". But the English version is correct. With the helping of the front end code checking tool, you can see the right sql, so it's just a front end display issue and I think it's easy to fix. :) I've tested by chrome, firefox and edge and got the same problem, so it's not the problem of brower . It could be confusing for the beginner though it's easy to resolve. was: As we can see, the sql in the [Chinese document about the quick start with sample cube|[http://kylin.apache.org/cn/docs/tutorial/kylin_sample.html]|http://kylin.apache.org/cn/docs/tutorial/kylin_sample.html],] misses a piece which should be "select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales group by part_dt order by part_dt". But the English version is correct. With the helping of the front end code checking tool, you can see the right sql, so it's just a front end display issue and I think it's easy to fix. :) I've tested by chrome, firefox and edge and got the same problem, so it's not the problem of brower . It could be confusing for the beginner though it's easy to resolve. > Chinese document on the kylin official site about sample cube got the sql > wrong > --- > > Key: KYLIN-3829 > URL: https://issues.apache.org/jira/browse/KYLIN-3829 > Project: Kylin > Issue Type: Improvement > Components: Documentation, Website >Affects Versions: v2.6.0 >Reporter: 王汝鹏 >Priority: Trivial > Labels: beginner, easyfix > Attachments: kylin.jpg > > > As we can see, the sql in the [Chinese document about the quick start with > sample cube|[http://kylin.apache.org/cn/docs/tutorial/kylin_sample.html]] > misses a piece which should be "select part_dt, sum(price) as total_selled, > count(distinct seller_id) as sellers from kylin_sales group by part_dt order > by part_dt". But the English version is correct. With the helping of the > front end code checking tool, you can see the right sql, so it's just a front > end display issue and I think it's easy to fix. :) > I've tested by chrome, firefox and edge and got the same problem, so it's not > the problem of brower . > It could be confusing for the beginner though it's easy to resolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3810) Kylin service nodes discovery with Zookeeper
[ https://issues.apache.org/jira/browse/KYLIN-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777515#comment-16777515 ] Chao Long commented on KYLIN-3810: -- Recently, I added a curator-based scheduler in kylin and I found some work duplicate with yours. In my work, I use the curator-x-discovery component to implement kylin service auto discovery and failover and implement the automatic election and monitoring of the job node leader through zk. So, I want to discuss with you guys which realization is better.:D KYLIN-3820 > Kylin service nodes discovery with Zookeeper > - > > Key: KYLIN-3810 > URL: https://issues.apache.org/jira/browse/KYLIN-3810 > Project: Kylin > Issue Type: Improvement > Components: Metadata, Others >Affects Versions: v2.3.1 >Reporter: zhangwei >Assignee: zhangwei >Priority: Minor > Attachments: zookeeperCacheSync.patch, 同步阻塞.png, 引入zookeeper.png > > > 由于Kylin的元数据同步使用了Boardcast类中的线程池，所以一旦负载均衡查询节点中一台机器岩机，Kylin的同步请求线程会堵塞在岩机的查询节点，最后将线程资源全部占满，如图同步到1,2节点的请求堵塞在线程队列中导致查询节点获取不到最新元数据从而无法查询到实时构建好的数据 > 我们的方案: > 引进了服务发现组建zookeeper -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3809) Support Zookeeper based rest server discovery
[ https://issues.apache.org/jira/browse/KYLIN-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777514#comment-16777514 ] Chao Long commented on KYLIN-3809: -- Recently, I added a curator-based scheduler in kylin and I found some work duplicate with yours. In my work, I use the curator-x-discovery component to implement kylin service auto discovery and failover and implement the automatic election and monitoring of the job node leader through zk. So, I want to discuss with you guys which realization is better.:D [KYLIN-3820|https://issues.apache.org/jira/browse/KYLIN-3820] > Support Zookeeper based rest server discovery > - > > Key: KYLIN-3809 > URL: https://issues.apache.org/jira/browse/KYLIN-3809 > Project: Kylin > Issue Type: New Feature >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > > Currently to broadcast config or meta changes, all kylin servers must be set > in kylin.properties. It's not convenient when adding or removing kylin server > especially in k8s env. > > So we can register the endpoint to zk and make the rest server discovery > automatically. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3830) return wrong result when 'SELECT SUM(dim1)' without set a relative metric of dim1.
Yuzhang QIU created KYLIN-3830: -- Summary: return wrong result when 'SELECT SUM(dim1)' without set a relative metric of dim1. Key: KYLIN-3830 URL: https://issues.apache.org/jira/browse/KYLIN-3830 Project: Kylin Issue Type: Bug Affects Versions: v2.5.2 Reporter: Yuzhang QIU Hi, dear team: I design an cube1 based on table table1 with dim1, dim2, dim3 and only one metric count(1), and 'SELECT SUM(dim1) FROM table1 group by dim2', Kylin process this SQL and return some result1. It seems ok. But as we know, Kylin don't store the detail data, the dimensions' members have been encoded and stored in Hbase as rowkey(cause I don't set any metric with an column). So, is the result1 right? Then, I clone cube1 to cube2, and set a metric SUM(dim1). the same SQL has been passed to kylin and got result2. It's different from result1 at the aggregation field. I also pass same SQL to hive and got result3, it's same with result2. Yes, I turn off the pushdown. I think there are some problems. I can't upload some picture of results for secret policy, sorry for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3811) Support the standalone HBase cluster using HDFS HA
[ https://issues.apache.org/jira/browse/KYLIN-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777689#comment-16777689 ] zhangwei commented on KYLIN-3811: - because LoadIncrementalHfile is hbase class and we don't want to change class of hbase so we hack this class in kylin side this is main change left is kylinloadincrementalHFile right is LoadIncrementalHfile !screenshot-1.png! > Support the standalone HBase cluster using HDFS HA > -- > > Key: KYLIN-3811 > URL: https://issues.apache.org/jira/browse/KYLIN-3811 > Project: Kylin > Issue Type: Improvement > Components: Storage - HBase >Affects Versions: v2.3.1 >Reporter: zhangwei >Assignee: zhangwei >Priority: Minor > Attachments: hdfs_ha.patch, hdfs_ha_2.patch, screenshot-1.png > > > 独立hbase 集群的 hdfs namenode主备切换导致Kylin无法正常工作的问题 > 痛点: > ylin.hbase.cluster.fs=hdfs://namenode:54310，在hbase的hdfs namenode切换之后会，和hbase > hdfs相关的步骤会出现失败 > 我们的方案:(所有的修改都基于kylin的代码，不需要hbase 做修改) > kylin.properties > kylin.hbase.cluster.fs=hdfs://nshbase > kylin.storage.hbase.cluster-hdfs-config-file=hbase-hdfs-site.xml > hbase-hdfs-site.xml配置 hbase hdfs的HA连接方式，其中dfs.nameservices= nshbase -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3811) Support the standalone HBase cluster using HDFS HA
[ https://issues.apache.org/jira/browse/KYLIN-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangwei updated KYLIN-3811: Attachment: screenshot-1.png > Support the standalone HBase cluster using HDFS HA > -- > > Key: KYLIN-3811 > URL: https://issues.apache.org/jira/browse/KYLIN-3811 > Project: Kylin > Issue Type: Improvement > Components: Storage - HBase >Affects Versions: v2.3.1 >Reporter: zhangwei >Assignee: zhangwei >Priority: Minor > Attachments: hdfs_ha.patch, hdfs_ha_2.patch, screenshot-1.png > > > 独立hbase 集群的 hdfs namenode主备切换导致Kylin无法正常工作的问题 > 痛点: > ylin.hbase.cluster.fs=hdfs://namenode:54310，在hbase的hdfs namenode切换之后会，和hbase > hdfs相关的步骤会出现失败 > 我们的方案:(所有的修改都基于kylin的代码，不需要hbase 做修改) > kylin.properties > kylin.hbase.cluster.fs=hdfs://nshbase > kylin.storage.hbase.cluster-hdfs-config-file=hbase-hdfs-site.xml > hbase-hdfs-site.xml配置 hbase hdfs的HA连接方式，其中dfs.nameservices= nshbase -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3831) 唯独超过62生成cuboid 错误
zhangwei created KYLIN-3831: --- Summary: 唯独超过62生成cuboid 错误 Key: KYLIN-3831 URL: https://issues.apache.org/jira/browse/KYLIN-3831 Project: Kylin Issue Type: Wish Components: Others Affects Versions: v2.3.1 Reporter: zhangwei -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3831) 唯独超过62生成cuboid 错误
[
https://issues.apache.org/jira/browse/KYLIN-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777694#comment-16777694
]
zhangwei commented on KYLIN-3831:
-
kylin.cube.rowkey.max-size 这个参数其实不能设置超过默认的63
因为生成cuboid的算法对64位的long做位移，一旦维度过多就会算出cuboid = -1 导致无法生成cube
测试代码
public class TestBitIndex { public static void main(String[] args) { long
fullMask = 0L; for (int i = 62; i >=0 ; i--) { fullMask |= 1L << i; }
System.out.println(fullMask); } }
9223372036854775807
public class TestBitIndex { public static void main(String[] args) { long
fullMask = 0L; for (int i = 65; i >=0 ; i--) { fullMask |= 1L << i; }
System.out.println(fullMask); } }
-1
DefaultCuboidScheduler
@Override
public List getSpanningCuboid(long cuboid) {
if (cuboid > max || cuboid < 0) {
throw new IllegalArgumentException("Cuboid " + cuboid + " is out of
scope 0-" + max);
}
List spanning = parent2child.get(cuboid);
if (spanning == null) {
return Collections.EMPTY_LIST;
}
return spanning;
}
> 唯独超过62生成cuboid 错误
> -
>
> Key: KYLIN-3831
> URL: https://issues.apache.org/jira/browse/KYLIN-3831
> Project: Kylin
> Issue Type: Wish
> Components: Others
>Affects Versions: v2.3.1
>Reporter: zhangwei
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3831) 唯独超过62生成cuboid 错误
[ https://issues.apache.org/jira/browse/KYLIN-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangwei reassigned KYLIN-3831: --- Assignee: zhangwei > 唯独超过62生成cuboid 错误 > - > > Key: KYLIN-3831 > URL: https://issues.apache.org/jira/browse/KYLIN-3831 > Project: Kylin > Issue Type: Wish > Components: Others >Affects Versions: v2.3.1 >Reporter: zhangwei >Assignee: zhangwei >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
