[GitHub] [kylin] coveralls commented on issue #835: KYLIN-4159 The first step of job will fail and throw "Column \'xx\' i…
coveralls commented on issue #835: KYLIN-4159 The first step of job will fail and throw "Column \'xx\' i… URL: https://github.com/apache/kylin/pull/835#issuecomment-529769853 ## Pull Request Test Coverage Report for [Build 4994](https://coveralls.io/builds/25621705) * **0** of **3** **(0.0%)** changed or added relevant lines in **1** file are covered. * **1** unchanged line in **1** file lost coverage. * Overall coverage decreased (**-0.0003%**) to **28.078%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [source-jdbc/src/main/java/org/apache/kylin/source/jdbc/JdbcHiveInputBase.java](https://coveralls.io/builds/25621705/source?filename=source-jdbc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fjdbc%2FJdbcHiveInputBase.java#L232) | 0 | 3 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [source-jdbc/src/main/java/org/apache/kylin/source/jdbc/JdbcHiveInputBase.java](https://coveralls.io/builds/25621705/source?filename=source-jdbc%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fjdbc%2FJdbcHiveInputBase.java#L233) | 1 | 23.62% | | Totals | [](https://coveralls.io/builds/25621705) | | :-- | --: | | Change from base [Build 4995](https://coveralls.io/builds/25621724): | -0.0003% | | Covered Lines: | 23454 | | Relevant Lines: | 83533 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] hit-lacus commented on a change in pull request #835: KYLIN-4159 The first step of job will fail and throw "Column \'xx\' i…
hit-lacus commented on a change in pull request #835: KYLIN-4159 The first step of job will fail and throw "Column \'xx\' i… URL: https://github.com/apache/kylin/pull/835#discussion_r322540274 ## File path: source-jdbc/src/main/java/org/apache/kylin/source/jdbc/JdbcHiveInputBase.java ## @@ -222,11 +223,13 @@ protected AbstractExecutable createSqoopToFlatHiveStep(String jobWorkingDir, Str String splitTableAlias; String splitColumn; +String quoteFullNamedColumn; // `table.column` String splitDatabase; TblColRef splitColRef = determineSplitColumn(); splitTableAlias = splitColRef.getTableAlias(); splitColumn = getColumnIdentityQuoted(splitColRef, jdbcMetadataDialect, metaMap, true); +quoteFullNamedColumn = quoteIdentifier(partCol, jdbcMetadataDialect.getDialect()).toLowerCase(); Review comment: Use `toLowerCase` is forbidden. ``` Forbidden method invocation: java.lang.String#toLowerCase() [Uses default locale] [ERROR] in org.apache.kylin.source.jdbc.JdbcHiveInputBase$JdbcBaseBatchCubingInputSide (JdbcHiveInputBase.java:232) [ERROR] Scanned 34 (and 354 related) class file(s) for forbidden API invocations (in 0.10s), 1 error(s). ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] hit-lacus commented on a change in pull request #835: KYLIN-4159 The first step of job will fail and throw "Column \'xx\' i…
hit-lacus commented on a change in pull request #835: KYLIN-4159 The first step of job will fail and throw "Column \'xx\' i… URL: https://github.com/apache/kylin/pull/835#discussion_r322540274 ## File path: source-jdbc/src/main/java/org/apache/kylin/source/jdbc/JdbcHiveInputBase.java ## @@ -222,11 +223,13 @@ protected AbstractExecutable createSqoopToFlatHiveStep(String jobWorkingDir, Str String splitTableAlias; String splitColumn; +String quoteFullNamedColumn; // `table.column` String splitDatabase; TblColRef splitColRef = determineSplitColumn(); splitTableAlias = splitColRef.getTableAlias(); splitColumn = getColumnIdentityQuoted(splitColRef, jdbcMetadataDialect, metaMap, true); +quoteFullNamedColumn = quoteIdentifier(partCol, jdbcMetadataDialect.getDialect()).toLowerCase(); Review comment: Use `toLowerCase` is forbidden. ``` Forbidden method invocation: java.lang.String#toLowerCase() [Uses default locale] [ERROR] in org.apache.kylin.source.jdbc.JdbcHiveInputBase$JdbcBaseBatchCubingInputSide (JdbcHiveInputBase.java:232) ``` [ERROR] Scanned 34 (and 354 related) class file(s) for forbidden API invocations (in 0.10s), 1 error(s). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (KYLIN-4162) After drop the build task on the monitor page, subsequent segments cannot be constructed.
[
https://issues.apache.org/jira/browse/KYLIN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925772#comment-16925772
]
Xiaoxiang Yu edited comment on KYLIN-4162 at 9/9/19 3:32 PM:
-
Hi, [~zhangyaqian], following is my understanding, did I understand it in right
way? Happy to hear your reply.
-
-
If a discard job be found in Coordinator, it maybe dropped because of no quota.
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1186
{code:java}
else if (ExecutableState.DISCARDED.equals(jobState)) {
// if the job has been discard manually, just think that the segment is not
in building
logger.info("job:{} is discard, reset the job state in metaStore", jobId);
SegmentBuildState.BuildState state = new SegmentBuildState.BuildState();
state.setBuildStartTime(0);
state.setState(SegmentBuildState.BuildState.State.WAIT);
state.setJobId(cubingJob.getId());
streamMetadataStore.updateSegmentBuildState(cubeName,
segmentState.getSegmentName(), state);
segmentState.setState(state);
}
{code}
In above code, if coordinator find the state of job is *DISCARDED*, coordinator
will change the state of current segment in stream metadata from *building* to
*wait*, and try to add to *result* to mark it as it need to be resubmit a new
build job.
But, before add the segment the *result*, coordinator will first check it has
enough quota. If not, coordinator won't add it into *result*, thus that segment
will be lost.
When check in the next time, it will be ignored because the state is *wait*.
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1160
{code:java}
if (segmentState.isInBuilding()) {
{code}
was (Author: hit_lacus):
If a discard job found in Coordinator, it maybe be detected but dropped .
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1186
{code:java}
else if (ExecutableState.DISCARDED.equals(jobState)) {
// if the job has been discard manually, just think that the segment is not
in building
logger.info("job:{} is discard, reset the job state in metaStore", jobId);
SegmentBuildState.BuildState state = new SegmentBuildState.BuildState();
state.setBuildStartTime(0);
state.setState(SegmentBuildState.BuildState.State.WAIT);
state.setJobId(cubingJob.getId());
streamMetadataStore.updateSegmentBuildState(cubeName,
segmentState.getSegmentName(), state);
segmentState.setState(state);
}
{code}
In above code, if coordinator find the state of job is *DISCARDED*, coordinator
will change the state of current segment in stream metadata from *building* to
*wait*, and try to add to *result* to mark it need to be rebuilt.
But, before add the segment the *result*, coordinator will first check it has
enough quota. If not, coordinator won't add it into *result*, thus that segment
will be lost.
When check in next time, it will be ignore because the state is *wait*.
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1160
{code:java}
if (segmentState.isInBuilding()) {
{code}
Did I understand it in right way? Happy to hear your reply.
> After drop the build task on the monitor page, subsequent segments cannot be
> constructed.
> -
>
> Key: KYLIN-4162
> URL: https://issues.apache.org/jira/browse/KYLIN-4162
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
>Affects Versions: v3.0.0-alpha2
>Reporter: Yaqian Zhang
>Priority: Major
> Fix For: Future
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (KYLIN-4162) After drop the build task on the monitor page, subsequent segments cannot be constructed.
[
https://issues.apache.org/jira/browse/KYLIN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925772#comment-16925772
]
Xiaoxiang Yu edited comment on KYLIN-4162 at 9/9/19 3:26 PM:
-
If a discard job found in Coordinator, it maybe be detected but dropped .
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1186
{code:java}
else if (ExecutableState.DISCARDED.equals(jobState)) {
// if the job has been discard manually, just think that the segment is not
in building
logger.info("job:{} is discard, reset the job state in metaStore", jobId);
SegmentBuildState.BuildState state = new SegmentBuildState.BuildState();
state.setBuildStartTime(0);
state.setState(SegmentBuildState.BuildState.State.WAIT);
state.setJobId(cubingJob.getId());
streamMetadataStore.updateSegmentBuildState(cubeName,
segmentState.getSegmentName(), state);
segmentState.setState(state);
}
{code}
In above code, if coordinator find the state of job is *DISCARDED*, coordinator
will change the state of current segment in stream metadata from *building* to
*wait*, and try to add to *result* to mark it need to be rebuilt.
But, before add the segment the *result*, coordinator will first check it has
enough quota. If not, coordinator won't add it into *result*, thus that segment
will be lost.
When check in next time, it will be ignore because the state is *wait*.
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1160
{code:java}
if (segmentState.isInBuilding()) {
{code}
Did I understand it in right way? Happy to hear your reply.
was (Author: hit_lacus):
If a discard job found in Coordinator, it will be detected at
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1186
.
{code:java}
else if (ExecutableState.DISCARDED.equals(jobState)) {
// if the job has been discard manually, just think that the segment is not
in building
logger.info("job:{} is discard, reset the job state in metaStore", jobId);
SegmentBuildState.BuildState state = new SegmentBuildState.BuildState();
state.setBuildStartTime(0);
state.setState(SegmentBuildState.BuildState.State.WAIT);
state.setJobId(cubingJob.getId());
streamMetadataStore.updateSegmentBuildState(cubeName,
segmentState.getSegmentName(), state);
segmentState.setState(state);
}
{code}
In above code, if coordinator find the state of job is *DISCARDED*, coordinator
will change the entry in stream metadata from building to wait, and try to add
to *result* to mark it need to be rebuilt.
But, before add the segment the *result*, coordinator will first check it has
enough quota. If not, coordinator won't add it into *result*, thus that segment
will be lost.
When check in next time, it will be ignore because the state is *wait*.
{code:java}
if (segmentState.isInBuilding()) {
{code}
> After drop the build task on the monitor page, subsequent segments cannot be
> constructed.
> -
>
> Key: KYLIN-4162
> URL: https://issues.apache.org/jira/browse/KYLIN-4162
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
>Affects Versions: v3.0.0-alpha2
>Reporter: Yaqian Zhang
>Priority: Major
> Fix For: Future
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KYLIN-4162) After drop the build task on the monitor page, subsequent segments cannot be constructed.
[
https://issues.apache.org/jira/browse/KYLIN-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925772#comment-16925772
]
Xiaoxiang Yu commented on KYLIN-4162:
-
If a discard job found in Coordinator, it will be detected at
https://github.com/apache/kylin/blob/1cd284bcd38c4787e13b102a28df7a533ef82de6/stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java#L1186
.
{code:java}
else if (ExecutableState.DISCARDED.equals(jobState)) {
// if the job has been discard manually, just think that the segment is not
in building
logger.info("job:{} is discard, reset the job state in metaStore", jobId);
SegmentBuildState.BuildState state = new SegmentBuildState.BuildState();
state.setBuildStartTime(0);
state.setState(SegmentBuildState.BuildState.State.WAIT);
state.setJobId(cubingJob.getId());
streamMetadataStore.updateSegmentBuildState(cubeName,
segmentState.getSegmentName(), state);
segmentState.setState(state);
}
{code}
In above code, if coordinator find the state of job is *DISCARDED*, coordinator
will change the entry in stream metadata from building to wait, and try to add
to *result* to mark it need to be rebuilt.
But, before add the segment the *result*, coordinator will first check it has
enough quota. If not, coordinator won't add it into *result*, thus that segment
will be lost.
When check in next time, it will be ignore because the state is *wait*.
{code:java}
if (segmentState.isInBuilding()) {
{code}
> After drop the build task on the monitor page, subsequent segments cannot be
> constructed.
> -
>
> Key: KYLIN-4162
> URL: https://issues.apache.org/jira/browse/KYLIN-4162
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
>Affects Versions: v3.0.0-alpha2
>Reporter: Yaqian Zhang
>Priority: Major
> Fix For: Future
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (KYLIN-4153) Failed to read big resource /dict/xxxx at "Build Dimension Dictionary" Step
[
https://issues.apache.org/jira/browse/KYLIN-4153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925702#comment-16925702
]
ASF GitHub Bot commented on KYLIN-4153:
---
nichunen commented on pull request #818: KYLIN-4153 Delete marker if real file
not exists
URL: https://github.com/apache/kylin/pull/818
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Failed to read big resource /dict/ at "Build Dimension Dictionary" Step
>
>
> Key: KYLIN-4153
> URL: https://issues.apache.org/jira/browse/KYLIN-4153
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.6.0
>Reporter: Xiaoxiang Yu
>Assignee: Xiaoxiang Yu
>Priority: Major
>
> At the version of *Kylin 2.6.0*, kylin team has introduce an important
> refactor of Kylin's Metadata Store, which add a lot of enhancement such as
> upload/download metadata concurrently, store metadata with JDBC etc. Please
> refer to https://issues.apache.org/jira/browse/KYLIN-3671 for detail.
>
> When kylin want to save a *big resource*(such as dict or snapshot) into
> metadata store, it won't store it into metadata store(HBase or RDBMS)
> directly. Instead, kylin will first {color:red}save it into HDFS(Step
> 1){color}, and then {color:red}write a empty byte array as marker into
> metadata store(Step 2) {color}. If first action succeed and second action
> failed, a rollback method will be called to revert modification for HDFS
> files. We could regard it as a complete and atomic transaction.
>
> {color:#0747A6}Here is part of the source code added in KYLIN-3671.{color}
> Check it at
> https://github.com/apache/kylin/blob/8737bc1f555a2789a67462c8f8420b6ab3be97ce/core-common/src/main/java/org/apache/kylin/common/persistence/PushdownResourceStore.java#L58
> .
> {code:java}
> final void putBigResource(String resPath, ContentWriter content, long newTS)
> throws IOException {
> // pushdown the big resource to DFS file
> RollbackablePushdown pushdown = writePushdown(resPath, content); // Step
> 1: write big resource into HDFS
> try {
> // write a marker in resource store, to indicate the resource is now
> available
> logger.debug("Writing marker for big resource {}", resPath);
> putResourceWithRetry(resPath,
> ContentWriter.create(BytesUtil.EMPTY_BYTE_ARRAY), newTS); // Step 2: write
> marker into HBase/RDBMS
> } catch (Throwable ex) {
> pushdown.rollback();
> throw ex;
> } finally {
> pushdown.close();
> }
> }
> {code}
>
>
>
> But in some case, both step 1 and step 2 succeed but an exception still
> throwed in step 2,{color:red} the rollback won't clear marker written in Step
> 2{color}, which break the atomicity of this put action, thus cause the
> FileNotFoundException when Kylin want to read that dict later.
>
>
>
> {color:#0747A6}Here is part of reporter's kylin.log of incomplete rollback
> action.{color}
>
>
> {noformat}
> 2019-08-29 05:13:51,237 INFO [Scheduler 169045403 Job
> ca4a4a08-54e2-b922-70bb-2aa2bf58709f-492] dict.DictionaryManager:388 : Saving
> dictionary at
> /dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict
> 2019-08-29 05:13:51,238 DEBUG [Scheduler 169045403 Job
> ca4a4a08-54e2-b922-70bb-2aa2bf58709f-492] persistence.HDFSResourceStore:98 :
> Writing pushdown file
> /kylin/kylin_metadata/resources/dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict.temp.-1798610090
> 2019-08-29 05:13:51,256 DEBUG [Scheduler 169045403 Job
> ca4a4a08-54e2-b922-70bb-2aa2bf58709f-492] persistence.HDFSResourceStore:117 :
> Move
> /kylin/kylin_metadata/resources/dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict.temp.-1798610090
> to
> /kylin/kylin_metadata/resources/dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict
> 2019-08-29 05:13:51,258 DEBUG [Scheduler 169045403 Job
> ca4a4a08-54e2-b922-70bb-2aa2bf58709f-492] persistence.HDFSResourceStore:65 :
> Writing marker for big resource
> /dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict
> 2019-08-29 05:13:56,263 WARN
> [hconnection-0x56f3258e-shared--pool10944-t54867] client.AsyncProcess:1263 :
> #10545, table=kylin_metadata, attempt=1/1 failed=1ops, last exception:
> java.io.IOException: Call to tx-dn41.data/10.14.243.51:60020 failed on local
> exception:
[jira] [Commented] (KYLIN-4153) Failed to read big resource /dict/xxxx at "Build Dimension Dictionary" Step
[
https://issues.apache.org/jira/browse/KYLIN-4153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925703#comment-16925703
]
ASF subversion and git services commented on KYLIN-4153:
Commit 7e117e27764dc94cd627b0bd3dc4f4bbbf7f4a3e in kylin's branch
refs/heads/master from XiaoxiangYu
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=7e117e2 ]
KYLIN-4153 Delete marker if real file not exists
> Failed to read big resource /dict/ at "Build Dimension Dictionary" Step
>
>
> Key: KYLIN-4153
> URL: https://issues.apache.org/jira/browse/KYLIN-4153
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.6.0
>Reporter: Xiaoxiang Yu
>Assignee: Xiaoxiang Yu
>Priority: Major
>
> At the version of *Kylin 2.6.0*, kylin team has introduce an important
> refactor of Kylin's Metadata Store, which add a lot of enhancement such as
> upload/download metadata concurrently, store metadata with JDBC etc. Please
> refer to https://issues.apache.org/jira/browse/KYLIN-3671 for detail.
>
> When kylin want to save a *big resource*(such as dict or snapshot) into
> metadata store, it won't store it into metadata store(HBase or RDBMS)
> directly. Instead, kylin will first {color:red}save it into HDFS(Step
> 1){color}, and then {color:red}write a empty byte array as marker into
> metadata store(Step 2) {color}. If first action succeed and second action
> failed, a rollback method will be called to revert modification for HDFS
> files. We could regard it as a complete and atomic transaction.
>
> {color:#0747A6}Here is part of the source code added in KYLIN-3671.{color}
> Check it at
> https://github.com/apache/kylin/blob/8737bc1f555a2789a67462c8f8420b6ab3be97ce/core-common/src/main/java/org/apache/kylin/common/persistence/PushdownResourceStore.java#L58
> .
> {code:java}
> final void putBigResource(String resPath, ContentWriter content, long newTS)
> throws IOException {
> // pushdown the big resource to DFS file
> RollbackablePushdown pushdown = writePushdown(resPath, content); // Step
> 1: write big resource into HDFS
> try {
> // write a marker in resource store, to indicate the resource is now
> available
> logger.debug("Writing marker for big resource {}", resPath);
> putResourceWithRetry(resPath,
> ContentWriter.create(BytesUtil.EMPTY_BYTE_ARRAY), newTS); // Step 2: write
> marker into HBase/RDBMS
> } catch (Throwable ex) {
> pushdown.rollback();
> throw ex;
> } finally {
> pushdown.close();
> }
> }
> {code}
>
>
>
> But in some case, both step 1 and step 2 succeed but an exception still
> throwed in step 2,{color:red} the rollback won't clear marker written in Step
> 2{color}, which break the atomicity of this put action, thus cause the
> FileNotFoundException when Kylin want to read that dict later.
>
>
>
> {color:#0747A6}Here is part of reporter's kylin.log of incomplete rollback
> action.{color}
>
>
> {noformat}
> 2019-08-29 05:13:51,237 INFO [Scheduler 169045403 Job
> ca4a4a08-54e2-b922-70bb-2aa2bf58709f-492] dict.DictionaryManager:388 : Saving
> dictionary at
> /dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict
> 2019-08-29 05:13:51,238 DEBUG [Scheduler 169045403 Job
> ca4a4a08-54e2-b922-70bb-2aa2bf58709f-492] persistence.HDFSResourceStore:98 :
> Writing pushdown file
> /kylin/kylin_metadata/resources/dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict.temp.-1798610090
> 2019-08-29 05:13:51,256 DEBUG [Scheduler 169045403 Job
> ca4a4a08-54e2-b922-70bb-2aa2bf58709f-492] persistence.HDFSResourceStore:117 :
> Move
> /kylin/kylin_metadata/resources/dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict.temp.-1798610090
> to
> /kylin/kylin_metadata/resources/dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict
> 2019-08-29 05:13:51,258 DEBUG [Scheduler 169045403 Job
> ca4a4a08-54e2-b922-70bb-2aa2bf58709f-492] persistence.HDFSResourceStore:65 :
> Writing marker for big resource
> /dict/KYLIN_VIEW.USER_SECRET_TABLE/COUNTRY/66292068-e8eb-975a-3e44-b56c933c14cc.dict
> 2019-08-29 05:13:56,263 WARN
> [hconnection-0x56f3258e-shared--pool10944-t54867] client.AsyncProcess:1263 :
> #10545, table=kylin_metadata, attempt=1/1 failed=1ops, last exception:
> java.io.IOException: Call to tx-dn41.data/10.14.243.51:60020 failed on local
> exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=2662317,
> waitTime=5001, operationTimeout=5000 expired. on
> tx-dn41.data,60020,1565943919204, tracking started Thu Aug 29 05:13:51
> GMT+08:00 2019; not retrying 1 - final failure
>
[GitHub] [kylin] nichunen merged pull request #818: KYLIN-4153 Delete marker if real file not exists
nichunen merged pull request #818: KYLIN-4153 Delete marker if real file not exists URL: https://github.com/apache/kylin/pull/818 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] nichunen commented on a change in pull request #818: KYLIN-4153 Delete marker if real file not exists
nichunen commented on a change in pull request #818: KYLIN-4153 Delete marker
if real file not exists
URL: https://github.com/apache/kylin/pull/818#discussion_r320229235
##
File path:
core-common/src/main/java/org/apache/kylin/common/persistence/PushdownResourceStore.java
##
@@ -167,23 +170,27 @@ protected InputStream openPushdown(String resPath)
throws IOException {
try {
Path p = pushdownPath(resPath);
FileSystem fs = pushdownFS();
-if (fs.exists(p))
+if (fs.exists(p)) {
return fs.open(p);
-else
+} else {
throw new FileNotFoundException(p.toString() + " (FS: " + fs
+ ")");
-
+}
+} catch (FileNotFoundException fileNotFound) {
Review comment:
Seems no need to catch "FileNotFoundException"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (KYLIN-4143) truncate spark executable job output
[ https://issues.apache.org/jira/browse/KYLIN-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925688#comment-16925688 ] ASF subversion and git services commented on KYLIN-4143: Commit 893a75a8b1accec37fd15a2d055ee58243fa56c4 in kylin's branch refs/heads/master from Zhou Kang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=893a75a ] KYLIN-4143 truncate spark executable job output > truncate spark executable job output > - > > Key: KYLIN-4143 > URL: https://issues.apache.org/jira/browse/KYLIN-4143 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: ZhouKang >Priority: Major > Attachments: KYLIN-4143.master.001.patch > > > > truncate spark job's output when the job exec ret is not equal 0, which made > the execute output content too large. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KYLIN-4143) truncate spark executable job output
[ https://issues.apache.org/jira/browse/KYLIN-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925687#comment-16925687 ] ASF GitHub Bot commented on KYLIN-4143: --- nichunen commented on pull request #831: KYLIN-4143 truncate spark executable job output URL: https://github.com/apache/kylin/pull/831 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > truncate spark executable job output > - > > Key: KYLIN-4143 > URL: https://issues.apache.org/jira/browse/KYLIN-4143 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: ZhouKang >Priority: Major > Attachments: KYLIN-4143.master.001.patch > > > > truncate spark job's output when the job exec ret is not equal 0, which made > the execute output content too large. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [kylin] nichunen merged pull request #831: KYLIN-4143 truncate spark executable job output
nichunen merged pull request #831: KYLIN-4143 truncate spark executable job output URL: https://github.com/apache/kylin/pull/831 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4055) cube quey and ad-hoc query return different meta info
[ https://issues.apache.org/jira/browse/KYLIN-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925645#comment-16925645 ] ASF GitHub Bot commented on KYLIN-4055: --- nichunen commented on pull request #715: KYLIN-4055 cube quey and ad-hoc query return different meta info URL: https://github.com/apache/kylin/pull/715 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > cube quey and ad-hoc query return different meta info > - > > Key: KYLIN-4055 > URL: https://issues.apache.org/jira/browse/KYLIN-4055 > Project: Kylin > Issue Type: Improvement >Reporter: jinguowei >Assignee: Yaqian Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [kylin] nichunen commented on issue #715: KYLIN-4055 cube quey and ad-hoc query return different meta info
nichunen commented on issue #715: KYLIN-4055 cube quey and ad-hoc query return different meta info URL: https://github.com/apache/kylin/pull/715#issuecomment-529451992 Will be done by https://github.com/apache/kylin/pull/823/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] nichunen closed pull request #715: KYLIN-4055 cube quey and ad-hoc query return different meta info
nichunen closed pull request #715: KYLIN-4055 cube quey and ad-hoc query return different meta info URL: https://github.com/apache/kylin/pull/715 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (KYLIN-4162) After dropping the build task on the monitor page, subsequent segments cannot be constructed.
Yaqian Zhang created KYLIN-4162: --- Summary: After dropping the build task on the monitor page, subsequent segments cannot be constructed. Key: KYLIN-4162 URL: https://issues.apache.org/jira/browse/KYLIN-4162 Project: Kylin Issue Type: Bug Components: Real-time Streaming Affects Versions: v3.0.0-alpha2 Reporter: Yaqian Zhang Fix For: Future -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KYLIN-3966) Metric multiplication and metric division not working in query
[ https://issues.apache.org/jira/browse/KYLIN-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925412#comment-16925412 ] Yaqian Zhang commented on KYLIN-3966: - Hello Gladson Vas: As far as I am concerned, Kylin currently does not support column computing with * or /. > Metric multiplication and metric division not working in query > -- > > Key: KYLIN-3966 > URL: https://issues.apache.org/jira/browse/KYLIN-3966 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Reporter: Gladson Vas >Priority: Blocker > > When i try to multiply or divide 2 metric columns in a query > eg: select col1*col2 from table > select col1/col2 from table. > it throws the following errors > That both of the two sides of the BinaryTupleExpression own columns is not > supported for * while executing SQL > That both of the two sides of the BinaryTupleExpression own columns is not > supported for / while executing SQL. > But when i use the + or - operators in the query it seems to be working fine. > Is there any way to support these 2 operators as well? > Thanks, > Gladson > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (KYLIN-4068) Automatically add limit has bug
[
https://issues.apache.org/jira/browse/KYLIN-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
weibin0516 resolved KYLIN-4068.
---
Resolution: Fixed
> Automatically add limit has bug
> ---
>
> Key: KYLIN-4068
> URL: https://issues.apache.org/jira/browse/KYLIN-4068
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.6.2
>Reporter: weibin0516
>Assignee: weibin0516
>Priority: Major
> Fix For: v3.0.0-alpha2
>
>
> {code:sql}
> SELECT E_Name FROM Employees_China
> UNION
> SELECT E_Name FROM Employees_USA
> {code}
> will convert to
> {code:sql}
> SELECT E_Name FROM Employees_China
> UNION
> SELECT E_Name FROM Employees_USA
> LIMIT 5
> {code}
> This limit is not working on the result of union, but on SELECT E_Name FROM
> Employees_USA.
> We should use a more secure way to achieve the limit effect.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KYLIN-4068) Automatically add limit has bug
[
https://issues.apache.org/jira/browse/KYLIN-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
weibin0516 updated KYLIN-4068:
--
Fix Version/s: v3.0.0-alpha2
Description:
{code:sql}
SELECT E_Name FROM Employees_China
UNION
SELECT E_Name FROM Employees_USA
{code}
will convert to
{code:sql}
SELECT E_Name FROM Employees_China
UNION
SELECT E_Name FROM Employees_USA
LIMIT 5
{code}
This limit is not working on the result of union, but on SELECT E_Name FROM
Employees_USA.
We should use a more secure way to achieve the limit effect.
was:
{code:sql}
SELECT E_Name FROM Employees_China
UNION
SELECT E_Name FROM Employees_USA
{code}
will convert to
{code:sql}
SELECT E_Name FROM Employees_China
UNION
SELECT E_Name FROM Employees_USA
LIMIT 5
{code}
This limit is not working on the result of union, but on SELECT E_Name FROM
Employees_USA.
We should use a more secure way to achieve the limit effect.
> Automatically add limit has bug
> ---
>
> Key: KYLIN-4068
> URL: https://issues.apache.org/jira/browse/KYLIN-4068
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.6.2
>Reporter: weibin0516
>Assignee: weibin0516
>Priority: Major
> Fix For: v3.0.0-alpha2
>
>
> {code:sql}
> SELECT E_Name FROM Employees_China
> UNION
> SELECT E_Name FROM Employees_USA
> {code}
> will convert to
> {code:sql}
> SELECT E_Name FROM Employees_China
> UNION
> SELECT E_Name FROM Employees_USA
> LIMIT 5
> {code}
> This limit is not working on the result of union, but on SELECT E_Name FROM
> Employees_USA.
> We should use a more secure way to achieve the limit effect.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
