[jira] [Created] (KYLIN-5361) suggest set email content from hard code to configurable files
Li Can created KYLIN-5361: - Summary: suggest set email content from hard code to configurable files Key: KYLIN-5361 URL: https://issues.apache.org/jira/browse/KYLIN-5361 Project: Kylin Issue Type: Improvement Components: Others Affects Versions: 5.0-alpha Reporter: Li Can Assignee: Li Can Fix For: 5.0-alpha Attachments: emailcontent.png These all email information cannot configure by users with comfortable way, only way just to change the hard code, this method is not suitable for diversity customers. So we need adjust the hard code to configurable files, and set relative path to read the file, and then the users can easily modify the contents with the files rather than hard code. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5360) Kylin4 " select xx from table1 join table2 on table2.column = anyone constant value" query bug
Liu Zhao created KYLIN-5360:
---
Summary: Kylin4 " select xx from table1 join table2 on
table2.column = anyone constant value" query bug
Key: KYLIN-5360
URL: https://issues.apache.org/jira/browse/KYLIN-5360
Project: Kylin
Issue Type: Bug
Components: Query Engine
Affects Versions: v4.0.2, v4.0.1
Reporter: Liu Zhao
Attachments: image-2022-12-12-14-10-37-996.png,
image-2022-12-12-14-10-49-173.png
1) b.age = 22 放在 join 的 on 表达式部分，查询抛出异常：
with tmp_t as (
select
name,age
from
LZ_TEST
)
select
a.name,
b.age
from
LZ_TEST a
left join tmp_t b on a.name = b.name and b.age = 22
2）仅将 b.age = 22 移到 where 条件部分，查询成功
with tmp_t as (
select
name,age
from
LZ_TEST
)
select
a.name,
b.age
from
LZ_TEST a
left join tmp_t b on a.name = b.name
where b.age = 22
查看源码，发现在 calcite 的
org.apache.calcite.plan.RelOptUtil#splitJoinCondition(org.apache.calcite.rel.RelNode,
org.apache.calcite.rel.RelNode, org.apache.calcite.rex.RexNode,
java.util.List, java.util.List,
java.util.List) 对 equal 的判断有以下部分处理，要求表达式左右child只能是
RexInputRef，如果是常量值（RexLiteral）就会判断错误
{code:java}
// "=" and "IS NOT DISTINCT FROM" are the same except for how they
// treat nulls.
if (kind == SqlKind.EQUALS
|| (filterNulls != null && kind == SqlKind.IS_NOT_DISTINCT_FROM)) {
final List operands = call.getOperands();
if ((operands.get(0) instanceof RexInputRef)
&& (operands.get(1) instanceof RexInputRef)) {
RexInputRef op0 = (RexInputRef) operands.get(0);
RexInputRef op1 = (RexInputRef) operands.get(1);
RexInputRef leftField;
RexInputRef rightField;
if ((op0.getIndex() < leftFieldCount)
&& (op1.getIndex() >= leftFieldCount)) {
// Arguments were of form 'op0 = op1'
leftField = op0;
rightField = op1;
} else if (
(op1.getIndex() < leftFieldCount)
&& (op0.getIndex() >= leftFieldCount)) {
// Arguments were of form 'op1 = op0'
leftField = op1;
rightField = op0;
} else {
nonEquiList.add(condition);
return;
}
leftKeys.add(leftField.getIndex());
rightKeys.add(rightField.getIndex() - leftFieldCount);
if (filterNulls != null) {
filterNulls.add(kind == SqlKind.EQUALS);
}
return;
}
// Arguments were not field references, one from each side, so
// we fail. Fall through.
}
{code}
!image-2022-12-12-14-10-37-996.png!
!image-2022-12-12-14-10-49-173.png!
修复：
1）方式一：更改calcite splitJoinCondition 逻辑
2）方式二：更改 org.apache.kylin.query.optrule.OLAPJoinRule#convert 逻辑
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5245) When a job is submitted with deployMode=cluster and the application driver is abnormal, Kylin displays the job status as success
[ https://issues.apache.org/jira/browse/KYLIN-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17645923#comment-17645923 ] ASF GitHub Bot commented on KYLIN-5245: --- liuzhao-lz opened a new pull request, #2046: URL: https://github.com/apache/kylin/pull/2046 ## Proposed changes 对已修复的bug，选用更合理的实现方式. ## Branch to commit - [ ] Branch **kylin3** for v2.x to v3.x - [x] Branch **kylin4** for v4.x - [ ] Branch **kylin5** for v5.x ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [ ] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have added necessary documentation (if appropriate) - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at u...@kylin.apache.org or d...@kylin.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... > When a job is submitted with deployMode=cluster and the application driver is > abnormal, Kylin displays the job status as success > > > Key: KYLIN-5245 > URL: https://issues.apache.org/jira/browse/KYLIN-5245 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v4.0.1 >Reporter: Liu Zhao >Priority: Major > Attachments: image-2022-08-31-14-34-56-081.png > > > !image-2022-08-31-14-34-56-081.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [kylin] liuzhao-lz opened a new pull request, #2046: KYLIN-5245 code optimization, a more elegant processing method.
liuzhao-lz opened a new pull request, #2046: URL: https://github.com/apache/kylin/pull/2046 ## Proposed changes 对已修复的bug，选用更合理的实现方式. ## Branch to commit - [ ] Branch **kylin3** for v2.x to v3.x - [x] Branch **kylin4** for v4.x - [ ] Branch **kylin5** for v5.x ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have created an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [ ] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have added necessary documentation (if appropriate) - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at u...@kylin.apache.org or d...@kylin.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@kylin.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-5277) MetadataCleanupJob.isJobComplete should include NSparkMergingJob
[ https://issues.apache.org/jira/browse/KYLIN-5277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17645917#comment-17645917 ] ASF GitHub Bot commented on KYLIN-5277: --- liuzhao-lz commented on PR #2005: URL: https://github.com/apache/kylin/pull/2005#issuecomment-1345896607 The log data generated by kylin4 'NSparkMergingJob' should also be deleted, not just the 'NSparkCubingJob' log data. > MetadataCleanupJob.isJobComplete should include NSparkMergingJob > > > Key: KYLIN-5277 > URL: https://issues.apache.org/jira/browse/KYLIN-5277 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v4.0.1 >Reporter: Liu Zhao >Priority: Major > Fix For: Future > > > MetadataCleanupJob.isJobComplete should include NSparkMergingJob, because > NSparkMergingJob include autoMerge,etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [kylin] liuzhao-lz commented on pull request #2005: KYLIN-5277 MetadataCleanupJob.isJobComplete should include NSparkMerg…
liuzhao-lz commented on PR #2005: URL: https://github.com/apache/kylin/pull/2005#issuecomment-1345896607 The log data generated by kylin4 'NSparkMergingJob' should also be deleted, not just the 'NSparkCubingJob' log data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@kylin.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-5293) NLP occured when discard job
[ https://issues.apache.org/jira/browse/KYLIN-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17645914#comment-17645914 ] ASF GitHub Bot commented on KYLIN-5293: --- liuzhao-lz commented on PR #2017: URL: https://github.com/apache/kylin/pull/2017#issuecomment-1345881287 org.apache.kylin.engine.spark.metadata.cube.PathManager#deleteSegmentParquetStoragePath and org.apache.kylin.engine.spark.metadata.cube.PathManager#deleteJobTempPath has delete log information. > NLP occured when discard job > > > Key: KYLIN-5293 > URL: https://issues.apache.org/jira/browse/KYLIN-5293 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v4.0.1 >Reporter: Liu Zhao >Priority: Major > Attachments: image-2022-11-14-14-19-46-242.png, > image-2022-11-14-14-19-53-813.png > > > Segment not yet generated when discard job. > !image-2022-11-14-14-19-46-242.png! > !image-2022-11-14-14-19-53-813.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [kylin] liuzhao-lz commented on pull request #2017: KYLIN-5293 NLP occur when discard job
liuzhao-lz commented on PR #2017: URL: https://github.com/apache/kylin/pull/2017#issuecomment-1345881287 org.apache.kylin.engine.spark.metadata.cube.PathManager#deleteSegmentParquetStoragePath and org.apache.kylin.engine.spark.metadata.cube.PathManager#deleteJobTempPath has delete log information. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@kylin.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-5359) Kylin4 replace function expression convert error
[
https://issues.apache.org/jira/browse/KYLIN-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17645906#comment-17645906
]
ASF GitHub Bot commented on KYLIN-5359:
---
liuzhao-lz opened a new pull request, #2045:
URL: https://github.com/apache/kylin/pull/2045
## Proposed changes
statement "select name, replace(name, substring(name, 1, 1), '--') as
new_name from LZ_TEST_YUFA " query failed,
errorMsg：java.lang.ClassCastException: org.apache.spark.sql.Column cannot be
cast to java.lang.String while executing SQL: "select * from (select name,
replace(name, substring(name, 1, 1), '--') as new_name from LZ_TEST_YUFA) limit
5
Reason: In the ExpressionConverter, the substring returns a column object,
while in the replace conversion, the function parameter is forced to be
converted to a string.
## Branch to commit
- [ ] Branch **kylin3** for v2.x to v3.x
- [x] Branch **kylin4** for v4.x
- [ ] Branch **kylin5** for v5.x
## Types of changes
What types of changes does your code introduce to Kylin?
_Put an `x` in the boxes that apply_
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after
creating the PR. If you're unsure about any of them, don't hesitate to ask.
We're here to help! This is simply a reminder of what we are going to look for
before merging your code._
- [x] I have created an issue on [Kylin's
jira](https://issues.apache.org/jira/browse/KYLIN), and have described the
bug/feature there in detail
- [x] Commit messages in my PR start with the related jira ID, like
"KYLIN- Make Kylin project open-source"
- [ ] Compiling and unit tests pass locally with my changes
- [x] I have added tests that prove my fix is effective or that my feature
works
- [ ] I have added necessary documentation (if appropriate)
- [ ] Any dependent changes have been merged
## Further comments
If this is a relatively large or complex change, kick off the discussion at
u...@kylin.apache.org or d...@kylin.apache.org by explaining why you chose the
solution you did and what alternatives you considered, etc...
> Kylin4 replace function expression convert error
>
>
> Key: KYLIN-5359
> URL: https://issues.apache.org/jira/browse/KYLIN-5359
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v4.0.1, v4.0.2
>Reporter: Liu Zhao
>Priority: Major
> Attachments: image-2022-12-12-11-25-32-037.png,
> image-2022-12-12-11-25-41-178.png
>
>
> 查询语句形如：select name, replace(name, substring(name, 1, 1), '--') as new_name
> from LZ_TEST_YUFA 执行出错
> errorMsg：java.lang.ClassCastException: org.apache.spark.sql.Column cannot be
> cast to java.lang.String while executing SQL: "select * from (select name,
> replace(name, substring(name, 1, 1), '--') as new_name from LZ_TEST_YUFA)
> limit 5
> 原因：在ExpressionConverter中subtring返回的是一个column对象，而在replace转换中强制将函数参数转换为string
> {code:java}
> // ExpressionConverter
> case "replace " =>
> regexp_replace(k_lit(children.head),
> {color:red}children.apply(1).asInstanceOf[String],
> children.apply(2).asInstanceOf[String]{color})
> case "substring" | "substr" =>
> if (children.length == 3) { //substr(str1,startPos,length)
>{color:red} k_lit(children.head)
> .substr(k_lit(children.apply(1)),
> k_lit(children.apply(2))){color}
> } else if (children.length == 2) { //substr(str1,startPos)
> k_lit(children.head).
> substr(k_lit(children.apply(1)), k_lit(Int.MaxValue))
> } else {
> throw new UnsupportedOperationException(
> s"substring must provide three or two parameters under sparder")
> }
> case "initcapb" =>
> initcap(k_lit(children.head))
> {code}
> !image-2022-12-12-11-25-32-037.png!
> !image-2022-12-12-11-25-41-178.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [kylin] liuzhao-lz opened a new pull request, #2045: KYLIN-5359 'replace' function expression convert error
liuzhao-lz opened a new pull request, #2045: URL: https://github.com/apache/kylin/pull/2045 ## Proposed changes statement "select name, replace(name, substring(name, 1, 1), '--') as new_name from LZ_TEST_YUFA " query failed, errorMsg：java.lang.ClassCastException: org.apache.spark.sql.Column cannot be cast to java.lang.String while executing SQL: "select * from (select name, replace(name, substring(name, 1, 1), '--') as new_name from LZ_TEST_YUFA) limit 5 Reason: In the ExpressionConverter, the substring returns a column object, while in the replace conversion, the function parameter is forced to be converted to a string.   ## Branch to commit - [ ] Branch **kylin3** for v2.x to v3.x - [x] Branch **kylin4** for v4.x - [ ] Branch **kylin5** for v5.x ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have created an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [x] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have added necessary documentation (if appropriate) - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at u...@kylin.apache.org or d...@kylin.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@kylin.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KYLIN-5359) Kylin4 replace function expression convert error
Liu Zhao created KYLIN-5359:
---
Summary: Kylin4 replace function expression convert error
Key: KYLIN-5359
URL: https://issues.apache.org/jira/browse/KYLIN-5359
Project: Kylin
Issue Type: Bug
Components: Query Engine
Affects Versions: v4.0.2, v4.0.1
Reporter: Liu Zhao
Attachments: image-2022-12-12-11-25-32-037.png,
image-2022-12-12-11-25-41-178.png
查询语句形如：select name, replace(name, substring(name, 1, 1), '--') as new_name from
LZ_TEST_YUFA 执行出错
errorMsg：java.lang.ClassCastException: org.apache.spark.sql.Column cannot be
cast to java.lang.String while executing SQL: "select * from (select name,
replace(name, substring(name, 1, 1), '--') as new_name from LZ_TEST_YUFA) limit
5
原因：在ExpressionConverter中subtring返回的是一个column对象，而在replace转换中强制将函数参数转换为string
{code:java}
// ExpressionConverter
case "replace " =>
regexp_replace(k_lit(children.head),
{color:red}children.apply(1).asInstanceOf[String],
children.apply(2).asInstanceOf[String]{color})
case "substring" | "substr" =>
if (children.length == 3) { //substr(str1,startPos,length)
{color:red} k_lit(children.head)
.substr(k_lit(children.apply(1)),
k_lit(children.apply(2))){color}
} else if (children.length == 2) { //substr(str1,startPos)
k_lit(children.head).
substr(k_lit(children.apply(1)), k_lit(Int.MaxValue))
} else {
throw new UnsupportedOperationException(
s"substring must provide three or two parameters under sparder")
}
case "initcapb" =>
initcap(k_lit(children.head))
{code}
!image-2022-12-12-11-25-32-037.png!
!image-2022-12-12-11-25-41-178.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
