[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256805#comment-17256805 ] ASF GitHub Bot commented on KYLIN-4813: --- hit-lacus closed pull request #1525: URL: https://github.com/apache/kylin/pull/1525 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256804#comment-17256804 ] ASF GitHub Bot commented on KYLIN-4813: --- hit-lacus commented on pull request #1525: URL: https://github.com/apache/kylin/pull/1525#issuecomment-752824252 It is better to double check the internal or add more test coverage. I will close this at the moment. Let's re-design this later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256772#comment-17256772
]
ASF GitHub Bot commented on KYLIN-4813:
---
zzcclp commented on a change in pull request #1525:
URL: https://github.com/apache/kylin/pull/1525#discussion_r550371200
##
File path:
core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
##
@@ -206,12 +207,15 @@ protected Properties getProperties(Collection
propertyKeys) {
}
protected StrSubstitutor getSubstitutor() {
-// env > properties
-final Map all = Maps.newHashMap();
-all.putAll((Map) properties);
-all.putAll(System.getenv());
+return new StrSubstitutor(getPropertiesMap());
+}
-return new StrSubstitutor(all);
+protected Map getPropertiesMap() {
+if (propertiesMap.isEmpty()) {
+propertiesMap.putAll((Map) properties);
+propertiesMap.putAll(System.getenv());
+}
Review comment:
When calling 'getPropertiesMap()' at the first time, it put all values
of properties and System.getenv() into propertiesMap, but if there are some
values be changed, the values in propertiesMap cann't be changed too?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256343#comment-17256343 ] ASF GitHub Bot commented on KYLIN-4813: --- zhangayqian opened a new pull request #1525: URL: https://github.com/apache/kylin/pull/1525 ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [ ] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239539#comment-17239539 ] ASF subversion and git services commented on KYLIN-4813: Commit 2dca9ef9a52cf1adb12684e0ba576b91e96b3792 in kylin's branch refs/heads/kylin-on-parquet-v2 from yaqian.zhang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=2dca9ef ] KYLIN-4813 Refactor spark build log > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239540#comment-17239540 ] ASF subversion and git services commented on KYLIN-4813: Commit c0b27873f7ff6cfcaa4d404214c41288bd81a4f4 in kylin's branch refs/heads/kylin-on-parquet-v2 from yaqian.zhang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=c0b2787 ] KYLIN-4813 Add spark executor log4j KYLIN-4813 Minor fix > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239543#comment-17239543 ] ASF subversion and git services commented on KYLIN-4813: Commit c4d3168268f189aaec4e7406d08dcc10fffb2888 in kylin's branch refs/heads/kylin-on-parquet-v2 from zl03 [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=c4d3168 ] KYLIN-4813 Add download all log link in front-end > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239544#comment-17239544 ] ASF subversion and git services commented on KYLIN-4813: Commit 9a4df78f291cd7e34a30cbbdd9ec43124e6cd46f in kylin's branch refs/heads/kylin-on-parquet-v2 from yaqian.zhang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=9a4df78 ] KYLIN-4813 Some adjustments for executor-side log collection > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239542#comment-17239542 ] ASF subversion and git services commented on KYLIN-4813: Commit 86aae4dc1cd6f364c36e9f47db84b20956b2b911 in kylin's branch refs/heads/kylin-on-parquet-v2 from Xiaoxiang Yu [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=86aae4d ] KYLIN-4813 Fix several bugs for executor-side log collection - add --files to upload user-defined log4j properties - fix ClassName error - fix executor.extraJavaOptions be overwrote in Driver side - fix security issue (HTTP response splitting) - code style etc. > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239541#comment-17239541 ] ASF subversion and git services commented on KYLIN-4813: Commit c0b27873f7ff6cfcaa4d404214c41288bd81a4f4 in kylin's branch refs/heads/kylin-on-parquet-v2 from yaqian.zhang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=c0b2787 ] KYLIN-4813 Add spark executor log4j KYLIN-4813 Minor fix > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239538#comment-17239538 ] ASF GitHub Bot commented on KYLIN-4813: --- hit-lacus merged pull request #1481: URL: https://github.com/apache/kylin/pull/1481 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239534#comment-17239534 ] ASF GitHub Bot commented on KYLIN-4813: --- zzcclp commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-734684079 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239219#comment-17239219 ] ASF GitHub Bot commented on KYLIN-4813: --- codecov-io edited a comment on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-729431698 # [Codecov](https://codecov.io/gh/apache/kylin/pull/1481?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`kylin-on-parquet-v2@0894c0d`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/kylin/pull/1481?src=pr=tree) ```diff @@ Coverage Diff @@ ## kylin-on-parquet-v2#1481 +/- ## == Coverage ? 24.10% Complexity ? 4609 == Files ? 1135 Lines ?64421 Branches ? 9504 == Hits ?15527 Misses ?47282 Partials ? 1612 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/1481?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/1481?src=pr=footer). Last update [0894c0d...68df894](https://codecov.io/gh/apache/kylin/pull/1481?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239217#comment-17239217 ] ASF GitHub Bot commented on KYLIN-4813: --- lgtm-com[bot] commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-734250755 This pull request **fixes 1 alert** when merging 68df8945f3a064aab424c129dd04c4fdceb62788 into c7e08d60e4a96d1e343bcf43f757624c16cdfff2 - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-cc06ce3d2bf44876c13c268c4bb9b9f584df33dc) **fixed alerts:** * 1 for Spurious Javadoc @param tags This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239196#comment-17239196 ] ASF GitHub Bot commented on KYLIN-4813: --- hit-lacus commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-734232550 What do you think of this patch, @zzcclp ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239113#comment-17239113 ] ASF GitHub Bot commented on KYLIN-4813: --- lgtm-com[bot] commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-734150444 This pull request **fixes 1 alert** when merging 1b5490d236ee811971f2520852ef92fa84b06046 into 8da4b26c2cb3e9d46100ca64fe5bb4fcdfaf56ac - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-f1b21b6b7fb87a494bf4c0840bd39831df21a499) **fixed alerts:** * 1 for Spurious Javadoc @param tags This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238463#comment-17238463 ] ASF GitHub Bot commented on KYLIN-4813: --- lgtm-com[bot] commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-733421209 This pull request **fixes 1 alert** when merging d87f319d9257268f4f0bbe10fb141f4c3c02d72e into 8da4b26c2cb3e9d46100ca64fe5bb4fcdfaf56ac - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-e78acbefcbb532dbc08827a89430a41798508fc8) **fixed alerts:** * 1 for Spurious Javadoc @param tags This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238090#comment-17238090 ] ASF GitHub Bot commented on KYLIN-4813: --- lgtm-com[bot] commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-732947289 This pull request **fixes 1 alert** when merging 71b91b4bb99cc170d9728e40d5eab24b23950eaa into 8da4b26c2cb3e9d46100ca64fe5bb4fcdfaf56ac - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-62e5293ad9653a757559bc4d47d7de8b030e91c0) **fixed alerts:** * 1 for Spurious Javadoc @param tags This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237230#comment-17237230 ] ASF GitHub Bot commented on KYLIN-4813: --- lgtm-com[bot] commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-732020059 This pull request **fixes 1 alert** when merging 1cc1720e1df4ba4f1ec103ee35c4ea00fde5ce1d into 0b4df4cf67b27c6f9a7a9e5c9006ddf528acc09b - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-67c3390398865d489f47fc417df774fc5d82c048) **fixed alerts:** * 1 for Spurious Javadoc @param tags This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237123#comment-17237123
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus commented on a change in pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#discussion_r528476041
##
File path:
kylin-spark-project/kylin-spark-common/src/main/java/org/apache/kylin/engine/spark/common/logging/SparkExecutorHdfsAppender.java
##
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.engine.spark.common.logging;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.log4j.helpers.LogLog;
+import org.apache.log4j.spi.LoggingEvent;
+import org.apache.spark.SparkEnv;
+import org.apache.spark.deploy.yarn.YarnSparkHadoopUtil;
+
+import java.io.File;
+import java.io.IOException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.List;
+import java.util.UUID;
+
+public class SparkExecutorHdfsAppender extends AbstractHdfsLogAppender {
+
+private static final long A_DAY_MILLIS = 24 * 60 * 60 * 1000L;
+private static final long A_HOUR_MILLIS = 60 * 60 * 1000L;
+private SimpleDateFormat dateFormat = new SimpleDateFormat("-MM-dd");
Review comment:
Modify to `new SimpleDateFormat("-MM-dd", Locale.getDefault())` .
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236958#comment-17236958 ] ASF GitHub Bot commented on KYLIN-4813: --- hit-lacus commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-731760075 @zhangayqian Please review my patch at https://github.com/zhangayqian/kylin/pull/1 , merge it if it is fine to you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236077#comment-17236077
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus edited a comment on pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#issuecomment-731095920
## Cause Analysis
### executor-log4j.properties
```properties
kylin.engine.spark-conf.spark.executor.extraJavaOptions=-Dfile.encoding=UTF-8
-Dhdp.version=current -Dlog4j.configuration=spark-executor-log4j.properties
-Dlog4j.debug -Dkylin.hdfs.working.dir=${kylin.env.hdfs-working-dir}
-Dkylin.metadata.identifier=${kylin.metadata.url.identifier}
-Dkylin.spark.category=job -Dkylin.spark.project=${job.project}
-Dkylin.spark.identifier=${job.id} -Dkylin.spark.jobName=${job.stepId}
-Duser.timezone=${user.timezone}
```
### spark-submit command in NSparkExecutable
```sh
2020-11-20 18:22:10,342 INFO [Scheduler 1960610874 Job
79331eef-a64f-411a-a8b5-f8696d301438-104] job.NSparkExecutable:41 : cmd:
2020-11-20 18:22:10,342 INFO [Scheduler 1960610874 Job
79331eef-a64f-411a-a8b5-f8696d301438-104] job.NSparkExecutable:41 : export
HADOOP_CONF_DIR=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/hadoop_conf
&&
/root/open-source/deploy/kylin-instances/spark-2.4.6-bin-hadoop2.7/bin/spark-submit
--class org.apache.kylin.engine.spark.application.SparkEntry
--conf 'spark.executor.instances=1'
--conf 'spark.yarn.queue=default'
--conf 'spark.history.fs.logDirectory=hdfs:///kylin/spark-history'
--conf 'spark.master=yarn'
--conf 'spark.executor.extraJavaOptions=-Dfile.encoding=UTF-8
-Dhdp.version=current -Dlog4j.configuration=spark-executor-log4j.properties
-Dlog4j.debug -Dkylin.hdfs.working.dir=${kylin.env.hdfs-working-dir}
-Dkylin.metadata.identifier=${kylin.metadata.url.identifier}
-Dkylin.spark.category=job -Dkylin.spark.project=${job.project}
-Dkylin.spark.identifier=${job.id} -Dkylin.spark.jobName=${job.stepId}
-Duser.timezone=${user.timezone}'
--conf 'spark.hadoop.yarn.timeline-service.enabled=false'
--conf 'spark.driver.cores=1' --conf 'spark.executor.memory=4G'
--conf 'spark.eventLog.enabled=true'
--conf 'spark.eventLog.dir=hdfs:///kylin/spark-history'
--conf 'spark.executor.cores=1'
--conf 'spark.executor.memoryOverhead=1024M'
--conf 'spark.driver.memory=1G'
--conf 'spark.shuffle.service.enabled=true'
--conf
'spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/conf/spark-driver-log4j.properties
-Dkylin.kerberos.enabled=false
-Dkylin.hdfs.working.dir=hdfs://cdh-master:8020/regression_testing/400alpha/xxyu/
-Dspark.driver.log4j.appender.hdfs.File=hdfs://cdh-master:8020/regression_testing/400alpha/xxyu/learn_kylin/spark_logs/driver/79331eef-a64f-411a-a8b5-f8696d301438-01/execute_output.json.1605867729760.log
-Dspark.driver.rest.server.ip=10.1.3.90 -Dspark.driver.rest.server.port=7070
-Dspark.driver.param.taskId=79331eef-a64f-411a-a8b5-f8696d301438-01
-Dspark.driver.local.logDir=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/logs/spark'
--conf 'spark.executor.extraClassPath=kylin-parquet-job-4.0.0-SNAPSHOT.jar'
--conf
'spark.driver.extraClassPath=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar'
--files
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/conf/spark-executor-log4j.properties
--name job_step_79331eef-a64f-411a-a8b5-f8696d301438-01
--jars
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar
-className org.apache.kylin.engine.spark.job.CubeBuildJob
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/tomcat/temp/segmentIds2791542431972229325
```
### Output of executor log
Following variable is empty :
- metadataIdentifier
- project
- jobName
- hdfsWorkingDir
```sh
[root@cdh-master kylin]# yarn logs -applicationId
application_1589169585068_30012
20/11/20 18:29:25 INFO client.RMProxy: Connecting to ResourceManager at
cdh-master/10.1.3.90:8032
Container: container_1589169585068_30012_01_02 on cdh-worker-1_8041
=
LogType:stderr
Log Upload Time:Fri Nov 20 18:23:56 +0800 2020
LogLength:3373769
Log Contents:
log4j:WARN No such property [rollingPeriod] in
org.apache.kylin.engine.spark.common.logging.SparkExecutorHdfsAppender.
log4j:WARN SparkExecutorHdfsLogAppender starting ...
log4j:WARN hdfsWorkingDir ->
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236074#comment-17236074
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus edited a comment on pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#issuecomment-731095920
## Cause Analysis
### spark-submit command in NSparkExecutable
```sh
2020-11-20 18:22:10,342 INFO [Scheduler 1960610874 Job
79331eef-a64f-411a-a8b5-f8696d301438-104] job.NSparkExecutable:41 : cmd:
2020-11-20 18:22:10,342 INFO [Scheduler 1960610874 Job
79331eef-a64f-411a-a8b5-f8696d301438-104] job.NSparkExecutable:41 : export
HADOOP_CONF_DIR=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/hadoop_conf
&&
/root/open-source/deploy/kylin-instances/spark-2.4.6-bin-hadoop2.7/bin/spark-submit
--class org.apache.kylin.engine.spark.application.SparkEntry
--conf 'spark.executor.instances=1'
--conf 'spark.yarn.queue=default'
--conf 'spark.history.fs.logDirectory=hdfs:///kylin/spark-history'
--conf 'spark.master=yarn'
--conf 'spark.executor.extraJavaOptions=-Dfile.encoding=UTF-8
-Dhdp.version=current -Dlog4j.configuration=spark-executor-log4j.properties
-Dlog4j.debug -Dkylin.hdfs.working.dir=${kylin.env.hdfs-working-dir}
-Dkylin.metadata.identifier=${kylin.metadata.url.identifier}
-Dkylin.spark.category=job -Dkylin.spark.project=${job.project}
-Dkylin.spark.identifier=${job.id} -Dkylin.spark.jobName=${job.stepId}
-Duser.timezone=${user.timezone}'
--conf 'spark.hadoop.yarn.timeline-service.enabled=false'
--conf 'spark.driver.cores=1' --conf 'spark.executor.memory=4G'
--conf 'spark.eventLog.enabled=true'
--conf 'spark.eventLog.dir=hdfs:///kylin/spark-history'
--conf 'spark.executor.cores=1'
--conf 'spark.executor.memoryOverhead=1024M'
--conf 'spark.driver.memory=1G'
--conf 'spark.shuffle.service.enabled=true'
--conf
'spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/conf/spark-driver-log4j.properties
-Dkylin.kerberos.enabled=false
-Dkylin.hdfs.working.dir=hdfs://cdh-master:8020/regression_testing/400alpha/xxyu/
-Dspark.driver.log4j.appender.hdfs.File=hdfs://cdh-master:8020/regression_testing/400alpha/xxyu/learn_kylin/spark_logs/driver/79331eef-a64f-411a-a8b5-f8696d301438-01/execute_output.json.1605867729760.log
-Dspark.driver.rest.server.ip=10.1.3.90 -Dspark.driver.rest.server.port=7070
-Dspark.driver.param.taskId=79331eef-a64f-411a-a8b5-f8696d301438-01
-Dspark.driver.local.logDir=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/logs/spark'
--conf 'spark.executor.extraClassPath=kylin-parquet-job-4.0.0-SNAPSHOT.jar'
--conf
'spark.driver.extraClassPath=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar'
--files
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/conf/spark-executor-log4j.properties
--name job_step_79331eef-a64f-411a-a8b5-f8696d301438-01
--jars
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar
-className org.apache.kylin.engine.spark.job.CubeBuildJob
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/tomcat/temp/segmentIds2791542431972229325
```
### Output of executor log
Following variable is empty :
- metadataIdentifier
- project
- jobName
- hdfsWorkingDir
```sh
[root@cdh-master kylin]# yarn logs -applicationId
application_1589169585068_30012
20/11/20 18:29:25 INFO client.RMProxy: Connecting to ResourceManager at
cdh-master/10.1.3.90:8032
Container: container_1589169585068_30012_01_02 on cdh-worker-1_8041
=
LogType:stderr
Log Upload Time:Fri Nov 20 18:23:56 +0800 2020
LogLength:3373769
Log Contents:
log4j:WARN No such property [rollingPeriod] in
org.apache.kylin.engine.spark.common.logging.SparkExecutorHdfsAppender.
log4j:WARN SparkExecutorHdfsLogAppender starting ...
log4j:WARN hdfsWorkingDir ->
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/yarn/nm/usercache/root/filecache/835102/__spark_libs__3815910076379456523.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN metadataIdentifier ->
log4j:WARN category -> job
log4j:WARN identifier -> application_1589169585068_30012
log4j:WARN
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236073#comment-17236073
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus edited a comment on pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#issuecomment-731095920
## Cause Analysis
spark-submit command in NSparkExecutable
```sh
2020-11-20 18:22:10,342 INFO [Scheduler 1960610874 Job
79331eef-a64f-411a-a8b5-f8696d301438-104] job.NSparkExecutable:41 : cmd:
2020-11-20 18:22:10,342 INFO [Scheduler 1960610874 Job
79331eef-a64f-411a-a8b5-f8696d301438-104] job.NSparkExecutable:41 : export
HADOOP_CONF_DIR=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/hadoop_conf
&&
/root/open-source/deploy/kylin-instances/spark-2.4.6-bin-hadoop2.7/bin/spark-submit
--class org.apache.kylin.engine.spark.application.SparkEntry
--conf 'spark.executor.instances=1'
--conf 'spark.yarn.queue=default'
--conf 'spark.history.fs.logDirectory=hdfs:///kylin/spark-history'
--conf 'spark.master=yarn'
--conf 'spark.executor.extraJavaOptions=-Dfile.encoding=UTF-8
-Dhdp.version=current -Dlog4j.configuration=spark-executor-log4j.properties
-Dlog4j.debug -Dkylin.hdfs.working.dir=${kylin.env.hdfs-working-dir}
-Dkylin.metadata.identifier=${kylin.metadata.url.identifier}
-Dkylin.spark.category=job -Dkylin.spark.project=${job.project}
-Dkylin.spark.identifier=${job.id} -Dkylin.spark.jobName=${job.stepId}
-Duser.timezone=${user.timezone}'
--conf 'spark.hadoop.yarn.timeline-service.enabled=false'
--conf 'spark.driver.cores=1' --conf 'spark.executor.memory=4G'
--conf 'spark.eventLog.enabled=true'
--conf 'spark.eventLog.dir=hdfs:///kylin/spark-history'
--conf 'spark.executor.cores=1'
--conf 'spark.executor.memoryOverhead=1024M'
--conf 'spark.driver.memory=1G'
--conf 'spark.shuffle.service.enabled=true'
--conf
'spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/conf/spark-driver-log4j.properties
-Dkylin.kerberos.enabled=false
-Dkylin.hdfs.working.dir=hdfs://cdh-master:8020/regression_testing/400alpha/xxyu/
-Dspark.driver.log4j.appender.hdfs.File=hdfs://cdh-master:8020/regression_testing/400alpha/xxyu/learn_kylin/spark_logs/driver/79331eef-a64f-411a-a8b5-f8696d301438-01/execute_output.json.1605867729760.log
-Dspark.driver.rest.server.ip=10.1.3.90 -Dspark.driver.rest.server.port=7070
-Dspark.driver.param.taskId=79331eef-a64f-411a-a8b5-f8696d301438-01
-Dspark.driver.local.logDir=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/logs/spark'
--conf 'spark.executor.extraClassPath=kylin-parquet-job-4.0.0-SNAPSHOT.jar'
--conf
'spark.driver.extraClassPath=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar'
--files
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/conf/spark-executor-log4j.properties
--name job_step_79331eef-a64f-411a-a8b5-f8696d301438-01
--jars
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar
-className org.apache.kylin.engine.spark.job.CubeBuildJob
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/tomcat/temp/segmentIds2791542431972229325
```
Output of executor log
Following var is empty :
- metadataIdentifier
- project
- jobName
- hdfsWorkingDir
```sh
[root@cdh-master kylin]# yarn logs -applicationId
application_1589169585068_30012
20/11/20 18:29:25 INFO client.RMProxy: Connecting to ResourceManager at
cdh-master/10.1.3.90:8032
Container: container_1589169585068_30012_01_02 on cdh-worker-1_8041
=
LogType:stderr
Log Upload Time:Fri Nov 20 18:23:56 +0800 2020
LogLength:3373769
Log Contents:
log4j:WARN No such property [rollingPeriod] in
org.apache.kylin.engine.spark.common.logging.SparkExecutorHdfsAppender.
log4j:WARN SparkExecutorHdfsLogAppender starting ...
log4j:WARN hdfsWorkingDir ->
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/yarn/nm/usercache/root/filecache/835102/__spark_libs__3815910076379456523.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN metadataIdentifier ->
log4j:WARN category -> job
log4j:WARN identifier -> application_1589169585068_30012
log4j:WARN
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236072#comment-17236072
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus edited a comment on pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#issuecomment-731095920
## Cause Analysis
```sh
2020-11-20 18:22:10,342 INFO [Scheduler 1960610874 Job
79331eef-a64f-411a-a8b5-f8696d301438-104] job.NSparkExecutable:41 : cmd:
2020-11-20 18:22:10,342 INFO [Scheduler 1960610874 Job
79331eef-a64f-411a-a8b5-f8696d301438-104] job.NSparkExecutable:41 : export
HADOOP_CONF_DIR=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/hadoop_conf
&&
/root/open-source/deploy/kylin-instances/spark-2.4.6-bin-hadoop2.7/bin/spark-submit
--class org.apache.kylin.engine.spark.application.SparkEntry
--conf 'spark.executor.instances=1'
--conf 'spark.yarn.queue=default'
--conf 'spark.history.fs.logDirectory=hdfs:///kylin/spark-history'
--conf 'spark.master=yarn'
--conf 'spark.executor.extraJavaOptions=-Dfile.encoding=UTF-8
-Dhdp.version=current -Dlog4j.configuration=spark-executor-log4j.properties
-Dlog4j.debug -Dkylin.hdfs.working.dir=${kylin.env.hdfs-working-dir}
-Dkylin.metadata.identifier=${kylin.metadata.url.identifier}
-Dkylin.spark.category=job -Dkylin.spark.project=${job.project}
-Dkylin.spark.identifier=${job.id} -Dkylin.spark.jobName=${job.stepId}
-Duser.timezone=${user.timezone}'
--conf 'spark.hadoop.yarn.timeline-service.enabled=false'
--conf 'spark.driver.cores=1' --conf 'spark.executor.memory=4G'
--conf 'spark.eventLog.enabled=true'
--conf 'spark.eventLog.dir=hdfs:///kylin/spark-history'
--conf 'spark.executor.cores=1'
--conf 'spark.executor.memoryOverhead=1024M'
--conf 'spark.driver.memory=1G'
--conf 'spark.shuffle.service.enabled=true'
--conf
'spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/conf/spark-driver-log4j.properties
-Dkylin.kerberos.enabled=false
-Dkylin.hdfs.working.dir=hdfs://cdh-master:8020/regression_testing/400alpha/xxyu/
-Dspark.driver.log4j.appender.hdfs.File=hdfs://cdh-master:8020/regression_testing/400alpha/xxyu/learn_kylin/spark_logs/driver/79331eef-a64f-411a-a8b5-f8696d301438-01/execute_output.json.1605867729760.log
-Dspark.driver.rest.server.ip=10.1.3.90 -Dspark.driver.rest.server.port=7070
-Dspark.driver.param.taskId=79331eef-a64f-411a-a8b5-f8696d301438-01
-Dspark.driver.local.logDir=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/logs/spark'
--conf 'spark.executor.extraClassPath=kylin-parquet-job-4.0.0-SNAPSHOT.jar'
--conf
'spark.driver.extraClassPath=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar'
--files
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/conf/spark-executor-log4j.properties
--name job_step_79331eef-a64f-411a-a8b5-f8696d301438-01
--jars
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar
-className org.apache.kylin.engine.spark.job.CubeBuildJob
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/tomcat/temp/segmentIds2791542431972229325
```
```sh
[root@cdh-master kylin]# yarn logs -applicationId
application_1589169585068_30012
20/11/20 18:29:25 INFO client.RMProxy: Connecting to ResourceManager at
cdh-master/10.1.3.90:8032
Container: container_1589169585068_30012_01_02 on cdh-worker-1_8041
=
LogType:stderr
Log Upload Time:Fri Nov 20 18:23:56 +0800 2020
LogLength:3373769
Log Contents:
log4j:WARN No such property [rollingPeriod] in
org.apache.kylin.engine.spark.common.logging.SparkExecutorHdfsAppender.
log4j:WARN SparkExecutorHdfsLogAppender starting ...
log4j:WARN hdfsWorkingDir ->
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/yarn/nm/usercache/root/filecache/835102/__spark_libs__3815910076379456523.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN metadataIdentifier ->
log4j:WARN category -> job
log4j:WARN identifier -> application_1589169585068_30012
log4j:WARN project ->
log4j:WARN jobName ->
log4j:WARN SparkExecutorHdfsLogAppender started ...
```
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236070#comment-17236070
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus edited a comment on pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#issuecomment-731095920
## Cause Analysis
```sh
2020-11-20 18:22:10,342 INFO [Scheduler 1960610874 Job
79331eef-a64f-411a-a8b5-f8696d301438-104] job.NSparkExecutable:41 : cmd:
2020-11-20 18:22:10,342 INFO [Scheduler 1960610874 Job
79331eef-a64f-411a-a8b5-f8696d301438-104] job.NSparkExecutable:41 : export
HADOOP_CONF_DIR=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/hadoop_conf
&&
/root/open-source/deploy/kylin-instances/spark-2.4.6-bin-hadoop2.7/bin/spark-submit
--class org.apache.kylin.engine.spark.application.SparkEntry
--conf 'spark.executor.instances=1'
--conf 'spark.yarn.queue=default'
--conf 'spark.history.fs.logDirectory=hdfs:///kylin/spark-history'
--conf 'spark.master=yarn'
--conf 'spark.executor.extraJavaOptions=-Dfile.encoding=UTF-8
-Dhdp.version=current -Dlog4j.configuration=spark-executor-log4j.properties
-Dlog4j.debug -Dkylin.hdfs.working.dir=${kylin.env.hdfs-working-dir}
-Dkylin.metadata.identifier=${kylin.metadata.url.identifier}
-Dkylin.spark.category=job -Dkylin.spark.project=${job.project}
-Dkylin.spark.identifier=${job.id} -Dkylin.spark.jobName=${job.stepId}
-Duser.timezone=${user.timezone}'
--conf 'spark.hadoop.yarn.timeline-service.enabled=false'
--conf 'spark.driver.cores=1' --conf 'spark.executor.memory=4G'
--conf 'spark.eventLog.enabled=true'
--conf 'spark.eventLog.dir=hdfs:///kylin/spark-history'
--conf 'spark.executor.cores=1'
--conf 'spark.executor.memoryOverhead=1024M'
--conf 'spark.driver.memory=1G'
--conf 'spark.shuffle.service.enabled=true'
--conf
'spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/conf/spark-driver-log4j.properties
-Dkylin.kerberos.enabled=false
-Dkylin.hdfs.working.dir=hdfs://cdh-master:8020/regression_testing/400alpha/xxyu/
-Dspark.driver.log4j.appender.hdfs.File=hdfs://cdh-master:8020/regression_testing/400alpha/xxyu/learn_kylin/spark_logs/driver/79331eef-a64f-411a-a8b5-f8696d301438-01/execute_output.json.1605867729760.log
-Dspark.driver.rest.server.ip=10.1.3.90 -Dspark.driver.rest.server.port=7070
-Dspark.driver.param.taskId=79331eef-a64f-411a-a8b5-f8696d301438-01
-Dspark.driver.local.logDir=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/logs/spark'
--conf 'spark.executor.extraClassPath=kylin-parquet-job-4.0.0-SNAPSHOT.jar'
--conf
'spark.driver.extraClassPath=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar'
--files
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/conf/spark-executor-log4j.properties
--name job_step_79331eef-a64f-411a-a8b5-f8696d301438-01
--jars
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar
-className org.apache.kylin.engine.spark.job.CubeBuildJob
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/tomcat/temp/segmentIds2791542431972229325
```
```sh
[root@cdh-master kylin]# yarn logs -applicationId
application_1589169585068_30012
20/11/20 18:29:25 INFO client.RMProxy: Connecting to ResourceManager at
cdh-master/10.1.3.90:8032
Container: container_1589169585068_30012_01_02 on cdh-worker-1_8041
=
LogType:stderr
Log Upload Time:Fri Nov 20 18:23:56 +0800 2020
LogLength:3373769
Log Contents:
log4j:WARN No such property [rollingPeriod] in
org.apache.kylin.engine.spark.common.logging.SparkExecutorHdfsAppender.
log4j:WARN SparkExecutorHdfsLogAppender starting ...
log4j:WARN hdfsWorkingDir ->
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/yarn/nm/usercache/root/filecache/835102/__spark_libs__3815910076379456523.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN metadataIdentifier ->
log4j:WARN category -> job
log4j:WARN identifier -> application_1589169585068_30012
log4j:WARN project ->
log4j:WARN jobName ->
log4j:WARN SparkExecutorHdfsLogAppender started ...
```
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236067#comment-17236067
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus edited a comment on pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#issuecomment-731095920
## Cause Analysis
```sh
2020-11-20 18:22:10,342 INFO [Scheduler 1960610874 Job
79331eef-a64f-411a-a8b5-f8696d301438-104] job.NSparkExecutable:41 : cmd:
2020-11-20 18:22:10,342 INFO [Scheduler 1960610874 Job
79331eef-a64f-411a-a8b5-f8696d301438-104] job.NSparkExecutable:41 : export
HADOOP_CONF_DIR=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/hadoop_conf
&&
/root/open-source/deploy/kylin-instances/spark-2.4.6-bin-hadoop2.7/bin/spark-submit
--class org.apache.kylin.engine.spark.application.SparkEntry --conf
'spark.executor.instances=1' --conf 'spark.yarn.queue=default' --conf
'spark.history.fs.logDirectory=hdfs:///kylin/spark-history' --conf
'spark.master=yarn' --conf
'spark.executor.extraJavaOptions=-Dfile.encoding=UTF-8 -Dhdp.version=current
-Dlog4j.configuration=spark-executor-log4j.properties -Dlog4j.debug
-Dkylin.hdfs.working.dir=${kylin.env.hdfs-working-dir}
-Dkylin.metadata.identifier=${kylin.metadata.url.identifier}
-Dkylin.spark.category=job -Dkylin.spark.project=${job.project}
-Dkylin.spark.identifier=${job.id} -Dkylin.spark.jobName=${job.stepId}
-Duser.timezone=${user.timezone}' --conf
'spark.hadoop.yarn.timeline-service.enabled=false' --conf
'spark.driver.cores=1' --conf 'spark.executor.memory=4G' --conf
'spark.eventLog.enabled=true' --conf
'spark.eventLog.dir=hdfs:///kylin/spark-history' --conf
'spark.executor.cores=1' --conf 'spark.executor.memoryOverhead=1024M' --conf
'spark.driver.memory=1G' --conf 'spark.shuffle.service.enabled=true' --conf
'spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/conf/spark-driver-log4j.properties
-Dkylin.kerberos.enabled=false
-Dkylin.hdfs.working.dir=hdfs://cdh-master:8020/regression_testing/400alpha/xxyu/
-Dspark.driver.log4j.appender.hdfs.File=hdfs://cdh-master:8020/regression_testing/400alpha/xxyu/learn_kylin/spark_logs/driver/79331eef-a64f-411a-a8b5-f8696d301438-01/execute_output.json.1605867729760.log
-Dspark.driver.rest.server.ip=10.1.3.90 -Dspark.driver.rest.server.port=7070
-Dspark.driver.param.taskId=79331eef-a64f-411a-a8b5-f8696d301438-01
-Dspark.driver.local.logDir=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/logs/spark'
--conf 'spark.executor.extraClassPath=kylin-parquet-job-4.0.0-SNAPSHOT.jar'
--conf
'spark.driver.extraClassPath=/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar'
--files
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/conf/spark-executor-log4j.properties
--name job_step_79331eef-a64f-411a-a8b5-f8696d301438-01 --jars
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/lib/kylin-parquet-job-4.0.0-SNAPSHOT.jar
-className org.apache.kylin.engine.spark.job.CubeBuildJob
/root/lib/kylin-dist/apache-kylin-4.0.0-SNAPSHOT-bin/tomcat/temp/segmentIds2791542431972229325
```
```sh
[root@cdh-master kylin]# yarn logs -applicationId
application_1589169585068_30012
20/11/20 18:29:25 INFO client.RMProxy: Connecting to ResourceManager at
cdh-master/10.1.3.90:8032
Container: container_1589169585068_30012_01_02 on cdh-worker-1_8041
=
LogType:stderr
Log Upload Time:Fri Nov 20 18:23:56 +0800 2020
LogLength:3373769
Log Contents:
log4j:WARN No such property [rollingPeriod] in
org.apache.kylin.engine.spark.common.logging.SparkExecutorHdfsAppender.
log4j:WARN SparkExecutorHdfsLogAppender starting ...
log4j:WARN hdfsWorkingDir ->
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/yarn/nm/usercache/root/filecache/835102/__spark_libs__3815910076379456523.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN metadataIdentifier ->
log4j:WARN category -> job
log4j:WARN identifier -> application_1589169585068_30012
log4j:WARN project ->
log4j:WARN jobName ->
log4j:WARN SparkExecutorHdfsLogAppender started ...
```
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236063#comment-17236063 ] ASF GitHub Bot commented on KYLIN-4813: --- hit-lacus edited a comment on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-731095920 ## Cause Analysis ```sh [root@cdh-master kylin]# yarn logs -applicationId application_1589169585068_30012 20/11/20 18:29:25 INFO client.RMProxy: Connecting to ResourceManager at cdh-master/10.1.3.90:8032 Container: container_1589169585068_30012_01_02 on cdh-worker-1_8041 = LogType:stderr Log Upload Time:Fri Nov 20 18:23:56 +0800 2020 LogLength:3373769 Log Contents: log4j:WARN No such property [rollingPeriod] in org.apache.kylin.engine.spark.common.logging.SparkExecutorHdfsAppender. log4j:WARN SparkExecutorHdfsLogAppender starting ... log4j:WARN hdfsWorkingDir -> SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/yarn/nm/usercache/root/filecache/835102/__spark_libs__3815910076379456523.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] log4j:WARN metadataIdentifier -> log4j:WARN category -> job log4j:WARN identifier -> application_1589169585068_30012 log4j:WARN project -> log4j:WARN jobName -> log4j:WARN SparkExecutorHdfsLogAppender started ... ```   This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236060#comment-17236060 ] ASF GitHub Bot commented on KYLIN-4813: --- hit-lacus edited a comment on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-731095920 ## Cause Analysis ```sh [root@cdh-master kylin]# yarn logs -applicationId application_1589169585068_30012 20/11/20 18:29:25 INFO client.RMProxy: Connecting to ResourceManager at cdh-master/10.1.3.90:8032 Container: container_1589169585068_30012_01_02 on cdh-worker-1_8041 = LogType:stderr Log Upload Time:Fri Nov 20 18:23:56 +0800 2020 LogLength:3373769 Log Contents: log4j:WARN No such property [rollingPeriod] in org.apache.kylin.engine.spark.common.logging.SparkExecutorHdfsAppender. log4j:WARN SparkExecutorHdfsLogAppender starting ... log4j:WARN hdfsWorkingDir -> SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/yarn/nm/usercache/root/filecache/835102/__spark_libs__3815910076379456523.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] log4j:WARN metadataIdentifier -> log4j:WARN category -> job log4j:WARN identifier -> application_1589169585068_30012 log4j:WARN project -> log4j:WARN jobName -> log4j:WARN SparkExecutorHdfsLogAppender started ... ```  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236057#comment-17236057 ] ASF GitHub Bot commented on KYLIN-4813: --- hit-lacus commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-731095920 ## Cause Analysis  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236004#comment-17236004 ] ASF GitHub Bot commented on KYLIN-4813: --- hit-lacus edited a comment on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-731015025 ### Reference - http://spark.apache.org/docs/latest/running-on-yarn.html https://user-images.githubusercontent.com/14030549/99775913-0452aa00-2b4b-11eb-8362-9bc812b9d41b.png;> - http://spark.apache.org/docs/latest/configuration.html https://user-images.githubusercontent.com/14030549/99780167-0ddf1080-2b51-11eb-9872-d3fc5adbc990.png;> This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235991#comment-17235991
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus commented on a change in pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#discussion_r527520352
##
File path:
core-job/src/main/java/org/apache/kylin/job/execution/AbstractExecutable.java
##
@@ -587,7 +596,7 @@ private static boolean isRetryableException(String
exceptionName) {
public final String getProject() {
if (project == null) {
-throw new IllegalStateException("project is not set for abstract
executable " + getId());
+logger.error("project is not set for abstract executable " +
getId());
Review comment:
I can not understand here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235981#comment-17235981 ] ASF GitHub Bot commented on KYLIN-4813: --- hit-lacus commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-731015025 https://user-images.githubusercontent.com/14030549/99775913-0452aa00-2b4b-11eb-8362-9bc812b9d41b.png;> This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235910#comment-17235910
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus commented on a change in pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#discussion_r527427654
##
File path: build/conf/spark-executor-log4j.properties
##
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# It's called spark-executor-log4j.properties so that it won't distract users
from the other more important log4j config file: kylin-server-log4j.properties
+# enable this by -Dlog4j.configuration=spark-executor-log4j.properties
+log4j.rootLogger=INFO,stderr,hdfs
+
+log4j.appender.stderr=org.apache.log4j.ConsoleAppender
+log4j.appender.stderr.layout=org.apache.kylin.common.logging.SensitivePatternLayout
+log4j.appender.stderr.target=System.err
+#Don't add line number (%L) as it's too costly!
+log4j.appender.stderr.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2} :
%m%n
+
+
+log4j.appender.hdfs=org.apache.kylin.engine.spark.common.logging.SparkExecutorHdfsLogAppender
+
+log4j.appender.hdfs.hdfsWorkingDir=${kylin.hdfs.working.dir}
+log4j.appender.hdfs.metadataIdentifier=${kylin.metadata.identifier}
+log4j.appender.hdfs.category=${kylin.spark.category}
+log4j.appender.hdfs.identifier=${kylin.spark.identifier}
+log4j.appender.hdfs.jobName=${kylin.spark.jobName}
+log4j.appender.hdfs.project=${kylin.spark.project}
+
+log4j.appender.hdfs.rollingPeriod=5
+log4j.appender.hdfs.logQueueCapacity=5000
+#flushPeriod count as millis
+log4j.appender.hdfs.flushInterval=5000
+
+log4j.appender.hdfs.layout=org.apache.kylin.engine.spark.common.logging.SensitivePatternLayout
Review comment:
Should be `org.apache.kylin.common.logging.SensitivePatternLayout` .
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235909#comment-17235909
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus commented on a change in pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#discussion_r527427023
##
File path: build/conf/spark-executor-log4j.properties
##
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# It's called spark-executor-log4j.properties so that it won't distract users
from the other more important log4j config file: kylin-server-log4j.properties
+# enable this by -Dlog4j.configuration=spark-executor-log4j.properties
+log4j.rootLogger=INFO,stderr,hdfs
+
+log4j.appender.stderr=org.apache.log4j.ConsoleAppender
+log4j.appender.stderr.layout=org.apache.kylin.common.logging.SensitivePatternLayout
+log4j.appender.stderr.target=System.err
+#Don't add line number (%L) as it's too costly!
+log4j.appender.stderr.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2} :
%m%n
+
+
+log4j.appender.hdfs=org.apache.kylin.engine.spark.common.logging.SparkExecutorHdfsLogAppender
Review comment:
`SparkExecutorHdfsLogAppender` or `SparkExecutorHdfsAppender` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235807#comment-17235807
]
ASF GitHub Bot commented on KYLIN-4813:
---
zzcclp commented on a change in pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#discussion_r527322199
##
File path: core-common/src/main/resources/kylin-defaults.properties
##
@@ -283,6 +287,7 @@
kylin.query.spark-conf.spark.serializer=org.apache.spark.serializer.JavaSerializ
#kylin.query.spark-conf.spark.sql.shuffle.partitions=40
#kylin.query.spark-conf.spark.yarn.jars=hdfs://localhost:9000/spark2_jars/*
+kylin.storage.columnar.spark-conf.spark.executor.extraJavaOptions=-Dhdp.version=current
-Dlog4j.configuration=spark-executor-log4j.properties -Dlog4j.debug
-Dkylin.hdfs.working.dir=${kylin.env.hdfs-working-dir}
-Dkylin.metadata.identifier=${kylin.metadata.url.identifier}
-Dkylin.spark.category=sparder -Dkylin.spark.project=${job.project}
-XX:MaxDirectMemorySize=896M
Review comment:
Please replace 'kylin.storage.columnar' with 'kylin.query'.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235251#comment-17235251
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus commented on a change in pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#discussion_r526665177
##
File path:
core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
##
@@ -258,6 +258,9 @@ final public void setProperty(String key, String value) {
final protected void reloadKylinConfig(Properties properties) {
this.properties = BCC.check(properties);
+setProperty("kylin.metadata.url.identifier", getMetadataUrlPrefix());
+setProperty("kylin.log.spark-driver-properties-file",
getLogSparkDriverPropertiesFile());
Review comment:
What does `kylin.log.spark-driver-properties-file` means?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235193#comment-17235193
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus commented on a change in pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#discussion_r526613762
##
File path: core-common/src/main/resources/kylin-defaults.properties
##
@@ -283,6 +287,7 @@
kylin.query.spark-conf.spark.serializer=org.apache.spark.serializer.JavaSerializ
#kylin.query.spark-conf.spark.sql.shuffle.partitions=40
#kylin.query.spark-conf.spark.yarn.jars=hdfs://localhost:9000/spark2_jars/*
+kylin.storage.columnar.spark-conf.spark.executor.extraJavaOptions=-Dhdp.version=current
-Dlog4j.configuration=spark-executor-log4j.properties -Dlog4j.debug
-Dkylin.hdfs.working.dir=${kylin.env.hdfs-working-dir}
-Dkylin.metadata.identifier=${kylin.metadata.url.identifier}
-Dkylin.spark.category=sparder -Dkylin.spark.project=${job.project}
-XX:MaxDirectMemorySize=896M
Review comment:
`kylin.engine.spark-conf` or `kylin.engine.spark-conf` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235150#comment-17235150
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus commented on a change in pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#discussion_r526573750
##
File path:
server-base/src/main/java/org/apache/kylin/rest/controller/JobController.java
##
@@ -160,17 +168,41 @@ public JobInstance get(@PathVariable String jobId) {
* @return
* @throws IOException
*/
+
@RequestMapping(value = "/{jobId}/steps/{stepId}/output", method = {
RequestMethod.GET }, produces = {
"application/json" })
@ResponseBody
public Map getStepOutput(@PathVariable String jobId,
@PathVariable String stepId) {
Map result = new HashMap();
result.put("jobId", jobId);
result.put("stepId", String.valueOf(stepId));
-result.put("cmd_output",
jobService.getExecutableManager().getOutput(stepId).getVerboseMsg());
+result.put("cmd_output", jobService.getJobOutput(jobId, stepId));
return result;
}
+/**
+ * Download a job step output from hdfs
+ * @param jobId
+ * @param stepId
+ * @param project
+ * @param response
+ * @return
+ */
+@RequestMapping(value = "/{job_id:.+}/steps/{step_id:.+}/log", method = {
RequestMethod.GET }, produces = { "application/json" })
+@ResponseBody
+public EnvelopeResponse downloadLogFile(@PathVariable("job_id")
String jobId,
+@PathVariable("step_id")
String stepId, @RequestParam(value = "project") String project,
+HttpServletResponse
response) throws IOException {
+checkRequiredArg("job_id", jobId);
+checkRequiredArg("step_id", stepId);
+checkRequiredArg("project", project);
+String downloadFilename = String.format(Locale.ROOT, "%s_%s.log",
project, stepId);
Review comment:
Please check report from LGTM or search "HTTP Response Splitting" for
background.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235148#comment-17235148
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus commented on a change in pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#discussion_r526573511
##
File path:
server-base/src/main/java/org/apache/kylin/rest/controller/JobController.java
##
@@ -160,17 +168,41 @@ public JobInstance get(@PathVariable String jobId) {
* @return
* @throws IOException
*/
+
@RequestMapping(value = "/{jobId}/steps/{stepId}/output", method = {
RequestMethod.GET }, produces = {
"application/json" })
@ResponseBody
public Map getStepOutput(@PathVariable String jobId,
@PathVariable String stepId) {
Map result = new HashMap();
result.put("jobId", jobId);
result.put("stepId", String.valueOf(stepId));
-result.put("cmd_output",
jobService.getExecutableManager().getOutput(stepId).getVerboseMsg());
+result.put("cmd_output", jobService.getJobOutput(jobId, stepId));
return result;
}
+/**
+ * Download a job step output from hdfs
+ * @param jobId
+ * @param stepId
+ * @param project
+ * @param response
+ * @return
+ */
+@RequestMapping(value = "/{job_id:.+}/steps/{step_id:.+}/log", method = {
RequestMethod.GET }, produces = { "application/json" })
+@ResponseBody
+public EnvelopeResponse downloadLogFile(@PathVariable("job_id")
String jobId,
+@PathVariable("step_id")
String stepId, @RequestParam(value = "project") String project,
+HttpServletResponse
response) throws IOException {
+checkRequiredArg("job_id", jobId);
+checkRequiredArg("step_id", stepId);
+checkRequiredArg("project", project);
+String downloadFilename = String.format(Locale.ROOT, "%s_%s.log",
project, stepId);
Review comment:
Before combine `project` with `stepId`, please check if it is vaild and
remove any illegal character.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235144#comment-17235144
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus commented on a change in pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#discussion_r526569415
##
File path:
server-base/src/main/java/org/apache/kylin/rest/controller/JobController.java
##
@@ -160,17 +168,41 @@ public JobInstance get(@PathVariable String jobId) {
* @return
* @throws IOException
*/
+
@RequestMapping(value = "/{jobId}/steps/{stepId}/output", method = {
RequestMethod.GET }, produces = {
"application/json" })
@ResponseBody
public Map getStepOutput(@PathVariable String jobId,
@PathVariable String stepId) {
Map result = new HashMap();
result.put("jobId", jobId);
result.put("stepId", String.valueOf(stepId));
-result.put("cmd_output",
jobService.getExecutableManager().getOutput(stepId).getVerboseMsg());
+result.put("cmd_output", jobService.getJobOutput(jobId, stepId));
return result;
}
+/**
+ * Download a job step output from hdfs
+ * @param jobId
+ * @param stepId
+ * @param project
+ * @param response
+ * @return
+ */
+@RequestMapping(value = "/{job_id:.+}/steps/{step_id:.+}/log", method = {
RequestMethod.GET }, produces = { "application/json" })
+@ResponseBody
+public EnvelopeResponse downloadLogFile(@PathVariable("job_id")
String jobId,
+@PathVariable("step_id")
String stepId, @RequestParam(value = "project") String project,
+HttpServletResponse
response) throws IOException {
+checkRequiredArg("job_id", jobId);
+checkRequiredArg("step_id", stepId);
+checkRequiredArg("project", project);
+String downloadFilename = String.format(Locale.ROOT, "%s_%s.log",
project, stepId);
Review comment:
I am afraid this will introduce security issue. For `downloadFilename`
is combined with two parameters(`project` & `step_id`) from HTTP request.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235143#comment-17235143
]
ASF GitHub Bot commented on KYLIN-4813:
---
hit-lacus commented on a change in pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#discussion_r526569415
##
File path:
server-base/src/main/java/org/apache/kylin/rest/controller/JobController.java
##
@@ -160,17 +168,41 @@ public JobInstance get(@PathVariable String jobId) {
* @return
* @throws IOException
*/
+
@RequestMapping(value = "/{jobId}/steps/{stepId}/output", method = {
RequestMethod.GET }, produces = {
"application/json" })
@ResponseBody
public Map getStepOutput(@PathVariable String jobId,
@PathVariable String stepId) {
Map result = new HashMap();
result.put("jobId", jobId);
result.put("stepId", String.valueOf(stepId));
-result.put("cmd_output",
jobService.getExecutableManager().getOutput(stepId).getVerboseMsg());
+result.put("cmd_output", jobService.getJobOutput(jobId, stepId));
return result;
}
+/**
+ * Download a job step output from hdfs
+ * @param jobId
+ * @param stepId
+ * @param project
+ * @param response
+ * @return
+ */
+@RequestMapping(value = "/{job_id:.+}/steps/{step_id:.+}/log", method = {
RequestMethod.GET }, produces = { "application/json" })
+@ResponseBody
+public EnvelopeResponse downloadLogFile(@PathVariable("job_id")
String jobId,
+@PathVariable("step_id")
String stepId, @RequestParam(value = "project") String project,
+HttpServletResponse
response) throws IOException {
+checkRequiredArg("job_id", jobId);
+checkRequiredArg("step_id", stepId);
+checkRequiredArg("project", project);
+String downloadFilename = String.format(Locale.ROOT, "%s_%s.log",
project, stepId);
Review comment:
I am afraid this will introduce security issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235131#comment-17235131
]
ASF GitHub Bot commented on KYLIN-4813:
---
zzcclp commented on a change in pull request #1481:
URL: https://github.com/apache/kylin/pull/1481#discussion_r526552939
##
File path:
core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
##
@@ -2803,10 +2822,15 @@ public boolean isSparkEngineEnabled() {
return
Boolean.parseBoolean(getOptional("kylin.query.spark-engine.enabled", "true"));
}
-public String getLogSparkPropertiesFile() {
-return getLogPropertyFile("kylin-parquet-log4j.properties");
+public String getLogSparkDriverPropertiesFile() {
+return getLogPropertyFile("spark-driver-log4j.properties");
+}
+
+public String getLogSparkExecutorPropertiesFile() {
+return getLogPropertyFile("spark-executor-log4j.properties");
}
+
Review comment:
remove this blank line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234330#comment-17234330 ] ASF GitHub Bot commented on KYLIN-4813: --- lgtm-com[bot] commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-729493945 This pull request **introduces 1 alert** and **fixes 1** when merging e3005aafa8c4364d4fb0be4d8cd2a19e651a7e56 into 0b4df4cf67b27c6f9a7a9e5c9006ddf528acc09b - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-c91dbb23386e54b46384017e505915a28cc283c1) **new alerts:** * 1 for HTTP response splitting **fixed alerts:** * 1 for Spurious Javadoc @param tags This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234251#comment-17234251 ] ASF GitHub Bot commented on KYLIN-4813: --- codecov-io commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-729431698 # [Codecov](https://codecov.io/gh/apache/kylin/pull/1481?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`kylin-on-parquet-v2@0894c0d`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/kylin/pull/1481?src=pr=tree) ```diff @@ Coverage Diff @@ ## kylin-on-parquet-v2#1481 +/- ## == Coverage ? 24.16% Complexity ? 4607 == Files ? 1133 Lines ?64230 Branches ? 9483 == Hits ?15518 Misses ?47102 Partials ? 1610 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/1481?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/1481?src=pr=footer). Last update [0894c0d...f1042da](https://codecov.io/gh/apache/kylin/pull/1481?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234217#comment-17234217 ] ASF GitHub Bot commented on KYLIN-4813: --- lgtm-com[bot] commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-729394829 This pull request **introduces 1 alert** and **fixes 1** when merging f1042da2b992d771dfe210c2bcd2913564e5123b into a3ac3feff315dd6b30403096dfbc812ad808f418 - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-99a311dc55c98f422d02143d0dff630bd8603a1e) **new alerts:** * 1 for HTTP response splitting **fixed alerts:** * 1 for Spurious Javadoc @param tags This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234210#comment-17234210 ] ASF GitHub Bot commented on KYLIN-4813: --- lgtm-com[bot] commented on pull request #1481: URL: https://github.com/apache/kylin/pull/1481#issuecomment-729374297 This pull request **introduces 1 alert** and **fixes 1** when merging a4a49d13f834c78047e26721e3661853243763e3 into a3ac3feff315dd6b30403096dfbc812ad808f418 - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-78a8f71c579ece2ed96051af1b8d91505c2b9de7) **new alerts:** * 1 for HTTP response splitting **fixed alerts:** * 1 for Spurious Javadoc @param tags This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
