[jira] [Commented] (TEZ-4051) Proto history events should selectively drop events
[ https://issues.apache.org/jira/browse/TEZ-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16780295#comment-16780295 ] Harish Jaiprakash commented on TEZ-4051: [~prasanth_j], please take a look at: TEZ-3601, setting the appropriate log level should accomplish what you need I think. Else you can modify similarly. > Proto history events should selectively drop events > --- > > Key: TEZ-4051 > URL: https://issues.apache.org/jira/browse/TEZ-4051 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.9.next >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: TEZ-4051.1.patch > > > Following up with TEZ-4048, provide an option to selection drop events that > may not be required which could potentially take up queue space. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4042) Speculative attempts should avoid running on the same node
[ https://issues.apache.org/jira/browse/TEZ-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772082#comment-16772082 ] Harish Jaiprakash commented on TEZ-4042: Sorry, should have read better. This affects only speculative execution. Ignore comment above. > Speculative attempts should avoid running on the same node > -- > > Key: TEZ-4042 > URL: https://issues.apache.org/jira/browse/TEZ-4042 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Ying Han >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4042) Speculative attempts should avoid running on the same node
[ https://issues.apache.org/jira/browse/TEZ-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772077#comment-16772077 ] Harish Jaiprakash commented on TEZ-4042: Not sure if this is the best strategy. What happens if there is only one node or yarn keeps giving containers from one node because rest are occupied and it gets marked as unhealthy? There should be a mechanism built so that it will fall back and try on the unhealthy node. At the very least please protect it via a config. So that it can be enabled or disabled. > Speculative attempts should avoid running on the same node > -- > > Key: TEZ-4042 > URL: https://issues.apache.org/jira/browse/TEZ-4042 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Ying Han >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4028) Events not visible from proto history logging for s3a filesystem until dag completes.
[ https://issues.apache.org/jira/browse/TEZ-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739114#comment-16739114 ] Harish Jaiprakash commented on TEZ-4028: Thanks [~ewohlstadter]. Adding a new constant and using it in both test and service. Please take a look. > Events not visible from proto history logging for s3a filesystem until dag > completes. > - > > Key: TEZ-4028 > URL: https://issues.apache.org/jira/browse/TEZ-4028 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: TEZ-4028.01.patch, TEZ-4028.02.patch > > > The events are not visible in the files because s3 filesystem > * flush writes to local disk and only upload/commit to s3 on close. > * does not support append > As an initial fix we log the dag submitted, initialized and started events > into a file and these can be read to get the dag plan, config from the AM. > The counters are anyways not available until the dag completes. > The in-progress information cannot be read, this can be obtained from the AM > once we have the above events. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4028) Events not visible from proto history logging for s3a filesystem until dag completes.
[ https://issues.apache.org/jira/browse/TEZ-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-4028: --- Attachment: TEZ-4028.02.patch > Events not visible from proto history logging for s3a filesystem until dag > completes. > - > > Key: TEZ-4028 > URL: https://issues.apache.org/jira/browse/TEZ-4028 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: TEZ-4028.01.patch, TEZ-4028.02.patch > > > The events are not visible in the files because s3 filesystem > * flush writes to local disk and only upload/commit to s3 on close. > * does not support append > As an initial fix we log the dag submitted, initialized and started events > into a file and these can be read to get the dag plan, config from the AM. > The counters are anyways not available until the dag completes. > The in-progress information cannot be read, this can be obtained from the AM > once we have the above events. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4028) Events not visible from proto history logging for s3a filesystem until dag completes.
[ https://issues.apache.org/jira/browse/TEZ-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-4028: --- Attachment: TEZ-4028.01.patch > Events not visible from proto history logging for s3a filesystem until dag > completes. > - > > Key: TEZ-4028 > URL: https://issues.apache.org/jira/browse/TEZ-4028 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: TEZ-4028.01.patch > > > The events are not visible in the files because s3 filesystem > * flush writes to local disk and only upload/commit to s3 on close. > * does not support append > As an initial fix we log the dag submitted, initialized and started events > into a file and these can be read to get the dag plan, config from the AM. > The counters are anyways not available until the dag completes. > The in-progress information cannot be read, this can be obtained from the AM > once we have the above events. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-4028) Events not visible from proto history logging for s3a filesystem until dag completes.
Harish Jaiprakash created TEZ-4028: -- Summary: Events not visible from proto history logging for s3a filesystem until dag completes. Key: TEZ-4028 URL: https://issues.apache.org/jira/browse/TEZ-4028 Project: Apache Tez Issue Type: Bug Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash The events are not visible in the files because s3 filesystem * flush writes to local disk and only upload/commit to s3 on close. * does not support append As an initial fix we log the dag submitted, initialized and started events into a file and these can be read to get the dag plan, config from the AM. The counters are anyways not available until the dag completes. The in-progress information cannot be read, this can be obtained from the AM once we have the above events. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3979) Redact configuration when it's being logged, sent to the timeline server or displayed in some way
[
https://issues.apache.org/jira/browse/TEZ-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16653071#comment-16653071
]
Harish Jaiprakash commented on TEZ-3979:
+1, change looks fine. Just one question, should you merge
mapreduce.job.redacted-properties in MRTask incase its set.
> Redact configuration when it's being logged, sent to the timeline server or
> displayed in some way
> -
>
> Key: TEZ-3979
> URL: https://issues.apache.org/jira/browse/TEZ-3979
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Affects Versions: 0.9.1
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: TEZ-3979.1.patch, TEZ-3979.2.patch, TEZ-3979.3.patch
>
>
> The configuration is being logged in some places like
> [here|https://github.com/apache/tez/blob/90c8195de0ddee9da3d4cd07519c8e287c85ad50/tez-dag/src/main/java/org/apache/tez/dag/history/utils/DAGUtils.java#L508]
> and
> [here|https://github.com/apache/tez/blob/90c8195de0ddee9da3d4cd07519c8e287c85ad50/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/MRTask.java#L177]
> . We should do something like [what it's done in the hadoop code
> base|https://github.com/apache/hadoop/blob/d951af22b42a22c09cfeecd3c866d5f1cd412120/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java#L558]
> and redact the properties held in {{mapreduce.job.redacted-properties.}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3960) Better error handling in proto history logger and add doAs support.
[ https://issues.apache.org/jira/browse/TEZ-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527945#comment-16527945 ] Harish Jaiprakash commented on TEZ-3960: Thanks [~hagleitn], committing patch. > Better error handling in proto history logger and add doAs support. > --- > > Key: TEZ-3960 > URL: https://issues.apache.org/jira/browse/TEZ-3960 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.10.0 > > Attachments: TEZ-3960.01.patch, TEZ-3960.02.patch > > > DagManifestScanner gets stuck for a days logs if there are errors in them. > Fix it using fixed number of retries. > The scanner should be able to use doAs to ensure it can read files if run > using a proxy admin user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3960) Better error handling in proto history logger and add doAs support.
[ https://issues.apache.org/jira/browse/TEZ-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525258#comment-16525258 ] Harish Jaiprakash commented on TEZ-3960: The test case runs fine on my local machine. All tests timed out in FileDescriptor.sync calls, should be some temporary IO issue. > Better error handling in proto history logger and add doAs support. > --- > > Key: TEZ-3960 > URL: https://issues.apache.org/jira/browse/TEZ-3960 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.10.0 > > Attachments: TEZ-3960.01.patch, TEZ-3960.02.patch > > > DagManifestScanner gets stuck for a days logs if there are errors in them. > Fix it using fixed number of retries. > The scanner should be able to use doAs to ensure it can read files if run > using a proxy admin user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3960) Better error handling in proto history logger and add doAs support.
[ https://issues.apache.org/jira/browse/TEZ-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3960: --- Attachment: TEZ-3960.02.patch > Better error handling in proto history logger and add doAs support. > --- > > Key: TEZ-3960 > URL: https://issues.apache.org/jira/browse/TEZ-3960 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.10.0 > > Attachments: TEZ-3960.01.patch, TEZ-3960.02.patch > > > DagManifestScanner gets stuck for a days logs if there are errors in them. > Fix it using fixed number of retries. > The scanner should be able to use doAs to ensure it can read files if run > using a proxy admin user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3959) HTTP 502 for bower install
[
https://issues.apache.org/jira/browse/TEZ-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16524071#comment-16524071
]
Harish Jaiprakash commented on TEZ-3959:
[~kshukla], do you know why the build does not kick in for this patch? This
patch should fix the master build, but looks like if master is broken it will
not build. So do we just commit this change?
> HTTP 502 for bower install
> --
>
> Key: TEZ-3959
> URL: https://issues.apache.org/jira/browse/TEZ-3959
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Eric Wohlstadter
>Assignee: Harish Jaiprakash
>Priority: Blocker
> Attachments: TEZ-3959.01.patch, TEZ-3959.02.patch
>
>
> [~harishjp] and I started seeing:
> {code:java}
> 18:56:57 2018/06/24 01:56:57 INFO: [ERROR] bower moment#2.12.0
> EINVRES Request to https://bower.herokuapp.com/packages/moment failed with
> 502{code}
> [~harishjp] did some digging and found that changing to version 1.8.4 seems
> to be a workaround, but it is dubious:
> {code}
> ERROR warning bower@1.8.4: We don't recommend using Bower for new projects.
> Please consider Yarn and Webpack or Parcel. You can read how to migrate
> legacy project here:
> https://bower.io/blog/2017/how-to-migrate-away-from-bower/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3960) Better error handling in proto history logger and add doAs support.
[ https://issues.apache.org/jira/browse/TEZ-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3960: --- Attachment: TEZ-3960.01.patch > Better error handling in proto history logger and add doAs support. > --- > > Key: TEZ-3960 > URL: https://issues.apache.org/jira/browse/TEZ-3960 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.10.0 > > Attachments: TEZ-3960.01.patch > > > DagManifestScanner gets stuck for a days logs if there are errors in them. > Fix it using fixed number of retries. > The scanner should be able to use doAs to ensure it can read files if run > using a proxy admin user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3960) Better error handling in proto history logger and add doAs support.
Harish Jaiprakash created TEZ-3960: -- Summary: Better error handling in proto history logger and add doAs support. Key: TEZ-3960 URL: https://issues.apache.org/jira/browse/TEZ-3960 Project: Apache Tez Issue Type: Bug Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash DagManifestScanner gets stuck for a days logs if there are errors in them. Fix it using fixed number of retries. The scanner should be able to use doAs to ensure it can read files if run using a proxy admin user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3959) HTTP 502 for bower install
[
https://issues.apache.org/jira/browse/TEZ-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522328#comment-16522328
]
Harish Jaiprakash commented on TEZ-3959:
[~sseth], Please help us fix this. There is a bower issue causing the build to
fail. This patch should fix it, but the build fails with "master compilation
may be broken." not sure how to proceed.
> HTTP 502 for bower install
> --
>
> Key: TEZ-3959
> URL: https://issues.apache.org/jira/browse/TEZ-3959
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Eric Wohlstadter
>Assignee: Harish Jaiprakash
>Priority: Blocker
> Attachments: TEZ-3959.01.patch, TEZ-3959.02.patch
>
>
> [~harishjp] and I started seeing:
> {code:java}
> 18:56:57 2018/06/24 01:56:57 INFO: [ERROR] bower moment#2.12.0
> EINVRES Request to https://bower.herokuapp.com/packages/moment failed with
> 502{code}
> [~harishjp] did some digging and found that changing to version 1.8.4 seems
> to be a workaround, but it is dubious:
> {code}
> ERROR warning bower@1.8.4: We don't recommend using Bower for new projects.
> Please consider Yarn and Webpack or Parcel. You can read how to migrate
> legacy project here:
> https://bower.io/blog/2017/how-to-migrate-away-from-bower/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3959) HTTP 502 for bower install
[
https://issues.apache.org/jira/browse/TEZ-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521960#comment-16521960
]
Harish Jaiprakash commented on TEZ-3959:
The older patch failed, do not know why. Creating another patch moving bower to
1.8.4.
> HTTP 502 for bower install
> --
>
> Key: TEZ-3959
> URL: https://issues.apache.org/jira/browse/TEZ-3959
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Eric Wohlstadter
>Assignee: Harish Jaiprakash
>Priority: Blocker
> Attachments: TEZ-3959.01.patch, TEZ-3959.02.patch
>
>
> [~harishjp] and I started seeing:
> {code:java}
> 18:56:57 2018/06/24 01:56:57 INFO: [ERROR] bower moment#2.12.0
> EINVRES Request to https://bower.herokuapp.com/packages/moment failed with
> 502{code}
> [~harishjp] did some digging and found that changing to version 1.8.4 seems
> to be a workaround, but it is dubious:
> {code}
> ERROR warning bower@1.8.4: We don't recommend using Bower for new projects.
> Please consider Yarn and Webpack or Parcel. You can read how to migrate
> legacy project here:
> https://bower.io/blog/2017/how-to-migrate-away-from-bower/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3959) HTTP 502 for bower install
[
https://issues.apache.org/jira/browse/TEZ-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated TEZ-3959:
---
Attachment: TEZ-3959.02.patch
> HTTP 502 for bower install
> --
>
> Key: TEZ-3959
> URL: https://issues.apache.org/jira/browse/TEZ-3959
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Eric Wohlstadter
>Assignee: Harish Jaiprakash
>Priority: Blocker
> Attachments: TEZ-3959.01.patch, TEZ-3959.02.patch
>
>
> [~harishjp] and I started seeing:
> {code:java}
> 18:56:57 2018/06/24 01:56:57 INFO: [ERROR] bower moment#2.12.0
> EINVRES Request to https://bower.herokuapp.com/packages/moment failed with
> 502{code}
> [~harishjp] did some digging and found that changing to version 1.8.4 seems
> to be a workaround, but it is dubious:
> {code}
> ERROR warning bower@1.8.4: We don't recommend using Bower for new projects.
> Please consider Yarn and Webpack or Parcel. You can read how to migrate
> legacy project here:
> https://bower.io/blog/2017/how-to-migrate-away-from-bower/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3959) HTTP 502 for bower install
[
https://issues.apache.org/jira/browse/TEZ-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated TEZ-3959:
---
Target Version/s: 0.9.next, 0.10.0 (was: 0.10.0)
> HTTP 502 for bower install
> --
>
> Key: TEZ-3959
> URL: https://issues.apache.org/jira/browse/TEZ-3959
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Eric Wohlstadter
>Assignee: Harish Jaiprakash
>Priority: Blocker
> Attachments: TEZ-3959.01.patch
>
>
> [~harishjp] and I started seeing:
> {code:java}
> 18:56:57 2018/06/24 01:56:57 INFO: [ERROR] bower moment#2.12.0
> EINVRES Request to https://bower.herokuapp.com/packages/moment failed with
> 502{code}
> [~harishjp] did some digging and found that changing to version 1.8.4 seems
> to be a workaround, but it is dubious:
> {code}
> ERROR warning bower@1.8.4: We don't recommend using Bower for new projects.
> Please consider Yarn and Webpack or Parcel. You can read how to migrate
> legacy project here:
> https://bower.io/blog/2017/how-to-migrate-away-from-bower/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3959) HTTP 502 for bower install
[
https://issues.apache.org/jira/browse/TEZ-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated TEZ-3959:
---
Fix Version/s: (was: 0.10.0)
(was: 0.9.next)
> HTTP 502 for bower install
> --
>
> Key: TEZ-3959
> URL: https://issues.apache.org/jira/browse/TEZ-3959
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Eric Wohlstadter
>Assignee: Harish Jaiprakash
>Priority: Blocker
> Attachments: TEZ-3959.01.patch
>
>
> [~harishjp] and I started seeing:
> {code:java}
> 18:56:57 2018/06/24 01:56:57 INFO: [ERROR] bower moment#2.12.0
> EINVRES Request to https://bower.herokuapp.com/packages/moment failed with
> 502{code}
> [~harishjp] did some digging and found that changing to version 1.8.4 seems
> to be a workaround, but it is dubious:
> {code}
> ERROR warning bower@1.8.4: We don't recommend using Bower for new projects.
> Please consider Yarn and Webpack or Parcel. You can read how to migrate
> legacy project here:
> https://bower.io/blog/2017/how-to-migrate-away-from-bower/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (TEZ-3959) HTTP 502 for bower install
[
https://issues.apache.org/jira/browse/TEZ-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521930#comment-16521930
]
Harish Jaiprakash edited comment on TEZ-3959 at 6/25/18 6:59 AM:
-
Fixed this by setting registry url, the tests are passing. We should move away
from bower ASAP.
[~Sreenath], Please take a look at this.
was (Author: harishjp):
Fixed this by setting registry url, the tests are passing. We need a permanent
fix at some point.
[~Sreenath], Please take a look at this.
> HTTP 502 for bower install
> --
>
> Key: TEZ-3959
> URL: https://issues.apache.org/jira/browse/TEZ-3959
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Eric Wohlstadter
>Assignee: Harish Jaiprakash
>Priority: Blocker
> Attachments: TEZ-3959.01.patch
>
>
> [~harishjp] and I started seeing:
> {code:java}
> 18:56:57 2018/06/24 01:56:57 INFO: [ERROR] bower moment#2.12.0
> EINVRES Request to https://bower.herokuapp.com/packages/moment failed with
> 502{code}
> [~harishjp] did some digging and found that changing to version 1.8.4 seems
> to be a workaround, but it is dubious:
> {code}
> ERROR warning bower@1.8.4: We don't recommend using Bower for new projects.
> Please consider Yarn and Webpack or Parcel. You can read how to migrate
> legacy project here:
> https://bower.io/blog/2017/how-to-migrate-away-from-bower/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3959) HTTP 502 for bower install
[
https://issues.apache.org/jira/browse/TEZ-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521930#comment-16521930
]
Harish Jaiprakash commented on TEZ-3959:
Fixed this by setting registry url, the tests are passing. We need a permanent
fix at some point.
[~Sreenath], Please take a look at this.
> HTTP 502 for bower install
> --
>
> Key: TEZ-3959
> URL: https://issues.apache.org/jira/browse/TEZ-3959
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Eric Wohlstadter
>Assignee: Harish Jaiprakash
>Priority: Blocker
> Attachments: TEZ-3959.01.patch
>
>
> [~harishjp] and I started seeing:
> {code:java}
> 18:56:57 2018/06/24 01:56:57 INFO: [ERROR] bower moment#2.12.0
> EINVRES Request to https://bower.herokuapp.com/packages/moment failed with
> 502{code}
> [~harishjp] did some digging and found that changing to version 1.8.4 seems
> to be a workaround, but it is dubious:
> {code}
> ERROR warning bower@1.8.4: We don't recommend using Bower for new projects.
> Please consider Yarn and Webpack or Parcel. You can read how to migrate
> legacy project here:
> https://bower.io/blog/2017/how-to-migrate-away-from-bower/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (TEZ-3959) HTTP 502 for bower install
[
https://issues.apache.org/jira/browse/TEZ-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash reassigned TEZ-3959:
--
Assignee: Harish Jaiprakash
> HTTP 502 for bower install
> --
>
> Key: TEZ-3959
> URL: https://issues.apache.org/jira/browse/TEZ-3959
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Eric Wohlstadter
>Assignee: Harish Jaiprakash
>Priority: Blocker
> Attachments: TEZ-3959.01.patch
>
>
> [~harishjp] and I started seeing:
> {code:java}
> 18:56:57 2018/06/24 01:56:57 INFO: [ERROR] bower moment#2.12.0
> EINVRES Request to https://bower.herokuapp.com/packages/moment failed with
> 502{code}
> [~harishjp] did some digging and found that changing to version 1.8.4 seems
> to be a workaround, but it is dubious:
> {code}
> ERROR warning bower@1.8.4: We don't recommend using Bower for new projects.
> Please consider Yarn and Webpack or Parcel. You can read how to migrate
> legacy project here:
> https://bower.io/blog/2017/how-to-migrate-away-from-bower/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3959) HTTP 502 for bower install
[
https://issues.apache.org/jira/browse/TEZ-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harish Jaiprakash updated TEZ-3959:
---
Attachment: TEZ-3959.01.patch
> HTTP 502 for bower install
> --
>
> Key: TEZ-3959
> URL: https://issues.apache.org/jira/browse/TEZ-3959
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Eric Wohlstadter
>Priority: Blocker
> Attachments: TEZ-3959.01.patch
>
>
> [~harishjp] and I started seeing:
> {code:java}
> 18:56:57 2018/06/24 01:56:57 INFO: [ERROR] bower moment#2.12.0
> EINVRES Request to https://bower.herokuapp.com/packages/moment failed with
> 502{code}
> [~harishjp] did some digging and found that changing to version 1.8.4 seems
> to be a workaround, but it is dubious:
> {code}
> ERROR warning bower@1.8.4: We don't recommend using Bower for new projects.
> Please consider Yarn and Webpack or Parcel. You can read how to migrate
> legacy project here:
> https://bower.io/blog/2017/how-to-migrate-away-from-bower/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3951) TezClient wait too long for the DAGClient for prewarm; tries to shut down the wrong DAG
[ https://issues.apache.org/jira/browse/TEZ-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16508051#comment-16508051 ] Harish Jaiprakash commented on TEZ-3951: Thanks [~sershe]. Committed to master and 0.9. > TezClient wait too long for the DAGClient for prewarm; tries to shut down the > wrong DAG > --- > > Key: TEZ-3951 > URL: https://issues.apache.org/jira/browse/TEZ-3951 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: TEZ-3951.01.patch, TEZ-3951.patch > > > Follow-up from TEZ-3943 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (TEZ-3920) Setting tez.am.log.level does not take full effect if more than one clause is present
[ https://issues.apache.org/jira/browse/TEZ-3920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448908#comment-16448908 ] Harish Jaiprakash edited comment on TEZ-3920 at 4/23/18 9:52 PM: - [~kshukla] Use ';' as the separator, not comma. The code is in ./tez-common/src/main/java/org/apache/tez/common/TezLog4jConfigurator.java Sorry, I did not see the first line. I've used this and it works not sure what is wrong. was (Author: harishjp): [~kshukla] Use ';' as the separator, not comma. The code is in ./tez-common/src/main/java/org/apache/tez/common/TezLog4jConfigurator.java > Setting tez.am.log.level does not take full effect if more than one clause is > present > - > > Key: TEZ-3920 > URL: https://issues.apache.org/jira/browse/TEZ-3920 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1 >Reporter: Kuhu Shukla >Priority: Major > > tez.am.log.level=DEBUG;org.apache.hadoop.ipc=INFO;org.apache.hadoop.security=INFO > Or > tez.am.log.level=DEBUG,org.apache.hadoop.ipc=INFO,org.apache.hadoop.security=INFO > only makes it so that all logs are at DEBUG level and the ipc/security logs > in this example still print at debug level. I used the command line for > testing this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3920) Setting tez.am.log.level does not take full effect if more than one clause is present
[ https://issues.apache.org/jira/browse/TEZ-3920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448908#comment-16448908 ] Harish Jaiprakash commented on TEZ-3920: [~kshukla] Use ';' as the separator, not comma. The code is in ./tez-common/src/main/java/org/apache/tez/common/TezLog4jConfigurator.java > Setting tez.am.log.level does not take full effect if more than one clause is > present > - > > Key: TEZ-3920 > URL: https://issues.apache.org/jira/browse/TEZ-3920 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1 >Reporter: Kuhu Shukla >Priority: Major > > tez.am.log.level=DEBUG;org.apache.hadoop.ipc=INFO;org.apache.hadoop.security=INFO > Or > tez.am.log.level=DEBUG,org.apache.hadoop.ipc=INFO,org.apache.hadoop.security=INFO > only makes it so that all logs are at DEBUG level and the ipc/security logs > in this example still print at debug level. I used the command line for > testing this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446517#comment-16446517 ] Harish Jaiprakash commented on TEZ-3915: [~jeagles], sorry for committing it early. I did look at TEZ-3914 and incorporate the CodedInputStream/CodedOutputStream reuse. Some more context on this patch: We found that ATSv15 does not scale on read, and way to go forward is to use ATSv2. ATSv2 still needs some design changes before we can get it integrated fully. For lot of debugging we rely on this data. And decided to create a format which logs to hdfs, we can download this file or mount it as a hive table and analyze the data in it. Adding the reader into it will also allow people to use it independently. I made a mistake by not sharing the design details with the community to start with. I'll share details on bigger patches earlier so that a discussion can be had before implementing something like that. Sorry about that. > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.9.next > > Attachments: TEZ-3915.01.patch, TEZ-3915.02.patch, TEZ-3915.03.patch, > TEZ-3915.04.patch, TEZ-3915.05.patch, TEZ-3915.06.patch, TEZ-3915.07.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3915: --- Attachment: TEZ-3915.07.patch > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.9.next > > Attachments: TEZ-3915.01.patch, TEZ-3915.02.patch, TEZ-3915.03.patch, > TEZ-3915.04.patch, TEZ-3915.05.patch, TEZ-3915.06.patch, TEZ-3915.07.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443399#comment-16443399 ] Harish Jaiprakash commented on TEZ-3915: [~jeagles], the files are sequence file logged into date partitioned directories. These can be setup as external table in hive to run any analysis. A hive serde to required read from these, but we do not want any hive dependencies. I'll see where this can be added. The reader and writers are completely contained in tez and anyone can use these to read and analyse the events. W.r.t current loggers, ATS has performance issues in read side and SimpleHistoryLogger is not partitioned and being a json format has lot of overhead. I initially looked at using the recovery protos, but those are designed specifically for recovery and do not have fields like config, counters, diagnostics, dagPlan, ... in them. Even if these are modified, a different converter has to be written to keep recovery protos light. > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.9.next > > Attachments: TEZ-3915.01.patch, TEZ-3915.02.patch, TEZ-3915.03.patch, > TEZ-3915.04.patch, TEZ-3915.05.patch, TEZ-3915.06.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3915: --- Attachment: TEZ-3915.06.patch > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.9.next > > Attachments: TEZ-3915.01.patch, TEZ-3915.02.patch, TEZ-3915.03.patch, > TEZ-3915.04.patch, TEZ-3915.05.patch, TEZ-3915.06.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3915: --- Attachment: TEZ-3915.05.patch > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.9.next > > Attachments: TEZ-3915.01.patch, TEZ-3915.02.patch, TEZ-3915.03.patch, > TEZ-3915.04.patch, TEZ-3915.05.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3915: --- Attachment: TEZ-3915.04.patch > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.9.next > > Attachments: TEZ-3915.01.patch, TEZ-3915.02.patch, TEZ-3915.03.patch, > TEZ-3915.04.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440484#comment-16440484 ] Harish Jaiprakash commented on TEZ-3915: Adding support to load am_launched event to extract config. > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.9.next > > Attachments: TEZ-3915.01.patch, TEZ-3915.02.patch, TEZ-3915.03.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3915: --- Attachment: TEZ-3915.03.patch > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.9.next > > Attachments: TEZ-3915.01.patch, TEZ-3915.02.patch, TEZ-3915.03.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439093#comment-16439093 ] Harish Jaiprakash commented on TEZ-3915: Fixed findbugs and release audit failures. TestAMRecovery failure is not related to this issue. Note: I had to increase the hadoop version from 2.7.0 to 2.7.2, since the append feature for SequenceFile was added in 2.7.2. I hope that is alright. > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.9.next > > Attachments: TEZ-3915.01.patch, TEZ-3915.02.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3915: --- Attachment: TEZ-3915.02.patch > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.9.next > > Attachments: TEZ-3915.01.patch, TEZ-3915.02.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439011#comment-16439011 ] Harish Jaiprakash commented on TEZ-3915: [~sseth]/[~ewohlstadter] Please review. > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.9.next > > Attachments: TEZ-3915.01.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3915: --- Issue Type: Improvement (was: Bug) > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.9.next > > Attachments: TEZ-3915.01.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3915: --- Fix Version/s: 0.9.next > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Fix For: 0.9.next > > Attachments: TEZ-3915.01.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3915) Create protobuf based history event logger.
[ https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3915: --- Attachment: TEZ-3915.01.patch > Create protobuf based history event logger. > --- > > Key: TEZ-3915 > URL: https://issues.apache.org/jira/browse/TEZ-3915 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: TEZ-3915.01.patch > > > A protobuf based history event logger, to log directly into hdfs. Implement a > reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3915) Create protobuf based history event logger.
Harish Jaiprakash created TEZ-3915: -- Summary: Create protobuf based history event logger. Key: TEZ-3915 URL: https://issues.apache.org/jira/browse/TEZ-3915 Project: Apache Tez Issue Type: Bug Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash A protobuf based history event logger, to log directly into hdfs. Implement a reader api also, to get the events from the files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[ https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390759#comment-16390759 ] Harish Jaiprakash commented on TEZ-3901: Sorry [~ewohlstadter]. I did not see the second patch. LGTM +1. > Add hadoop3 profile for upgrade to Jersey 1.19 > -- > > Key: TEZ-3901 > URL: https://issues.apache.org/jira/browse/TEZ-3901 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3901.1.patch, TEZ-3901.2.patch > > > From [~harishjp]: > "DAGAppMaster fails to start when using hadoop3 and ATSv15, because > TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez > packages jersey-client 1.9 with it. There are incompatible changes between > them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in > older hadoop and 1.19 in hadoop3." > > This patch includes some copy and paste of the hadoop28 profile to a hadoop3 > profile. Maven doesn't include anything like "profile inheritance". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[
https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390745#comment-16390745
]
Harish Jaiprakash commented on TEZ-3901:
{code:java}
$ mvn clean install[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] 'dependencies.dependency.version' for com.sun.jersey:jersey-client:jar
must be a valid version but is '${jersey.version}'. @
org.apache.tez:tez:0.9.2-SNAPSHOT, /Users/harishjp/devel/tez/pom.xml, line 727,
column 17
[ERROR] 'dependencies.dependency.version' for com.sun.jersey:jersey-json:jar
must be a valid version but is '${jersey.version}'. @
org.apache.tez:tez:0.9.2-SNAPSHOT, /Users/harishjp/devel/tez/pom.xml, line 732,
column 18
...{code}
I compile with java 8, which activates jdk1.8 profile and hadoop27 profile gets
deactivated, because of how maven handles activateByDefault and hence the
suggestion to move the property jersey.version to global properties and
override them for hadoop28. With this patch we cannot build with Java 8 and
hadoop 2.7.x.
> Add hadoop3 profile for upgrade to Jersey 1.19
> --
>
> Key: TEZ-3901
> URL: https://issues.apache.org/jira/browse/TEZ-3901
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3901.1.patch, TEZ-3901.2.patch
>
>
> From [~harishjp]:
> "DAGAppMaster fails to start when using hadoop3 and ATSv15, because
> TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez
> packages jersey-client 1.9 with it. There are incompatible changes between
> them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in
> older hadoop and 1.19 in hadoop3."
>
> This patch includes some copy and paste of the hadoop28 profile to a hadoop3
> profile. Maven doesn't include anything like "profile inheritance".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[ https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390511#comment-16390511 ] Harish Jaiprakash commented on TEZ-3901: This might explain: https://issues.apache.org/jira/browse/MNG-4917 since jdk18 is getting activated it disables hadoop27. Adding jersey.version=1.9 into the default properties list and override in a profile is working for me. If we pass -Phadoop27, tez-plugins/tez-yarn-timeline-history/pom.xml fails. The profile dependency is wrong there, its an older bug we never hit, since we assumed default build was hadoop27. > Add hadoop3 profile for upgrade to Jersey 1.19 > -- > > Key: TEZ-3901 > URL: https://issues.apache.org/jira/browse/TEZ-3901 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3901.1.patch > > > From [~harishjp]: > "DAGAppMaster fails to start when using hadoop3 and ATSv15, because > TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez > packages jersey-client 1.9 with it. There are incompatible changes between > them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in > older hadoop and 1.19 in hadoop3." > > This patch includes some copy and paste of the hadoop28 profile to a hadoop3 > profile. Maven doesn't include anything like "profile inheritance". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[
https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390472#comment-16390472
]
Harish Jaiprakash edited comment on TEZ-3901 at 3/8/18 12:05 AM:
-
mvn clean install fails for me. Not sure why its not reading the jersey.version
property from hadoop27 profile. My maven version is: 3.5.2.
The other 2 profiles are working fine.
{{[ERROR] [ERROR] Some problems were encountered while processing the POMs:}}
{{[ERROR] 'dependencies.dependency.version' for
com.sun.jersey:jersey-client:jar must be a valid version but is
'${jersey.version}'. @ org.apache.tez:tez:0.9.2-SNAPSHOT,
/Users/harishjp/devel/tez/pom.xml, line 727, column 17}}
{{[ERROR] 'dependencies.dependency.version' for com.sun.jersey:jersey-json:jar
must be a valid version but is '${jersey.version}'. @
org.apache.tez:tez:0.9.2-SNAPSHOT, /Users/harishjp/devel/tez/pom.xml, line 732,
column 18}}
was (Author: harishjp):
mvn clean install fails for me. Not sure why its not reading the jersey.version
property from hadoop27 profile. My maven version is: 3.5.2.
The other 2 profiles are working fine.
> Add hadoop3 profile for upgrade to Jersey 1.19
> --
>
> Key: TEZ-3901
> URL: https://issues.apache.org/jira/browse/TEZ-3901
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3901.1.patch
>
>
> From [~harishjp]:
> "DAGAppMaster fails to start when using hadoop3 and ATSv15, because
> TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez
> packages jersey-client 1.9 with it. There are incompatible changes between
> them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in
> older hadoop and 1.19 in hadoop3."
>
> This patch includes some copy and paste of the hadoop28 profile to a hadoop3
> profile. Maven doesn't include anything like "profile inheritance".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
[ https://issues.apache.org/jira/browse/TEZ-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390472#comment-16390472 ] Harish Jaiprakash commented on TEZ-3901: mvn clean install fails for me. Not sure why its not reading the jersey.version property from hadoop27 profile. My maven version is: 3.5.2. The other 2 profiles are working fine. > Add hadoop3 profile for upgrade to Jersey 1.19 > -- > > Key: TEZ-3901 > URL: https://issues.apache.org/jira/browse/TEZ-3901 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3901.1.patch > > > From [~harishjp]: > "DAGAppMaster fails to start when using hadoop3 and ATSv15, because > TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez > packages jersey-client 1.9 with it. There are incompatible changes between > them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in > older hadoop and 1.19 in hadoop3." > > This patch includes some copy and paste of the hadoop28 profile to a hadoop3 > profile. Maven doesn't include anything like "profile inheritance". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3884) Hadoop3-beta1 fixes for Tez tests
[
https://issues.apache.org/jira/browse/TEZ-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16330124#comment-16330124
]
Harish Jaiprakash commented on TEZ-3884:
Sorry about it.
DAGAppMaster fails to start when using hadoop3 and ATSv15, because
TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez
packages jersey-client 1.9 with it. There are incompatible changes between
them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in older
hadoop and 1.19 in hadoop3.
> Hadoop3-beta1 fixes for Tez tests
> -
>
> Key: TEZ-3884
> URL: https://issues.apache.org/jira/browse/TEZ-3884
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Gopal V
>Priority: Minor
> Attachments: TEZ-3884.patch.1
>
>
> {code}
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[48,30]
> cannot find symbol
> [ERROR] symbol: class DistributedFileSystem
> [ERROR] location: package org.apache.hadoop.hdfs
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[680,50]
> cannot find symbol
> [ERROR] symbol: class DistributedFileSystem
> [ERROR] location: class org.apache.tez.client.TestTezClientUtils
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/common/TestTezCommonUtils.java:[62,42]
> cannot access org.apache.hadoop.hdfs.DistributedFileSystem
> [ERROR] class file for org.apache.hadoop.hdfs.DistributedFileSystem not found
> [ERROR] -> [Help 1]
> [ERROR]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3884) Hadoop3-beta1 fixes for Tez tests
[
https://issues.apache.org/jira/browse/TEZ-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16330019#comment-16330019
]
Harish Jaiprakash commented on TEZ-3884:
Thanks [~ewohlstadter]. I was trying to build tez with hadoop3. We can remove
the dependency in
tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java by directly
using FileSystem instead of DistributedFileSystem. I did hope that should fix
things but its still fails in
tez-api/src/test/java/org/apache/tez/common/TestTezCommonUtils.java, because
hadoop-hdfs has made the dependency hadoop-hdfs-client scoped as provided. Not
sure why that is, but since we only use this as part of test, we should make
this dependency test scoped.
Another change we need is jersey-client version change from 1.9 to 1.19 for
hadoop3.
> Hadoop3-beta1 fixes for Tez tests
> -
>
> Key: TEZ-3884
> URL: https://issues.apache.org/jira/browse/TEZ-3884
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Gopal V
>Priority: Minor
> Attachments: TEZ-3884.patch.1
>
>
> {code}
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[48,30]
> cannot find symbol
> [ERROR] symbol: class DistributedFileSystem
> [ERROR] location: package org.apache.hadoop.hdfs
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[680,50]
> cannot find symbol
> [ERROR] symbol: class DistributedFileSystem
> [ERROR] location: class org.apache.tez.client.TestTezClientUtils
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/common/TestTezCommonUtils.java:[62,42]
> cannot access org.apache.hadoop.hdfs.DistributedFileSystem
> [ERROR] class file for org.apache.hadoop.hdfs.DistributedFileSystem not found
> [ERROR] -> [Help 1]
> [ERROR]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3851) I could see there is a missmatch in the bank name, The one i added to Tez is andhra bank(xx2496) but i got a message from Axis bank may be configuration issue in Data bas
[ https://issues.apache.org/jira/browse/TEZ-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16197968#comment-16197968 ] Harish Jaiprakash commented on TEZ-3851: You are looking for this: https://support.google.com/tez/?hl=en-GB#topic=7294297 > I could see there is a missmatch in the bank name, The one i added to Tez is > andhra bank(xx2496) but i got a message from Axis bank may be configuration > issue in Data base end. please look into this issue. my phone number with Tez > is 9642523682. > - > > Key: TEZ-3851 > URL: https://issues.apache.org/jira/browse/TEZ-3851 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: praveen >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3828) Allow relaxing locality when retried task's priority is kept same
[ https://issues.apache.org/jira/browse/TEZ-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162487#comment-16162487 ] Harish Jaiprakash commented on TEZ-3828: Thanks [~aplusplus]. +1 for the change. > Allow relaxing locality when retried task's priority is kept same > -- > > Key: TEZ-3828 > URL: https://issues.apache.org/jira/browse/TEZ-3828 > Project: Apache Tez > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: TEZ-3828.1.patch, TEZ-3828.2.patch, TEZ-3828.3.patch > > > TEZ-3716 introduced the conf to keep priority for retried task, but there is > no way to relax locality requirement in that case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TEZ-3775) Tez UI: Show DAG context in document title
[
https://issues.apache.org/jira/browse/TEZ-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065962#comment-16065962
]
Harish Jaiprakash commented on TEZ-3775:
{code}var titleText = typeof title == 'function' ? this.title() : title;{code}
Shouldn't we use title.apply(this)? It works currently because this.title and
this.get('title') are same.
> Tez UI: Show DAG context in document title
> ---
>
> Key: TEZ-3775
> URL: https://issues.apache.org/jira/browse/TEZ-3775
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: TEZ-3775.1.patch, TEZ-3775.2.patch, TEZ-3775.3.patch
>
>
> In Tez UI 0.7, DAG (vertex, app, task, attempt) context was shown in the
> document title. This was lost in the 0.9 UI migration. This jira attempts to
> bring that feature back. This feature is essential when supporting large
> clusters where a dev or support person may have dozens of dags open at the
> same time. Having context in the document title (the tab title), will allow
> us to quickly navigate.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (TEZ-3750) Add TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT to UnorderedPartitionedKVOutput.
[ https://issues.apache.org/jira/browse/TEZ-3750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3750: --- Summary: Add TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT to UnorderedPartitionedKVOutput. (was: Add TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT to UnorderedKVInput.) > Add TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT to > UnorderedPartitionedKVOutput. > > > Key: TEZ-3750 > URL: https://issues.apache.org/jira/browse/TEZ-3750 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3750.01.patch > > > The config TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT is > not propagated because its not white listed in UnorderedKVInput. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3749) Get map and reduce task memory from JobConf.
[ https://issues.apache.org/jira/browse/TEZ-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3749: --- Summary: Get map and reduce task memory from JobConf. (was: MRRSleep does not start in hadoop 3.) > Get map and reduce task memory from JobConf. > > > Key: TEZ-3749 > URL: https://issues.apache.org/jira/browse/TEZ-3749 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Minor > Fix For: 0.9.0 > > Attachments: TEZ-3749.01.patch > > > The default map memory and reduce memory is set to -1 in the hadoop config. > This has broken MRRSleep, as the resource request is sent with -1 memory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3749) Get map and reduce task memory from JobConf.
[ https://issues.apache.org/jira/browse/TEZ-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034007#comment-16034007 ] Harish Jaiprakash commented on TEZ-3749: Thanks [~sseth]. Committed to master: 1250ef59d28dd2a2cfafc8621fdf6fb551799ff8. > Get map and reduce task memory from JobConf. > > > Key: TEZ-3749 > URL: https://issues.apache.org/jira/browse/TEZ-3749 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Minor > Fix For: 0.9.0 > > Attachments: TEZ-3749.01.patch > > > The default map memory and reduce memory is set to -1 in the hadoop config. > This has broken MRRSleep, as the resource request is sent with -1 memory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3714) Tez UI: Hive Queries page: Use Dag ID and App ID if they are published form Hive side.
[ https://issues.apache.org/jira/browse/TEZ-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030704#comment-16030704 ] Harish Jaiprakash commented on TEZ-3714: Thanks [~Sreenath]. +1 for the patch. > Tez UI: Hive Queries page: Use Dag ID and App ID if they are published form > Hive side. > -- > > Key: TEZ-3714 > URL: https://issues.apache.org/jira/browse/TEZ-3714 > Project: Apache Tez > Issue Type: Bug >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3714.1.patch, TEZ-3714.2.patch > > > As of now Hive Queries pages does a reverse lookup to display DAG and App > IDs. i.e. For each query, the UI looks for DAGs where callerId = hive query > id. > This change prevents this ATS fetch when the data is available. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3750) Add TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT to UnorderedKVInput.
[ https://issues.apache.org/jira/browse/TEZ-3750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3750: --- Attachment: TEZ-3750.01.patch Add to UnorderedPartitionedKVOutput. Please do advise if this is required in UnorderedKVOutput. > Add TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT to > UnorderedKVInput. > > > Key: TEZ-3750 > URL: https://issues.apache.org/jira/browse/TEZ-3750 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3750.01.patch > > > The config TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT is > not propagated because its not white listed in UnorderedKVInput. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TEZ-3750) Add TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT to UnorderedKVInput.
Harish Jaiprakash created TEZ-3750: -- Summary: Add TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT to UnorderedKVInput. Key: TEZ-3750 URL: https://issues.apache.org/jira/browse/TEZ-3750 Project: Apache Tez Issue Type: Bug Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash The config TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT is not propagated because its not white listed in UnorderedKVInput. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3734) Remove config checks in Input/Output.
[ https://issues.apache.org/jira/browse/TEZ-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3734: --- Issue Type: Improvement (was: Bug) > Remove config checks in Input/Output. > - > > Key: TEZ-3734 > URL: https://issues.apache.org/jira/browse/TEZ-3734 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3734.01.patch > > > The configs in TezRuntimeConfiguration are not propagated if its not in > Input/Output checks, remove the checks and propagate all of > TezRuntimConfiguration to I/Os. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3749) MRRSleep does not start in hadoop 3.
[ https://issues.apache.org/jira/browse/TEZ-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3749: --- Attachment: TEZ-3749.01.patch Using JobConf to get the memory instead of reading config directly. TestMRHelpers.testContainerResourceConstruction would fail on hadoop 3.0. It does not fail anymore. No extra tests required. > MRRSleep does not start in hadoop 3. > > > Key: TEZ-3749 > URL: https://issues.apache.org/jira/browse/TEZ-3749 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Minor > Fix For: 0.9.0 > > Attachments: TEZ-3749.01.patch > > > The default map memory and reduce memory is set to -1 in the hadoop config. > This has broken MRRSleep, as the resource request is sent with -1 memory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TEZ-3749) MRRSleep does not start in hadoop 3.
Harish Jaiprakash created TEZ-3749: -- Summary: MRRSleep does not start in hadoop 3. Key: TEZ-3749 URL: https://issues.apache.org/jira/browse/TEZ-3749 Project: Apache Tez Issue Type: Bug Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash Priority: Minor Fix For: 0.9.0 The default map memory and reduce memory is set to -1 in the hadoop config. This has broken MRRSleep, as the resource request is sent with -1 memory. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3734) Remove config checks in Input/Output.
[ https://issues.apache.org/jira/browse/TEZ-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025804#comment-16025804 ] Harish Jaiprakash commented on TEZ-3734: Making sure all configs are used correctly with a test would be hard. Since, one can start using an existing config in a new I/O but will not get flagged as it would be covered in another I/O. Another way could be to wrap config into another config object which takes whitelisted runtime configs and throws an exception if a runtime config outside the whitelist is accessed from the I/O. This will ensure that the first time the config is tested either manually or through a test, the whitelisting is enforced or it fails. I can make this change if it sounds reasonable. > Remove config checks in Input/Output. > - > > Key: TEZ-3734 > URL: https://issues.apache.org/jira/browse/TEZ-3734 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3734.01.patch > > > The configs in TezRuntimeConfiguration are not propagated if its not in > Input/Output checks, remove the checks and propagate all of > TezRuntimConfiguration to I/Os. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3734) Remove config checks in Input/Output.
[ https://issues.apache.org/jira/browse/TEZ-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3734: --- Attachment: TEZ-3734.01.patch Removed all the config whitelisting from the Input and Output classes. Please take a look [~sseth], [~kshukla]. > Remove config checks in Input/Output. > - > > Key: TEZ-3734 > URL: https://issues.apache.org/jira/browse/TEZ-3734 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3734.01.patch > > > The configs in TezRuntimeConfiguration are not propagated if its not in > Input/Output checks, remove the checks and propagate all of > TezRuntimConfiguration to I/Os. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3734) Remove config checks in Input/Output.
[ https://issues.apache.org/jira/browse/TEZ-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3734: --- Description: The configs in TezRuntimeConfiguration are not propagated if its not in Input/Output checks, remove the checks and propagate all of TezRuntimConfiguration to I/Os. (was: The config TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT is not propagated because its not white listed in UnorderedKVInput.) > Remove config checks in Input/Output. > - > > Key: TEZ-3734 > URL: https://issues.apache.org/jira/browse/TEZ-3734 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > > The configs in TezRuntimeConfiguration are not propagated if its not in > Input/Output checks, remove the checks and propagate all of > TezRuntimConfiguration to I/Os. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3734) Remove config checks in Input/Output.
[ https://issues.apache.org/jira/browse/TEZ-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3734: --- Summary: Remove config checks in Input/Output. (was: Add TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT to UnorderedKVInput.) > Remove config checks in Input/Output. > - > > Key: TEZ-3734 > URL: https://issues.apache.org/jira/browse/TEZ-3734 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > > The config TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT is > not propagated because its not white listed in UnorderedKVInput. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TEZ-3734) Add TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT to UnorderedKVInput.
Harish Jaiprakash created TEZ-3734: -- Summary: Add TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT to UnorderedKVInput. Key: TEZ-3734 URL: https://issues.apache.org/jira/browse/TEZ-3734 Project: Apache Tez Issue Type: Bug Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash The config TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT is not propagated because its not white listed in UnorderedKVInput. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3007) Use AppFinalState.ENDED when unregistering with the RM in session mode
[ https://issues.apache.org/jira/browse/TEZ-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019661#comment-16019661 ] Harish Jaiprakash commented on TEZ-3007: Test case failure is unrelated. > Use AppFinalState.ENDED when unregistering with the RM in session mode > -- > > Key: TEZ-3007 > URL: https://issues.apache.org/jira/browse/TEZ-3007 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Harish Jaiprakash > Attachments: TEZ-3007.01.patch, TEZ-3007.02.patch, TEZ-3007.03.patch > > > YARN-4207 adds appFinalState.ENDED. When running in session mode - Tez should > make use of this state instead of setting success or failure - since the app > final state does not indicate anything about the potentially multiple queries > which executed in the app. > ShimSupport added earlier should facilitate this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3691) Setup fetchers to use shared executor.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Attachment: TEZ-3691.08.patch Adding config to Ordered and Unordered GroupedKVInput whitelists. > Setup fetchers to use shared executor. > -- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3691.01.patch, TEZ-3691.02.patch, TEZ-3691.03.patch, > TEZ-3691.04.patch, TEZ-3691.05.patch, TEZ-3691.06.patch, TEZ-3691.07.patch, > TEZ-3691.08.patch > > > Change fetchers in ShuffleScheduler & ShuffleManger to use the shared > executor service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3617) TestHistoryParser#testParserWithSuccessfulJob fails intermittently
[ https://issues.apache.org/jira/browse/TEZ-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019201#comment-16019201 ] Harish Jaiprakash commented on TEZ-3617: Hi [~skanekar], tried looking at this issue. The error is not in ATSImportTool. The following error from log implies timeline server is not able to serve request, but the timeline server has started. org.apache.hadoop.yarn.webapp.WebAppException: /v1/timeline/TEZ_APPLICATION/tez_application_1487131484149_0001: controller for v1 not found in org.apache.hadoop.yarn.webapp.Router.resolveDefault(Router.java:232) That should not happen, since path is registered programatically not config driven. The logs do show that a server was started at 8188. So completely confused as to how this can happen. > TestHistoryParser#testParserWithSuccessfulJob fails intermittently > -- > > Key: TEZ-3617 > URL: https://issues.apache.org/jira/browse/TEZ-3617 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.0 > Environment: Ubuntu 14.04 >Reporter: Sonia Garudi > Labels: ppc64le, x86 > Attachments: org.apache.tez.history.TestHistoryParser-output.txt > > > The TestHistoryParser#testParserWithSuccessfulJob test fails intermittently > in tez-history-parser project. > Error message : > testParserWithSuccessfulJob(org.apache.tez.history.TestHistoryParser) Time > elapsed: 29.952 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.tez.history.TestHistoryParser.verifyJobSpecificInfo(TestHistoryParser.java:266) > at > org.apache.tez.history.TestHistoryParser.testParserWithSuccessfulJob(TestHistoryParser.java:212) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3701) UnorderedPartitionedKVWriter - issues with parallel Deflater usage, synchronousqueue in threadpool
[ https://issues.apache.org/jira/browse/TEZ-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015210#comment-16015210 ] Harish Jaiprakash commented on TEZ-3701: Thanks [~rajesh.balamohan]. +1 for the change. > UnorderedPartitionedKVWriter - issues with parallel Deflater usage, > synchronousqueue in threadpool > -- > > Key: TEZ-3701 > URL: https://issues.apache.org/jira/browse/TEZ-3701 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.0 >Reporter: Harish Jaiprakash >Assignee: Rajesh Balamohan >Priority: Blocker > Attachments: TEZ-3701.2.patch, TEZ-3701.3.patch, TEZ-3701.4.patch, > TEZ-3701.5.patch > > > UnorderedPartitionedKVWriter add task to the executor, but does not wait for > them to finish before starting the final merge. This can cause finalMerge to > fail or write incorrect data. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3701) UnorderedPartitionedKVWriter - issues with parallel Deflater usage, synchronousqueue in threadpool
[ https://issues.apache.org/jira/browse/TEZ-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013767#comment-16013767 ] Harish Jaiprakash commented on TEZ-3701: Thanks [~rajesh.balamohan]. There is a race b/w SpillCallable availableSlots.release() vs availableSlots.acquire()/availableSlots.tryAcquire() in scheduleSpill. This happens because the future runs in the same thread of the executor service and will take a small amount of time after release to free the thread. We can get rejected exception in those cases. It would be nice to wrap only availableSlots.acquire() in a try catch and return early. This is just to ensure that in future nothing inside it will throw interrupted exception and forget to release availableSlots. > UnorderedPartitionedKVWriter - issues with parallel Deflater usage, > synchronousqueue in threadpool > -- > > Key: TEZ-3701 > URL: https://issues.apache.org/jira/browse/TEZ-3701 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.0 >Reporter: Harish Jaiprakash >Assignee: Rajesh Balamohan >Priority: Blocker > Attachments: TEZ-3701.2.patch, TEZ-3701.3.patch, TEZ-3701.4.patch > > > UnorderedPartitionedKVWriter add task to the executor, but does not wait for > them to finish before starting the final merge. This can cause finalMerge to > fail or write incorrect data. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3007) Use AppFinalState.ENDED when unregistering with the RM in session mode
[ https://issues.apache.org/jira/browse/TEZ-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3007: --- Attachment: TEZ-3007.03.patch Adding copyright to the new test file. > Use AppFinalState.ENDED when unregistering with the RM in session mode > -- > > Key: TEZ-3007 > URL: https://issues.apache.org/jira/browse/TEZ-3007 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Harish Jaiprakash > Attachments: TEZ-3007.01.patch, TEZ-3007.02.patch, TEZ-3007.03.patch > > > YARN-4207 adds appFinalState.ENDED. When running in session mode - Tez should > make use of this state instead of setting success or failure - since the app > final state does not indicate anything about the potentially multiple queries > which executed in the app. > ShimSupport added earlier should facilitate this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3691) Setup fetchers to use shared executor.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Attachment: TEZ-3691.07.patch Fixed test case failure. > Setup fetchers to use shared executor. > -- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3691.01.patch, TEZ-3691.02.patch, TEZ-3691.03.patch, > TEZ-3691.04.patch, TEZ-3691.05.patch, TEZ-3691.06.patch, TEZ-3691.07.patch > > > Change fetchers in ShuffleScheduler & ShuffleManger to use the shared > executor service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3007) Use AppFinalState.ENDED when unregistering with the RM in session mode
[ https://issues.apache.org/jira/browse/TEZ-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3007: --- Attachment: TEZ-3007.02.patch Thanks [~sseth] * Addressed comments. * Adding a test case. > Use AppFinalState.ENDED when unregistering with the RM in session mode > -- > > Key: TEZ-3007 > URL: https://issues.apache.org/jira/browse/TEZ-3007 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Harish Jaiprakash > Attachments: TEZ-3007.01.patch, TEZ-3007.02.patch > > > YARN-4207 adds appFinalState.ENDED. When running in session mode - Tez should > make use of this state instead of setting success or failure - since the app > final state does not indicate anything about the potentially multiple queries > which executed in the app. > ShimSupport added earlier should facilitate this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3714) Tez UI: Hive Queries page: Use Dag ID and App ID if they are published form Hive side.
[ https://issues.apache.org/jira/browse/TEZ-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16007849#comment-16007849 ] Harish Jaiprakash commented on TEZ-3714: +1, Thanks [~Sreenath]. > Tez UI: Hive Queries page: Use Dag ID and App ID if they are published form > Hive side. > -- > > Key: TEZ-3714 > URL: https://issues.apache.org/jira/browse/TEZ-3714 > Project: Apache Tez > Issue Type: Bug >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Attachments: TEZ-3714.1.patch > > > As of now Hive Queries pages does a reverse lookup to display DAG and App > IDs. i.e. For each query, the UI looks for DAGs where callerId = hive query > id. > This change prevents this ATS fetch when the data is available. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3717) tez-yarn-timeline-history-with-fs does not build with hadoop-2.8.
[ https://issues.apache.org/jira/browse/TEZ-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3717: --- Summary: tez-yarn-timeline-history-with-fs does not build with hadoop-2.8. (was: Master does not compile with -Phadoop28.) > tez-yarn-timeline-history-with-fs does not build with hadoop-2.8. > - > > Key: TEZ-3717 > URL: https://issues.apache.org/jira/browse/TEZ-3717 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3717.01.patch > > > Tez build with -Phadoop28 fails with DFSConfigKeys class not found. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3717) Master does not compile with -Phadoop28.
[ https://issues.apache.org/jira/browse/TEZ-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003895#comment-16003895 ] Harish Jaiprakash commented on TEZ-3717: Thanks [~sseth]. Sorry for the duplicate bug, this change is to build tez master against hadoop master (hadoop-2.8.2-SNAPSHOT). The build did go through. I tried building against branch 3.0.0-alpha2 of hadoop. It compiles, have not tried running tests on either of the branch. > Master does not compile with -Phadoop28. > > > Key: TEZ-3717 > URL: https://issues.apache.org/jira/browse/TEZ-3717 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3717.01.patch > > > Tez build with -Phadoop28 fails with DFSConfigKeys class not found. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3691) Setup fetchers to use shared executor.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Attachment: TEZ-3691.06.patch Created a config enable and disable shared thread pools for fetcher. > Setup fetchers to use shared executor. > -- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3691.01.patch, TEZ-3691.02.patch, TEZ-3691.03.patch, > TEZ-3691.04.patch, TEZ-3691.05.patch, TEZ-3691.06.patch > > > Change fetchers in ShuffleScheduler & ShuffleManger to use the shared > executor service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3007) Use AppFinalState.ENDED when unregistering with the RM in session mode
[ https://issues.apache.org/jira/browse/TEZ-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3007: --- Attachment: TEZ-3007.01.patch Adding a new method in hadoop shim to fix the computed state. > Use AppFinalState.ENDED when unregistering with the RM in session mode > -- > > Key: TEZ-3007 > URL: https://issues.apache.org/jira/browse/TEZ-3007 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Harish Jaiprakash > Attachments: TEZ-3007.01.patch > > > YARN-4207 adds appFinalState.ENDED. When running in session mode - Tez should > make use of this state instead of setting success or failure - since the app > final state does not indicate anything about the potentially multiple queries > which executed in the app. > ShimSupport added earlier should facilitate this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3717) Master does not compile with -Phadoop28.
[ https://issues.apache.org/jira/browse/TEZ-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3717: --- Attachment: TEZ-3717.01.patch Adding hdfs-client as explicit test dependency for tez-yarn-timeline-history-with-fs. > Master does not compile with -Phadoop28. > > > Key: TEZ-3717 > URL: https://issues.apache.org/jira/browse/TEZ-3717 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3717.01.patch > > > Tez build with -Phadoop28 fails with DFSConfigKeys class not found. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (TEZ-3717) Master does not compile with -Phadoop28.
Harish Jaiprakash created TEZ-3717: -- Summary: Master does not compile with -Phadoop28. Key: TEZ-3717 URL: https://issues.apache.org/jira/browse/TEZ-3717 Project: Apache Tez Issue Type: Bug Reporter: Harish Jaiprakash Assignee: Harish Jaiprakash Tez build with -Phadoop28 fails with DFSConfigKeys class not found. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (TEZ-3247) Tez does not compile against hadoop-trunk
[ https://issues.apache.org/jira/browse/TEZ-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash resolved TEZ-3247. Resolution: Duplicate All hadoop-3.0 migration issues are tracked by jira: TEZ-3252. > Tez does not compile against hadoop-trunk > -- > > Key: TEZ-3247 > URL: https://issues.apache.org/jira/browse/TEZ-3247 > Project: Apache Tez > Issue Type: Bug >Reporter: Hitesh Shah >Assignee: Harish Jaiprakash > Attachments: 3247.patch, TEZ-3247.1.patch, TEZ-3247.2.patch > > > Some of the errors: > Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) > on project tez-runtime-library: Compilation failure: Compilation failure: > [ERROR] > /Users/hitesh/dev/apache/tez/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/ShuffleClientMetrics.java:[21,33] > package org.apache.hadoop.metrics does not exist > [ERROR] > /Users/hitesh/dev/apache/tez/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/ShuffleClientMetrics.java:[22,33] > package org.apache.hadoop.metrics does not exist > [ERROR] > /Users/hitesh/dev/apache/tez/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/ShuffleClientMetrics.java:[23,33] > package org.apache.hadoop.metrics does not exist > [ERROR] > /Users/hitesh/dev/apache/tez/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/ShuffleClientMetrics.java:[24,33] > package org.apache.hadoop.metrics does not exist > [ERROR] > /Users/hitesh/dev/apache/tez/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/ShuffleClientMetrics.java:[30,39] > cannot find symbol > [ERROR] symbol: class Updater > [ERROR] > /Users/hitesh/dev/apache/tez/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/ShuffleClientMetrics.java:[32,11] > cannot find symbol > [ERROR] symbol: class MetricsRecord > [ERROR] location: class > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.ShuffleClientMetrics > [ERROR] > /Users/hitesh/dev/apache/tez/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/orderedgrouped/ShuffleClientMetrics.java:[73,25] > cannot find symbol > [ERROR] symbol: class MetricsContext > Some of these classes are already considered deprecated in hadoop-2. We > should look to moving to supported and public classes that are available in > both hadoop-2 and hadoop-3. > Reported offline by [~harishjp] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3691) Setup fetchers to use shared executor.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997703#comment-15997703 ] Harish Jaiprakash commented on TEZ-3691: I did realize that, but I thought we'll have too many configs that way as we port more components to use the framework executors. Not sure if this is good to have as fine grained or global level config. > Setup fetchers to use shared executor. > -- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3691.01.patch, TEZ-3691.02.patch, TEZ-3691.03.patch, > TEZ-3691.04.patch, TEZ-3691.05.patch > > > Change fetchers in ShuffleScheduler & ShuffleManger to use the shared > executor service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3007) Use AppFinalState.ENDED when unregistering with the RM in session mode
[ https://issues.apache.org/jira/browse/TEZ-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996461#comment-15996461 ] Harish Jaiprakash commented on TEZ-3007: The state ENDED is not available in hadoop 2.7 branch. Which would mean extending Shim to provide this, but the shim cannot access DAGAppMasterState since its part of tez-dag and tez-dag already depends on hadoop-shim, so I'll not be able to pass DAGAppMasterState. Shall I just blindly map, previous FinalApplicationStatus.FAILED and FinalApplicationStatus.SUCCEEDED to FinalApplicationStatus.ENDED in the Hadoop28 shim? > Use AppFinalState.ENDED when unregistering with the RM in session mode > -- > > Key: TEZ-3007 > URL: https://issues.apache.org/jira/browse/TEZ-3007 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Harish Jaiprakash > > YARN-4207 adds appFinalState.ENDED. When running in session mode - Tez should > make use of this state instead of setting success or failure - since the app > final state does not indicate anything about the potentially multiple queries > which executed in the app. > ShimSupport added earlier should facilitate this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (TEZ-3007) Use AppFinalState.ENDED when unregistering with the RM in session mode
[ https://issues.apache.org/jira/browse/TEZ-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned TEZ-3007: -- Assignee: Harish Jaiprakash > Use AppFinalState.ENDED when unregistering with the RM in session mode > -- > > Key: TEZ-3007 > URL: https://issues.apache.org/jira/browse/TEZ-3007 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Harish Jaiprakash > > YARN-4207 adds appFinalState.ENDED. When running in session mode - Tez should > make use of this state instead of setting success or failure - since the app > final state does not indicate anything about the potentially multiple queries > which executed in the app. > ShimSupport added earlier should facilitate this. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3691) Setup fetchers to use shared executor.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Attachment: TEZ-3691.05.patch Fixed test case failure. > Setup fetchers to use shared executor. > -- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3691.01.patch, TEZ-3691.02.patch, TEZ-3691.03.patch, > TEZ-3691.04.patch, TEZ-3691.05.patch > > > Change fetchers in ShuffleScheduler & ShuffleManger to use the shared > executor service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3673) Allocate smaller buffers in UnorderedPartitionedKVWriter
[ https://issues.apache.org/jira/browse/TEZ-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3673: --- Attachment: TEZ-3673.05.patch Using remaining memory as last buffer when leftover size > 50% of buffer limit. > Allocate smaller buffers in UnorderedPartitionedKVWriter > > > Key: TEZ-3673 > URL: https://issues.apache.org/jira/browse/TEZ-3673 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3673.01.patch, TEZ-3673.02.patch, TEZ-3673.03.patch, > TEZ-3673.04.patch, TEZ-3673.05.patch > > > UnorderedPartitionedKVWriter allocates in bigger chunks. It may or may not > get filled up. In PipelinedSorter, we start off with 32MB chunks. But > UnorderedPartitionedKVWriter can be worse as it allocates bigger blocks. Need > to revisit this allocation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3691) Setup fetchers to use shared executor.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Attachment: TEZ-3691.04.patch Fixed merge conflicts - rebased to master. > Setup fetchers to use shared executor. > -- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3691.01.patch, TEZ-3691.02.patch, TEZ-3691.03.patch, > TEZ-3691.04.patch > > > Change fetchers in ShuffleScheduler & ShuffleManger to use the shared > executor service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3691) Setup fetchers to use shared executor.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Attachment: TEZ-3691.03.patch * Adding configuration to enable use of shared executor service. * Moving ShuffleManager to use shared executor service. > Setup fetchers to use shared executor. > -- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3691.01.patch, TEZ-3691.02.patch, TEZ-3691.03.patch > > > Change fetchers in ShuffleScheduler & ShuffleManger to use the shared > executor service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3691) Setup fetchers to use shared executor.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Summary: Setup fetchers to use shared executor. (was: Move ShuffleScheduler to use the framework ExecutorService.) > Setup fetchers to use shared executor. > -- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3691.01.patch, TEZ-3691.02.patch > > > Change ShuffleScheduler to use the shard executor service, so that threads > can be shared b/w tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3691) Setup fetchers to use shared executor.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Description: Change fetchers in ShuffleScheduler & ShuffleManger to use the shared executor service. (was: Change ShuffleScheduler to use the shard executor service, so that threads can be shared b/w tasks.) > Setup fetchers to use shared executor. > -- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3691.01.patch, TEZ-3691.02.patch > > > Change fetchers in ShuffleScheduler & ShuffleManger to use the shared > executor service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3673) Allocate smaller buffers in UnorderedPartitionedKVWriter
[ https://issues.apache.org/jira/browse/TEZ-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996124#comment-15996124 ] Harish Jaiprakash commented on TEZ-3673: I did this because if we are going to use region size as buffer limit and when remaining memory > 50% of buffer limit then anyways the entire region will be allocated (if successful) so might as well use it. > Allocate smaller buffers in UnorderedPartitionedKVWriter > > > Key: TEZ-3673 > URL: https://issues.apache.org/jira/browse/TEZ-3673 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3673.01.patch, TEZ-3673.02.patch, TEZ-3673.03.patch, > TEZ-3673.04.patch > > > UnorderedPartitionedKVWriter allocates in bigger chunks. It may or may not > get filled up. In PipelinedSorter, we start off with 32MB chunks. But > UnorderedPartitionedKVWriter can be worse as it allocates bigger blocks. Need > to revisit this allocation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3707) TezSharedExecutor race condition in awaitTermination vs isTerminated.
[ https://issues.apache.org/jira/browse/TEZ-3707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992574#comment-15992574 ] Harish Jaiprakash commented on TEZ-3707: Test case failures unrelated to this patch. > TezSharedExecutor race condition in awaitTermination vs isTerminated. > - > > Key: TEZ-3707 > URL: https://issues.apache.org/jira/browse/TEZ-3707 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3707.01.patch > > > awaitTermination finishes if all the futures are resolved. isTerminated > returns true if futures is empty. future done is invoked after the future is > resolve, which can lead to awaitTermination finishing but isTerminated is > false until done is invoked. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3691) Move ShuffleScheduler to use the framework ExecutorService.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Attachment: TEZ-3691.02.patch Fixed test case failures. > Move ShuffleScheduler to use the framework ExecutorService. > --- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3691.01.patch, TEZ-3691.02.patch > > > Change ShuffleScheduler to use the shard executor service, so that threads > can be shared b/w tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (TEZ-3691) Move ShuffleScheduler to use the framework ExecutorService.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned TEZ-3691: -- Assignee: Harish Jaiprakash > Move ShuffleScheduler to use the framework ExecutorService. > --- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3691.01.patch > > > Change ShuffleScheduler to use the shard executor service, so that threads > can be shared b/w tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3691) Move ShuffleScheduler to use the framework ExecutorService.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Attachment: TEZ-3691.01.patch Moved ShuffleScheduler to use shared executor service. > Move ShuffleScheduler to use the framework ExecutorService. > --- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash > Attachments: TEZ-3691.01.patch > > > Change ShuffleScheduler to use the shard executor service, so that threads > can be shared b/w tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3691) Move ShuffleScheduler to use the framework ExecutorService.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Description: Change ShuffleScheduler to use the shard executor service, so that threads can be shared b/w tasks. (was: Change pipeline sorter to use the shard executor service, so that threads can be shared b/w tasks.) > Move ShuffleScheduler to use the framework ExecutorService. > --- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash > > Change ShuffleScheduler to use the shard executor service, so that threads > can be shared b/w tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3691) Move ShuffleScheduler to use the framework ExecutorService.
[ https://issues.apache.org/jira/browse/TEZ-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3691: --- Summary: Move ShuffleScheduler to use the framework ExecutorService. (was: Move pipelined sorter to use the framework ExecutorService.) > Move ShuffleScheduler to use the framework ExecutorService. > --- > > Key: TEZ-3691 > URL: https://issues.apache.org/jira/browse/TEZ-3691 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Harish Jaiprakash > > Change pipeline sorter to use the shard executor service, so that threads can > be shared b/w tasks. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (TEZ-3673) Allocate smaller buffers in UnorderedPartitionedKVWriter
[ https://issues.apache.org/jira/browse/TEZ-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991949#comment-15991949 ] Harish Jaiprakash edited comment on TEZ-3673 at 5/2/17 6:41 AM: [~sseth] Please take a look. Couple of questions for computeNumBuffersAndSize: * Should I subtract 64 bytes in all sizePerBuffer assignments? Currently only if sizePerBuffer is set because of bufferLimit parameter it does the correction. Assuming that bufferLimit was set to be some multiple of region size. Reason being that any other value might not be at region size boundary and correction may not be helpful. * There is no logging or exception for wrong buffer merge percent, it silently corrects to be within limit. Is that alright? was (Author: harishjp): [~seth] Please take a look. Couple of questions for computeNumBuffersAndSize: * Should I subtract 64 bytes in all sizePerBuffer assignments? Currently only if sizePerBuffer is set because of bufferLimit parameter it does the correction. Assuming that bufferLimit was set to be some multiple of region size. Reason being that any other value might not be at region size boundary and correction may not be helpful. * There is no logging or exception for wrong buffer merge percent, it silently corrects to be within limit. Is that alright? > Allocate smaller buffers in UnorderedPartitionedKVWriter > > > Key: TEZ-3673 > URL: https://issues.apache.org/jira/browse/TEZ-3673 > Project: Apache Tez > Issue Type: Improvement >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: TEZ-3673.01.patch, TEZ-3673.02.patch, TEZ-3673.03.patch, > TEZ-3673.04.patch > > > UnorderedPartitionedKVWriter allocates in bigger chunks. It may or may not > get filled up. In PipelinedSorter, we start off with 32MB chunks. But > UnorderedPartitionedKVWriter can be worse as it allocates bigger blocks. Need > to revisit this allocation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TEZ-3700) Consumer attempt should kill itself instead of failing during validation checks with final merge avoidance
[ https://issues.apache.org/jira/browse/TEZ-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992369#comment-15992369 ] Harish Jaiprakash commented on TEZ-3700: Thanks [~rajesh.balamohan]. +1 for the patch. > Consumer attempt should kill itself instead of failing during validation > checks with final merge avoidance > -- > > Key: TEZ-3700 > URL: https://issues.apache.org/jira/browse/TEZ-3700 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan > Attachments: TEZ-3700.1.patch, TEZ-3700.2.patch, TEZ-3700.3.patch, > TEZ-3700.4.patch, TEZ-3700.5.patch > > > Currently when if data is received from different attempts with final merge > disabled (with/without pipleining), consumer attempt ends up with failure. > Instead it should issue kill request so that the job deos not end up with > failures. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (TEZ-3707) TezSharedExecutor race condition in awaitTermination vs isTerminated.
[ https://issues.apache.org/jira/browse/TEZ-3707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated TEZ-3707: --- Attachment: TEZ-3707.01.patch Fixed isTerminated to check if pending futures are already done. No extra tests, tested manually by putting sleep in the done method and verifying that isTerminated returns true. > TezSharedExecutor race condition in awaitTermination vs isTerminated. > - > > Key: TEZ-3707 > URL: https://issues.apache.org/jira/browse/TEZ-3707 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash > Attachments: TEZ-3707.01.patch > > > awaitTermination finishes if all the futures are resolved. isTerminated > returns true if futures is empty. future done is invoked after the future is > resolve, which can lead to awaitTermination finishing but isTerminated is > false until done is invoked. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
