[jira] [Commented] (TEZ-4300) Download protoc automatically compile/development time
[
https://issues.apache.org/jira/browse/TEZ-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470192#comment-17470192
]
Tez CI commented on TEZ-4300:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
12s{color} | {color:green} master passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
54s{color} | {color:green} master passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
8s{color} | {color:green} master passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} master passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
12s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
54s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
53s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
28s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
5s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} tez-protobuf-history-plugin in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 72m
36s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
[GitHub] [tez] tez-yetus commented on pull request #115: TEZ-4300: Download protoc automatically compile/development time
tez-yetus commented on pull request #115: URL: https://github.com/apache/tez/pull/115#issuecomment-1006947440 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 16m 17s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 5m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 8m 56s | master passed | | +1 :green_heart: | compile | 3m 12s | master passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | compile | 2m 54s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javadoc | 3m 8s | master passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 2m 29s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 38s | the patch passed | | +1 :green_heart: | compile | 3m 12s | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javac | 3m 12s | the patch passed | | +1 :green_heart: | compile | 2m 54s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | +1 :green_heart: | javac | 2m 54s | the patch passed | | +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 3s | The patch has no ill-formed XML file. | | +1 :green_heart: | javadoc | 2m 53s | the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 | | +1 :green_heart: | javadoc | 2m 28s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | ||| _ Other Tests _ | | +1 :green_heart: | unit | 2m 5s | tez-api in the patch passed. | | +1 :green_heart: | unit | 0m 31s | tez-protobuf-history-plugin in the patch passed. | | +1 :green_heart: | unit | 72m 36s | root in the patch passed. | | +1 :green_heart: | asflicense | 1m 8s | The patch does not generate ASF License warnings. | | | | 136m 24s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-115/9/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/tez/pull/115 | | JIRA Issue | TEZ-4300 | | Optional Tests | dupname asflicense shellcheck shelldocs javac javadoc unit xml compile | | uname | Linux 75aebc9c6c62 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/tez.sh | | git revision | master / 227c75014 | | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | | Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-115/9/testReport/ | | Max. process+thread count | 1387 (vs. ulimit of 5500) | | modules | C: tez-api tez-plugins/tez-protobuf-history-plugin . U: . | | Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-115/9/console | | versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (TEZ-3369) Add APIs in DAGClient to expose dag-structure level info, task and task attempt progress
[
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470191#comment-17470191
]
Chris Wensel commented on TEZ-3369:
---
first order, yes, would love to see any Cascading+Tez api fixes that allow
modern Cascading to compile/run on modern Tez.
second order, but less important, access to the Tez runtime metrics is always a
plus for those who want to build in monitoring of Tez from the Cascading/Tez
client side (via FlowStats apis).
This may mean disabling Cascading tests that introspect Tez runtime
metrics/telemetry. I'm fine with that if it gets us a 4.5 release.
> Add APIs in DAGClient to expose dag-structure level info, task and task
> attempt progress
>
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
>Priority: Major
>
> Hi,
> We seem to be running into issues when we try to use the newest version of
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
> at
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient
> |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit |
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
> (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>
> As there's no good way to get the vertexID / vertexChildren / vertexChild
> (correct me if I'm wrong), they end up extending from the
> DAGClientTimelineImpl which has the http client and json parsing code to
> allow [things like this |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
> public String getVertexID( String vertexName ) throws IOException,
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format =
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID,
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That
> way the cascading code wouldn't need updating if the uri changed / json
> format changed, it would end up being updated in these clients as well. I
> suspect adding this to DAGClient would require more work as it'll also need
> to be supported by the RPCClient and I don't think there are the relevant
> protos and such available.
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient
> (currently it just implements Closeable). This will not require any changes
> on the Cascading side as DAGClientTimelineImpl will continue to be a
> DAGClient.
> Full stack trace:
> {code}
> Exception in thread "flow
> com.twitter.data_platform.e2e_testing.jobs.parquet.E2ETestConvertThriftToParquet"
> java.lang.ClassCastException:
[jira] [Commented] (TEZ-3369) Add APIs in DAGClient to expose dag-structure level info, task and task attempt progress
[
https://issues.apache.org/jira/browse/TEZ-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470190#comment-17470190
]
Daniel Sudzilouski commented on TEZ-3369:
-
[~hitesh] [~cwensel] [~sseth] Sorry for bringing up this old thread, but I'm
wondering if there is still interest in a polished version of these proposed
changes.
I was looking at upgrading Cascading to hadoop3 and dependency collisions in
the pinned version of TEZ became a problem. I'm assuming Cascading would still
like to maintain feature parity with TEZ as a backend ([~cwensel]).
I'm interested in figuring out the context to make this happen, but I wanted to
first touch base to see if there's been any new developments that I might be
missing.
> Add APIs in DAGClient to expose dag-structure level info, task and task
> attempt progress
>
>
> Key: TEZ-3369
> URL: https://issues.apache.org/jira/browse/TEZ-3369
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Piyush Narang
>Assignee: Piyush Narang
>Priority: Major
>
> Hi,
> We seem to be running into issues when we try to use the newest version of
> Tez (0.9.0-SNAPSHOT) with Cascading. The issue seems to be:
> {code}
> java.lang.ClassCastException: cascading.stats.tez.util.TezTimelineClient
> cannot be cast to org.apache.tez.dag.api.client.DAGClient
> at
> cascading.stats.tez.util.TezStatsUtil.createTimelineClient(TezStatsUtil.java:142)
> {code}
> (Full stack trace at the end)
> Relevant Cascading code is:
> 1) [Cascading tries to create a TezTimelineClient and cast it to a DAGClient
> |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezStatsUtil.java#L142]
> 2) [TezTimelineClient extends from DAGClientTimelineImpl |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L53]
> 3) [DAGClientTimelineImpl extends from DAGClientInternal |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientTimelineImpl.java#L68]
> 4) [DAGClientInternal extends Closeable which is why things break |
> https://github.com/apache/tez/blob/dacd0191b684208d71ea457ca849f2d01212bb7e/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientInternal.java#L38].
> This behavior was 'broken' in this [commit |
> https://github.com/apache/tez/commit/2af886b509015200e1c04527275474cbc771c667]
> (release 0.8.3)
> The TezTimelineClient in Cascading seems to do two things:
> 1) DAGClient functionalities - ends up delegating to the inner DAGClient
> object.
> 2) Retrieve stuff like vertexID, vertexChildren and vertexChild (from this
> [interface|https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TimelineClient.java#L31]).
>
> As there's no good way to get the vertexID / vertexChildren / vertexChild
> (correct me if I'm wrong), they end up extending from the
> DAGClientTimelineImpl which has the http client and json parsing code to
> allow [things like this |
> https://github.com/Cascading/cascading/blob/3.1/cascading-hadoop2-tez-stats/src/main/java/cascading/stats/tez/util/TezTimelineClient.java#L93]:
> {code}
> @Override
> public String getVertexID( String vertexName ) throws IOException,
> TezException
> {
> // the filter 'vertexName' is in the 'otherinfo' field, so it must be
> requested, otherwise timeline server throws
> // an NPE. to be safe, we include both fields in the result
> String format =
> "%s/%s?primaryFilter=%s:%s=vertexName:%s=%s";
> String url = String.format( format, baseUri, TEZ_VERTEX_ID, TEZ_DAG_ID,
> dagId, vertexName, FILTER_BY_FIELDS );
> JSONObject jsonRoot = getJsonRootEntity( url );
> JSONArray entitiesNode = jsonRoot.optJSONArray( ENTITIES );
> ...
> {code}
> Some options I can think of:
> 1) Ideally these methods getVertexID / getVertexChildren / getVertexChild
> would be part of DAGClient? Or even part of the DAGClientTimelineImpl? That
> way the cascading code wouldn't need updating if the uri changed / json
> format changed, it would end up being updated in these clients as well. I
> suspect adding this to DAGClient would require more work as it'll also need
> to be supported by the RPCClient and I don't think there are the relevant
> protos and such available.
> 2) A simpler fix would be to have DAGClientInternal extend DAGClient
> (currently it just implements Closeable). This will not require any changes
> on the Cascading side as DAGClientTimelineImpl will continue to be a
> DAGClient.
> Full stack trace:
> {code}
> Exception in thread "flow
>
[jira] [Updated] (TEZ-4311) Bump hadoop dependency version to 3.3.x
[ https://issues.apache.org/jira/browse/TEZ-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-4311: -- Fix Version/s: 0.10.2 > Bump hadoop dependency version to 3.3.x > --- > > Key: TEZ-4311 > URL: https://issues.apache.org/jira/browse/TEZ-4311 > Project: Apache Tez > Issue Type: Sub-task >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 0.10.2 > > Attachments: > org.apache.tez.auxservices.TestShuffleHandler-output.txt, > org.apache.tez.dag.history.ats.acls.TestATSHistoryWithACLs-output.txt > > Time Spent: 3.5h > Remaining Estimate: 0h > > This change is for bumping hadoop.version to 3.3 when every other needed > subtask is done. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (TEZ-4311) Bump hadoop dependency version to 3.3.x
[ https://issues.apache.org/jira/browse/TEZ-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor resolved TEZ-4311. --- Resolution: Fixed > Bump hadoop dependency version to 3.3.x > --- > > Key: TEZ-4311 > URL: https://issues.apache.org/jira/browse/TEZ-4311 > Project: Apache Tez > Issue Type: Sub-task >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 0.10.2 > > Attachments: > org.apache.tez.auxservices.TestShuffleHandler-output.txt, > org.apache.tez.dag.history.ats.acls.TestATSHistoryWithACLs-output.txt > > Time Spent: 3.5h > Remaining Estimate: 0h > > This change is for bumping hadoop.version to 3.3 when every other needed > subtask is done. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [tez] abstractdog merged pull request #132: TEZ-4311: Bump hadoop dependency version to 3.3.x
abstractdog merged pull request #132: URL: https://github.com/apache/tez/pull/132 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tez] abstractdog edited a comment on pull request #132: TEZ-4311: Bump hadoop dependency version to 3.3.x
abstractdog edited a comment on pull request #132: URL: https://github.com/apache/tez/pull/132#issuecomment-1006885558 thanks @jteagles, I'm about to commit this soon just double-checked, the new code can still be successfully built with the older hadoop: ``` mvn clean install -DskipTests -Dhadoop.version=3.2.2 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (TEZ-4368) Check dependency version collisions after upgrading to Hadoop 3.3.1
[ https://issues.apache.org/jira/browse/TEZ-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-4368: -- Description: Tez tries to follow the third-party dependencies in hadoop, so theoretically we need to try to harmonize this on every hadoop upgrade. Even if problematic dependency collisions might possibly bubble up during the hadoop upgrade, we can still double-check it. possible TODO: write and share a simple script that can easily show ambigous compile-time dependencies in the tree (e.g. the output of 'mvn dependency:tree' command) > Check dependency version collisions after upgrading to Hadoop 3.3.1 > --- > > Key: TEZ-4368 > URL: https://issues.apache.org/jira/browse/TEZ-4368 > Project: Apache Tez > Issue Type: Sub-task >Reporter: László Bodor >Priority: Major > > Tez tries to follow the third-party dependencies in hadoop, so theoretically > we need to try to harmonize this on every hadoop upgrade. Even if problematic > dependency collisions might possibly bubble up during the hadoop upgrade, we > can still double-check it. > possible TODO: write and share a simple script that can easily show ambigous > compile-time dependencies in the tree (e.g. the output of 'mvn > dependency:tree' command) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (TEZ-4368) Check dependency version collisions after upgrading to 3.3.1
László Bodor created TEZ-4368: - Summary: Check dependency version collisions after upgrading to 3.3.1 Key: TEZ-4368 URL: https://issues.apache.org/jira/browse/TEZ-4368 Project: Apache Tez Issue Type: Sub-task Reporter: László Bodor -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (TEZ-4368) Check dependency version collisions after upgrading to Hadoop 3.3.1
[ https://issues.apache.org/jira/browse/TEZ-4368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-4368: -- Summary: Check dependency version collisions after upgrading to Hadoop 3.3.1 (was: Check dependency version collisions after upgrading to 3.3.1) > Check dependency version collisions after upgrading to Hadoop 3.3.1 > --- > > Key: TEZ-4368 > URL: https://issues.apache.org/jira/browse/TEZ-4368 > Project: Apache Tez > Issue Type: Sub-task >Reporter: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (TEZ-4167) Speed up TestPipelinedSorter
[ https://issues.apache.org/jira/browse/TEZ-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-4167: -- Fix Version/s: 0.10.2 > Speed up TestPipelinedSorter > > > Key: TEZ-4167 > URL: https://issues.apache.org/jira/browse/TEZ-4167 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Assignee: Jonathan Turner Eagles >Priority: Major > Fix For: 0.10.2 > > Attachments: TEZ-4167.001.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [tez] abstractdog merged pull request #174: TEZ-4167. Speed up TestPipelinedSorter
abstractdog merged pull request #174: URL: https://github.com/apache/tez/pull/174 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tez] abstractdog commented on pull request #174: TEZ-4167. Speed up TestPipelinedSorter
abstractdog commented on pull request #174: URL: https://github.com/apache/tez/pull/174#issuecomment-1006900109 yes, usual to-be-ignored checkstyle problems, I'm merging this, thanks for the patch @jteagles ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tez] jteagles commented on pull request #174: TEZ-4167. Speed up TestPipelinedSorter
jteagles commented on pull request #174: URL: https://github.com/apache/tez/pull/174#issuecomment-1006898251 @abstractdog, I think this is ready to go. Couple of silly checkstyles. Let me know if this is ok. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (TEZ-4367) Implement a test that validates the contents of the tez dist package
[ https://issues.apache.org/jira/browse/TEZ-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated TEZ-4367: -- Description: Let's say given a set of expected jars in the tez dist package, which can be defined somewhere in text (golden file). The test can be in maven build lifecycle or in a separate test. If the file listing in tez tar.gz doesn't match the expected golden file, test should fail. This would make us aware of any changes that we introduce with the patches. For example the regression introduced by TEZ-4114 and fixed by TEZ-4323 > Implement a test that validates the contents of the tez dist package > > > Key: TEZ-4367 > URL: https://issues.apache.org/jira/browse/TEZ-4367 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Priority: Major > > Let's say given a set of expected jars in the tez dist package, which can be > defined somewhere in text (golden file). > The test can be in maven build lifecycle or in a separate test. If the file > listing in tez tar.gz doesn't match the expected golden file, test should > fail. This would make us aware of any changes that we introduce with the > patches. For example the regression introduced by TEZ-4114 and fixed by > TEZ-4323 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (TEZ-4367) Implement a test that validates the contents of the tez dist package
László Bodor created TEZ-4367: - Summary: Implement a test that validates the contents of the tez dist package Key: TEZ-4367 URL: https://issues.apache.org/jira/browse/TEZ-4367 Project: Apache Tez Issue Type: Improvement Reporter: László Bodor -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [tez] abstractdog commented on pull request #132: TEZ-4311: Bump hadoop dependency version to 3.3.x
abstractdog commented on pull request #132: URL: https://github.com/apache/tez/pull/132#issuecomment-1006885558 thanks @jteagles , I'm about to commit this soon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency
[ https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470159#comment-17470159 ] László Bodor edited comment on TEZ-4154 at 1/6/22, 7:42 PM: resolving this one as won't fix hadoop 3.3 upgrade is done in TEZ-4311 tez protobuf 3.x upgrade is handled in TEZ-4361 protobuf 3.x build is supported in the scope of TEZ-4300, please let me know if you need further assistance with this cc: [~mgrigorov], [~emilio.fernandes70], [~julien.faye], [~albertkohl] was (Author: abstractdog): resolving this one as won't fix hadoop 3.3 upgrade is done in TEZ-4311 tez protobuf 3.x upgrade is handled in TEZ-4361 > tez-protobuf: a single module for handling protobuf dependency > -- > > Key: TEZ-4154 > URL: https://issues.apache.org/jira/browse/TEZ-4154 > Project: Apache Tez > Issue Type: Sub-task >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, > TEZ-4154.04.patch, request.zip > > > Similarly to what I've seen in TEZ-2164, I would like to create a module that > is a single point of control of how tez handles protobuf dependency. In the > near future, when tez will upgrade to hadoop 3.3, there will be a couple of > things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf > 3.x and handle the possible mess that can come from the fact that hadoop has > relocated its protobuf. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency
[ https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470159#comment-17470159 ] László Bodor commented on TEZ-4154: --- resolving this one as won't fix hadoop 3.3 upgrade is done in TEZ-4311 tez protobuf 3.x upgrade is handled in TEZ-4361 > tez-protobuf: a single module for handling protobuf dependency > -- > > Key: TEZ-4154 > URL: https://issues.apache.org/jira/browse/TEZ-4154 > Project: Apache Tez > Issue Type: Sub-task >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, > TEZ-4154.04.patch, request.zip > > > Similarly to what I've seen in TEZ-2164, I would like to create a module that > is a single point of control of how tez handles protobuf dependency. In the > near future, when tez will upgrade to hadoop 3.3, there will be a couple of > things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf > 3.x and handle the possible mess that can come from the fact that hadoop has > relocated its protobuf. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [tez] abstractdog commented on pull request #115: TEZ-4300: Download protoc automatically compile/development time
abstractdog commented on pull request #115: URL: https://github.com/apache/tez/pull/115#issuecomment-1006855776 checked, the patch works on branch-0.9 too, [this commit](https://github.com/abstractdog/tez/commit/f0f9923d782be9e4f6a0fedaae9f6f5103f89f06) contains a fix that I added on master too, please refer to the second commit: moved the profile to root pom.xml + redefined where it was needed is this okay to go in? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [tez] abstractdog closed pull request #115: TEZ-4300: Download protoc automatically compile/development time
abstractdog closed pull request #115: URL: https://github.com/apache/tez/pull/115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@tez.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (TEZ-4349) DAGClient gets stuck with invalid cached DAGStatus
[
https://issues.apache.org/jira/browse/TEZ-4349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470140#comment-17470140
]
Ahmed Hussein commented on TEZ-4349:
Thanks [~abstractdog] for your feedback and for committing the changes!
> DAGClient gets stuck with invalid cached DAGStatus
> --
>
> Key: TEZ-4349
> URL: https://issues.apache.org/jira/browse/TEZ-4349
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 0.10.2
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> I found that some Oozie launchers get stuck waiting for the job to complete.
> After investigation I found that {{dagClient.getDAGStatus(null)}} calls the
> override {{dagClient.getDAGStatus(null, 0)}} , which then calls
> {{getDAGStatusInternal}} making use of the cachedDagStatus field.
> The cachedDagStatus is never updated causing the launcher to wait
> indefinitely.
>
> [https://github.com/apache/tez/blob/master/tez-api/src/main/java/org/apache/tez/dag/api/client/DAGClientImpl.java#L212]
> {code:java}
> if (!dagCompleted) {
> if (dagStatus != null) {
> cachedDagStatus = dagStatus;
> return dagStatus;
> }
> if (cachedDagStatus != null) {
> // could not get from AM (not reachable/ was killed). return cached
> status.
> return cachedDagStatus;
> }
> }
> {code}
> +To Fix:+
> The {{cachedDagStatus}} should be valid for a certain amount of time, or
> certain number of retires.
> When the cachedDAGStatus expires, the DAGClient tries to pull from AM or the
> RM.
> An error in fetching the status from both AM and RM, would return null to the
> caller.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
