[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2022-01-07 Thread Jira


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470454#comment-17470454
 ] 

László Bodor commented on TEZ-4154:
---

[~mgrigorov]: even TEZ-4300 will help you because it lets you define a protobuf 
version command line, however, I'm not sure if the shell script introduced 
there is prepared for ARM64, let's continue the discussion there

> tez-protobuf: a single module for handling protobuf dependency
> --
>
> Key: TEZ-4154
> URL: https://issues.apache.org/jira/browse/TEZ-4154
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, 
> TEZ-4154.04.patch, request.zip
>
>
> Similarly to what I've seen in TEZ-2164, I would like to create a module that 
> is a single point of control of how tez handles protobuf dependency. In the 
> near future, when tez will upgrade to hadoop 3.3, there will be a couple of 
> things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf 
> 3.x and handle the possible mess that can come from the fact that hadoop has 
> relocated its protobuf.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2022-01-07 Thread Martin Tzvetanov Grigorov (Jira)


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470440#comment-17470440
 ] 

Martin Tzvetanov Grigorov commented on TEZ-4154:


Happy New Year!

Thank you, [~abstractdog] !

I will try to build Tez on Linux ARM64 again! But I guess I will have to wait 
for TEZ-4363 to see successful build!

> tez-protobuf: a single module for handling protobuf dependency
> --
>
> Key: TEZ-4154
> URL: https://issues.apache.org/jira/browse/TEZ-4154
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, 
> TEZ-4154.04.patch, request.zip
>
>
> Similarly to what I've seen in TEZ-2164, I would like to create a module that 
> is a single point of control of how tez handles protobuf dependency. In the 
> near future, when tez will upgrade to hadoop 3.3, there will be a couple of 
> things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf 
> 3.x and handle the possible mess that can come from the fact that hadoop has 
> relocated its protobuf.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2022-01-06 Thread Jira


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17470159#comment-17470159
 ] 

László Bodor commented on TEZ-4154:
---

resolving this one as won't fix
hadoop 3.3 upgrade is done in TEZ-4311
tez protobuf 3.x upgrade is handled in TEZ-4361

> tez-protobuf: a single module for handling protobuf dependency
> --
>
> Key: TEZ-4154
> URL: https://issues.apache.org/jira/browse/TEZ-4154
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, 
> TEZ-4154.04.patch, request.zip
>
>
> Similarly to what I've seen in TEZ-2164, I would like to create a module that 
> is a single point of control of how tez handles protobuf dependency. In the 
> near future, when tez will upgrade to hadoop 3.3, there will be a couple of 
> things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf 
> 3.x and handle the possible mess that can come from the fact that hadoop has 
> relocated its protobuf.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2021-01-31 Thread Martin Tzvetanov Grigorov (Jira)


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276097#comment-17276097
 ] 

Martin Tzvetanov Grigorov commented on TEZ-4154:


There were quite some commits since then. I guess the patch needs to be updated.

> tez-protobuf: a single module for handling protobuf dependency
> --
>
> Key: TEZ-4154
> URL: https://issues.apache.org/jira/browse/TEZ-4154
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, 
> TEZ-4154.04.patch
>
>
> Similarly to what I've seen in TEZ-2164, I would like to create a module that 
> is a single point of control of how tez handles protobuf dependency. In the 
> near future, when tez will upgrade to hadoop 3.3, there will be a couple of 
> things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf 
> 3.x and handle the possible mess that can come from the fact that hadoop has 
> relocated its protobuf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2021-01-31 Thread albert kohl (Jira)


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276090#comment-17276090
 ] 

albert kohl commented on TEZ-4154:
--

tried that too, same thing (more or less) should i be pulling down a specific 
branch to apply to? or just the master. it looked like from your build info in 
prior comments it should have been master.

> tez-protobuf: a single module for handling protobuf dependency
> --
>
> Key: TEZ-4154
> URL: https://issues.apache.org/jira/browse/TEZ-4154
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, 
> TEZ-4154.04.patch
>
>
> Similarly to what I've seen in TEZ-2164, I would like to create a module that 
> is a single point of control of how tez handles protobuf dependency. In the 
> near future, when tez will upgrade to hadoop 3.3, there will be a couple of 
> things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf 
> 3.x and handle the possible mess that can come from the fact that hadoop has 
> relocated its protobuf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2021-01-31 Thread Martin Tzvetanov Grigorov (Jira)


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276082#comment-17276082
 ] 

Martin Tzvetanov Grigorov commented on TEZ-4154:


[~albertkohl] You need to use just 04.patch, since it is the newest.

Jira supports versioning of the patches. I.e. one can upload patches with the 
same file name and Jira will grey out the older ones. This is less confusing to 
the users.

> tez-protobuf: a single module for handling protobuf dependency
> --
>
> Key: TEZ-4154
> URL: https://issues.apache.org/jira/browse/TEZ-4154
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, 
> TEZ-4154.04.patch
>
>
> Similarly to what I've seen in TEZ-2164, I would like to create a module that 
> is a single point of control of how tez handles protobuf dependency. In the 
> near future, when tez will upgrade to hadoop 3.3, there will be a couple of 
> things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf 
> 3.x and handle the possible mess that can come from the fact that hadoop has 
> relocated its protobuf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2021-01-27 Thread Martin Tzvetanov Grigorov (Jira)


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273384#comment-17273384
 ] 

Martin Tzvetanov Grigorov commented on TEZ-4154:


Hi [~albertkohl] !

You could use `git apply -p0 file`

> tez-protobuf: a single module for handling protobuf dependency
> --
>
> Key: TEZ-4154
> URL: https://issues.apache.org/jira/browse/TEZ-4154
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, 
> TEZ-4154.04.patch
>
>
> Similarly to what I've seen in TEZ-2164, I would like to create a module that 
> is a single point of control of how tez handles protobuf dependency. In the 
> near future, when tez will upgrade to hadoop 3.3, there will be a couple of 
> things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf 
> 3.x and handle the possible mess that can come from the fact that hadoop has 
> relocated its protobuf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2021-01-15 Thread Jira


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266046#comment-17266046
 ] 

László Bodor commented on TEZ-4154:
---

thanks [~emilio.fernandes70], [~julien.faye], unfortunately, this is not an 
easy one
protobuf3 in hadoop is a backward-incompatible change as I mentioned above and 
on TEZ-4152
we need to decide which way to go

> tez-protobuf: a single module for handling protobuf dependency
> --
>
> Key: TEZ-4154
> URL: https://issues.apache.org/jira/browse/TEZ-4154
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, 
> TEZ-4154.04.patch
>
>
> Similarly to what I've seen in TEZ-2164, I would like to create a module that 
> is a single point of control of how tez handles protobuf dependency. In the 
> near future, when tez will upgrade to hadoop 3.3, there will be a couple of 
> things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf 
> 3.x and handle the possible mess that can come from the fact that hadoop has 
> relocated its protobuf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2021-01-11 Thread Julien Faye (Jira)


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262656#comment-17262656
 ] 

Julien Faye commented on TEZ-4154:
--

Any progress here ?

What is needed to merge the proposed patches ?

I'd like to be able to build and package on ARM64 machines.

> tez-protobuf: a single module for handling protobuf dependency
> --
>
> Key: TEZ-4154
> URL: https://issues.apache.org/jira/browse/TEZ-4154
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, 
> TEZ-4154.04.patch
>
>
> Similarly to what I've seen in TEZ-2164, I would like to create a module that 
> is a single point of control of how tez handles protobuf dependency. In the 
> near future, when tez will upgrade to hadoop 3.3, there will be a couple of 
> things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf 
> 3.x and handle the possible mess that can come from the fact that hadoop has 
> relocated its protobuf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2020-11-23 Thread Emilio Fernandes (Jira)


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237342#comment-17237342
 ] 

Emilio Fernandes commented on TEZ-4154:
---

+1 for the suggested changes!

I hope they will be applied soon!

> tez-protobuf: a single module for handling protobuf dependency
> --
>
> Key: TEZ-4154
> URL: https://issues.apache.org/jira/browse/TEZ-4154
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, 
> TEZ-4154.04.patch
>
>
> Similarly to what I've seen in TEZ-2164, I would like to create a module that 
> is a single point of control of how tez handles protobuf dependency. In the 
> near future, when tez will upgrade to hadoop 3.3, there will be a couple of 
> things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf 
> 3.x and handle the possible mess that can come from the fact that hadoop has 
> relocated its protobuf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2020-11-10 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17229132#comment-17229132
 ] 

Hadoop QA commented on TEZ-4154:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
 3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  4m 
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  5m 
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  4m 
55s{color} | {color:blue} Used deprecated FindBugs config; considering 
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
51s{color} | {color:red} root in master has 2 extant findbugs warnings. {color} 
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 15s{color} | {color:orange} tez-protobuf: The patch generated 1 new + 0 
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 40s{color} | {color:orange} root: The patch generated 1 new + 43 unchanged - 
0 fixed = 44 total (was 43) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m 
17s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  5m 
13s{color} | {color:red} root generated 1 new + 2 unchanged - 0 fixed = 3 total 
(was 2) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
24s{color} | {color:green} tez-protobuf in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 21s{color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
28s{color} | {color:green} hadoop-shim-2.8 in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m  
4s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
38s{color} | {color:green} tez-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 
26s{color} | {color:green} tez-dag in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m  
4s{color} | {color:green} tez-ext-service-tests in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
22s{color} | {color:green} tez-mapreduce in the patch passed. {color} |
| {color:gre

[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2020-11-10 Thread Jira


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17229089#comment-17229089
 ] 

László Bodor commented on TEZ-4154:
---

thanks [~mgrigorov]
as a matter of fact, this jira only a project reorganization, the actual 
problem is described in TEZ-4152:
https://issues.apache.org/jira/browse/TEZ-4152?focusedCommentId=17090713&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17090713

seems like hadoop refactored protobuf in a way that we can only follow with a 
breaking change, or at least, I haven't found a better solution
now as Tez 0.10 is released, I'm wondering if we can simply do a breaking 
change, upgrade to hadoop3.3 and protobuf3
in this case, we'll need a branch-0.10 for current hadoop 3.1 compatibility and 
go on with the breaking changes on tez master
cc: [~jeagles] please advise here

> tez-protobuf: a single module for handling protobuf dependency
> --
>
> Key: TEZ-4154
> URL: https://issues.apache.org/jira/browse/TEZ-4154
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, 
> TEZ-4154.04.patch
>
>
> Similarly to what I've seen in TEZ-2164, I would like to create a module that 
> is a single point of control of how tez handles protobuf dependency. In the 
> near future, when tez will upgrade to hadoop 3.3, there will be a couple of 
> things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf 
> 3.x and handle the possible mess that can come from the fact that hadoop has 
> relocated its protobuf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2020-11-10 Thread Martin Tzvetanov Grigorov (Jira)


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17229046#comment-17229046
 ] 

Martin Tzvetanov Grigorov commented on TEZ-4154:


I am trying to build Tez on ARM64 architecture and the usage of Protobuf  2.5 
is one of the stoppers.

Upgrading to a 3.x (3.13.0) solves the issue!

Is there anything I can help with to get these patches merged  ?

> tez-protobuf: a single module for handling protobuf dependency
> --
>
> Key: TEZ-4154
> URL: https://issues.apache.org/jira/browse/TEZ-4154
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, 
> TEZ-4154.04.patch
>
>
> Similarly to what I've seen in TEZ-2164, I would like to create a module that 
> is a single point of control of how tez handles protobuf dependency. In the 
> near future, when tez will upgrade to hadoop 3.3, there will be a couple of 
> things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf 
> 3.x and handle the possible mess that can come from the fact that hadoop has 
> relocated its protobuf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2020-04-27 Thread Jira


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093670#comment-17093670
 ] 

László Bodor commented on TEZ-4154:
---

TestSpeculation is failed which is not related
could you please take a look at  [^TEZ-4154.04.patch]  [~ashutoshc], 
[~jeagles], [~rizhang]?

this patch is made as a preparation for handling hadoop 3.3 upgrade, the main 
goal is to abstract out everything which I bumped into while compiling tez on 
hadoop-3.4 snapshot in TEZ-4152

the patch basically:
1. introduces a new module, tez-protobuf, which encloses protobuf inclusion 
into tez project

2. defines maven replacer plugin in order modify source generation, using a new 
GeneratedMessage class, which extends protobuf's one --> my plan is that in 
case of a hadoop 3.3 upgrade, this GeneratedMessage inheritance can be easily 
changed to the new, relocated hadoop's GeneratedMessage in a single place (you 
can check [this 
comment|https://issues.apache.org/jira/browse/TEZ-4152?focusedCommentId=17090713&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17090713]
 for my previously failing attempts to achieve this)

3. ShuffleHandler TokenProto change: it was a compilation failure while 
depending on hadoop 3.3 (due to package collision, because of the same root 
cause, relocated protobuf classes in hadoop) 

> tez-protobuf: a single module for handling protobuf dependency
> --
>
> Key: TEZ-4154
> URL: https://issues.apache.org/jira/browse/TEZ-4154
> Project: Apache Tez
>  Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: TEZ-4154.01.patch, TEZ-4154.02.patch, TEZ-4154.03.patch, 
> TEZ-4154.04.patch
>
>
> Similarly to what I've seen in TEZ-2164, I would like to create a module that 
> is a single point of control of how tez handles protobuf dependency. In the 
> near future, when tez will upgrade to hadoop 3.3, there will be a couple of 
> things which should be taken care of (see TEZ-4152), e.g. upgrade to protobuf 
> 3.x and handle the possible mess that can come from the fact that hadoop has 
> relocated its protobuf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2020-04-27 Thread TezQA (Jira)


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093605#comment-17093605
 ] 

TezQA commented on TEZ-4154:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  3m 
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  5m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m  
4s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  5m  
8s{color} | {color:blue} Used deprecated FindBugs config; considering switching 
to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  5m  
3s{color} | {color:red} root in master has 2 extant findbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
57s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 19s{color} | {color:orange} tez-protobuf: The patch generated 1 new + 0 
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 46s{color} | {color:orange} root: The patch generated 1 new + 47 unchanged - 
0 fixed = 48 total (was 47) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m 
18s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  5m 
17s{color} | {color:red} root generated 1 new + 2 unchanged - 0 fixed = 3 total 
(was 2) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
21s{color} | {color:green} tez-protobuf in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 16s{color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
25s{color} | {color:green} hadoop-shim-2.8 in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
57s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
37s{color} | {color:green} tez-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 
30s{color} | {color:green} tez-dag in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m  
4s{color} | {color:green} tez-ext-service-tests in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
20s{color} | {color:green} tez-mapreduce in the patch passed. {color} |
| {color

[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency

2020-04-25 Thread TezQA (Jira)


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17092387#comment-17092387
 ] 

TezQA commented on TEZ-4154:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  4m 
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m  
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  4m 
44s{color} | {color:blue} Used deprecated FindBugs config; considering 
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
39s{color} | {color:red} root in master has 2 extant findbugs warnings. {color} 
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m  
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 20s{color} | {color:orange} tez-protobuf: The patch generated 2 new + 0 
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 46s{color} | {color:orange} root: The patch generated 3 new + 47 unchanged - 
0 fixed = 50 total (was 47) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 23s{color} | {color:orange} tez-plugins/tez-aux-services: The patch 
generated 1 new + 47 unchanged - 0 fixed = 48 total (was 47) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m 
17s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  5m 
10s{color} | {color:red} root generated 1 new + 2 unchanged - 0 fixed = 3 total 
(was 2) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
22s{color} | {color:green} tez-protobuf in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 15s{color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
25s{color} | {color:green} hadoop-shim-2.8 in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
58s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
39s{color} | {color:green} tez-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 
32s{color} | {color:green} tez-dag in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
59s{color}

[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency and its shading/relocating

2020-04-25 Thread TezQA (Jira)


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17092136#comment-17092136
 ] 

TezQA commented on TEZ-4154:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m  
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
 5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  5m 
 9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  4m 
35s{color} | {color:blue} Used deprecated FindBugs config; considering 
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
32s{color} | {color:red} root in master has 2 extant findbugs warnings. {color} 
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
 4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
25s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 25s{color} 
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 21s{color} | {color:orange} tez-protobuf: The patch generated 2 new + 0 
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 43s{color} | {color:orange} root: The patch generated 2 new + 0 unchanged - 
0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m 
19s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  7m  
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
23s{color} | {color:red} tez-protobuf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
28s{color} | {color:red} root in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
21s{color} | {color:green} tez-protobuf in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  2m 19s{color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
24s{color} | {color:green} hadoop-shim-2.8 in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
58s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
39s{color} | {color:green} tez-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 
19s{color} | {color:green} tez-dag in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
56s{color} | {color:green} tez-ext-service-tests in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {colo

[jira] [Commented] (TEZ-4154) tez-protobuf: a single module for handling protobuf dependency and its shading/relocating

2020-04-22 Thread TezQA (Jira)


[ 
https://issues.apache.org/jira/browse/TEZ-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17089861#comment-17089861
 ] 

TezQA commented on TEZ-4154:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m  
3s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
2s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 34 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m  
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  6m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  4m 
45s{color} | {color:blue} Used deprecated FindBugs config; considering 
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  4m 
41s{color} | {color:red} root in master has 2 extant findbugs warnings. {color} 
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
27s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 27s{color} 
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 33s{color} | {color:orange} root: The patch generated 2 new + 1921 unchanged 
- 2 fixed = 1923 total (was 1923) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 56s{color} | {color:orange} tez-dag: The patch generated 1 new + 1038 
unchanged - 1 fixed = 1039 total (was 1039) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 25s{color} | {color:orange} tez-tests: The patch generated 1 new + 47 
unchanged - 1 fixed = 48 total (was 48) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m 
18s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
43s{color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
21s{color} | {color:blue} tez-protobuf has no data from findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
33s{color} | {color:red} root in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
22s{color} | {color:green} tez-protobuf in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  2m  5s{color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
58s{color} | {color:green} tez-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
38s{color} | {color:green} tez-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  4m  2s{color} 
| {color:red} tez-dag in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m  6s{color} 
| {color:red} tez-ext-service-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
21s{color} | {color:green} tez-mapreduce i