[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1180135670 Thank you for taking the time to review this change so far, I have implemented your suggestions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1182582488 @steveloughran, I will conduct some more testing from a spark perspective and report back once I'm fully confident, since that is where I initially observed the failures. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1184222581 relates to https://github.com/eclipse-openj9/openj9/issues/14950 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1339652525 > Has there been any movement on this pr? I'll try and carve out a few hours this week 👍 some slight enhancement I think could be made in the class loader method I originally came up with (I want to test my concerns). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1339816599 This should be a bit more robust to extension as well as handle the concerns I had about the class loader from before. Is there any consensus/ruling around adding a test against the IBM JREs? I appreciate it would take a bit of time on CI and this is a once in a blue moon activity, but it could be a single suite of integration tests against auth that execute to verify the result against latest semeru is not IBM, and vice versa. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1342580453 Few changes made here based on checks against other architectures/platforms. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1343032982 Javadoc seemed to stackoverflow, that should be correct now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1344249638 @steveloughran is there a recommended way to kick CI different from pushing empty commits? I'm fairly confident the failures I'm observing here have been pulled from upstream based on what I'm seeing locally, if that's the case and I can hold off until a fix comes in... if it's not I'm a little baffled and may need to investigate deeper. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1346679972 @steveloughran I've popped up a PR against branch 3.3; should I do the same for 3.3.5? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1377805782 @ivrisivris Sorry, I don't know the answer to that one, but there is a workaround you might consider in the meantime which makes use of an agent to manipulate the vendor name - you can see an example [here](https://github.com/eclipse-openj9/openj9/issues/14950#issuecomment-1105797810) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1385582733 I'll check this out against 11.0.17 today -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1385975581 I've just span up a version of rc-0 3.3.5 against semeru 11.0.17 from and executed a few of the sample jobs and checked out the dev/null out. eg. ``` yarn jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar teragen 100 /test yarn jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar terasort /test /testout ``` I'm not seeing anything suspicious in the stderr - @ivrisivris can you give me some more detail on the specific cluster config / actions / workloads that triggered this so I can reproduce and diagnose a bit better? Are you using this with Spark or anything else? Are these being added to the classpath in some other way? I currently don't believe anything packaged IBM semeru 11.0.17 or hadoop rc-0 3.3.5 contains something to trigger the check added, (I've even ran a jar with just that class to check that, and this looks good). Whilst I wait on a response on the above I'll check that there isn't anywhere else that is not using the `IBM_JAVA` check to determine the use of those modules and report back shortly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1385992169 From what I can see, the below could still exhibit the same behaviour @steveloughran - I'll see if I can figure out how to get into these paths to validate it so I understand the scenario @ivrisivris is likely hitting. - https://github.com/apache/hadoop/blob/cf33164857530a38aa3edc99a9051a47aae9dede/hadoop-common-project/hadoop-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java#L606-L612 - https://github.com/apache/hadoop/blob/cf33164857530a38aa3edc99a9051a47aae9dede/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/JaasConfiguration.java#L70-L75 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1386308333 I hit a similar error trivially in spark 3.4.0/dev -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1422263273 @Tre2878 did you build the branch it was previously merged to from source? It's merged back retrospectively but once published initially there aren't more updates to a stable stream in terms of distributions. Ie. If you aren't switching versions and aren't building you aren't getting any back ported fixes. Please us 3.3.5 RC or build the branch yourself to pick up the changes 👍🏼 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1422280251 If you are finding some more areas where IBM classes are being called relating to auth, if you can demonstrate a configuration with specific details as well as how you reproduce it I will take a look. I need the JRE details, OS details and full hadoop config and command executed to hit this. If this is specifically relating to Kerberos, please add details for that too but obviously I won't need any secrets. I've spent some good hours trying to replicate it so far so having this detail will be awesome for me to understand where it's failing for some. May be best to pop this on the jira only. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1422283214 @Tre2878 - any of the files I listed here look sus to you https://github.com/apache/hadoop/pull/4537#issuecomment-1385992169? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1422734164 Yeah, I just need a way I can reliably hit the errors to understand a bit better what isn't covered so far to proceed further. This indeed looks similar to the previous issue reported, so it would seem to be a common use case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] JackBuggins commented on pull request #4537: HADOOP-18329 - Support for IBM Semeru JVM v>11.0.15.0 Vendor Name Changes
JackBuggins commented on PR #4537: URL: https://github.com/apache/hadoop/pull/4537#issuecomment-1424011963 @Tre2878 - That's correct, it's just branch-3.3 and trunk, which is in turn in the 3.3.5 stream, so right now upgrading/testing out the 3.3.5 stream is the way to get these changes. Looks like the 3.3.4 distro was cut around August 2022 (based on git releases), and the changes go to branch-3.3 during December 2022. I would need to defer to project owners if porting to branch 3.3.4 for others to pickup when building from source would be an option here. You could try cherry-picking the commit against 3.3 to branch 3.3.4 then build your own copy to test, or otherwise consider applying some type of agent workaround to modify the vendor name until you're in a position to do this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
