[
https://issues.apache.org/jira/browse/MAPREDUCE-7545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18119049#comment-18119049
]
ASF GitHub Bot commented on MAPREDUCE-7545:
-------------------------------------------
joseluisll opened a new pull request, #8754:
URL: https://github.com/apache/hadoop/pull/8754
TaskId.hashCode mixed in getTaskType().hashCode(). TaskType is an enum, and
an enum's hashCode is its identity hash, which HotSpot draws from a sequence
that shifts with whatever else asked for identity hashes first: which jars are
on the classpath, which classes get initialized. So every HashMap keyed by
TaskId iterated in an order that depended on the environment rather than on the
tasks.
TestRuntimeEstimators depends on that order. The five slow maps it builds
are speculated in the order the speculator meets them, and it expects three
speculations to win. In some orders only one does, and the test then fails
deterministically on that environment:
We got the wrong number of successful speculations.
==> expected: <3> but was: <1>
It does, for instance, on GitHub Actions with the classpath HADOOP-19970
produces, while passing locally on the same commit. Consuming four extra
identity hashes before the test starts reproduces the failure locally.
Hash the task type's ordinal instead. The hash no longer changes between
JVMs, so neither does the order.
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
### How was this patch tested?
### For code changes:
- [ ] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint
declared according to the connector-specific documentation? *Note:
Automated CI
testing doesn't cover all cases so manual testing with cloud storage
is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [ ] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [ ] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
> TaskId.hashCode depends on the JVM
> ----------------------------------
>
> Key: MAPREDUCE-7545
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7545
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Jose Luis López
> Assignee: Jose Luis López
> Priority: Major
>
> TaskId.hashCode() mixes in getTaskType().hashCode(). TaskType is an enum, and
> an
> enum's hashCode() is its identity hash. HotSpot draws identity hashes from a
> sequence that shifts with whatever asked for one earlier in the JVM: which
> jars
> are on the classpath, which classes are initialized. So every HashMap keyed by
> TaskId iterates in an order that depends on the environment, not on the tasks.
> TestRuntimeEstimators depends on that order. It builds five slow maps and
> expects three speculations to win. In some orders only one does, and all three
> tests then fail every time on that environment:
> TestRuntimeEstimators.testLegacyEstimator:254->coreTestEstimator:247
> We got the wrong number of successful speculations. ==> expected: <3> but
> was: <1>
> Seen on GitHub Actions (Java 17) on the HADOOP-19970 branch, twice, with the
> surefire reruns failing identically, while the same commit passes locally.
> Reproduced locally by consuming four extra identity hashes before the test
> starts (new Object().hashCode() in a static block), with no other change.
> Fix: hash the task type's ordinal instead of the enum itself, so the hash is
> the same in every JVM. A test in TestIds pins it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
