[
https://issues.apache.org/jira/browse/BEAM-3799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Groh resolved BEAM-3799.
-------------------------------
Resolution: Fixed
This was a mismatch between the assumptions made by {{MutationDetector}} about
the returned behavior of a {{StructuralValue}} and the implementation in
{{SerializableCoder}}
That mismatch likely requires additional work to be performed in
{{SerializeableCoder}} to more aggressively validate that elements that encode
to the same bytes have a meaningfully defined equivalence.
> Nexmark Query 10 breaks with direct runner
> ------------------------------------------
>
> Key: BEAM-3799
> URL: https://issues.apache.org/jira/browse/BEAM-3799
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
> Affects Versions: 2.4.0, 2.5.0
> Reporter: Ismaël Mejía
> Assignee: Thomas Groh
> Priority: Major
> Fix For: 2.4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> While running query 10 with the direct runner like this:
> {quote}mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main
> -Pdirect-runner -Dexec.args="--runner=DirectRunner --query=10
> --streaming=false --manageResources=false --monitorJobs=true
> --enforceEncodability=true --enforceImmutability=true" -pl 'sdks/java/nexmark'
> {quote}
> I found that it breaks with the direct runner with following exception (it
> works ok with the other runners):
> {quote}[WARNING]
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke
> (NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke (Method.java:498)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:294)
> at java.lang.Thread.run (Thread.java:748)
> Caused by: org.apache.beam.sdk.util.IllegalMutationException: PTransform
> Query10/Query10.UploadEvents/ParMultiDo(Anonymous) mutated value KV{null,
> 2015-07-15T00:00:09.999Z shard-00003-of-00025 0 ON_TIME null
> } after it was output (new value was KV{null, 2015-07-15T00:00:09.999Z
> shard-00003-of-00025 0 ON_TIME null
> }). Values must not be mutated in any way after being output.
> at
> org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.commit
> (ImmutabilityCheckingBundleFactory.java:134)
> at org.apache.beam.runners.direct.EvaluationContext.commitBundles
> (EvaluationContext.java:212)
> at org.apache.beam.runners.direct.EvaluationContext.handleResult
> (EvaluationContext.java:152)
> at
> org.apache.beam.runners.direct.QuiescenceDriver$TimerIterableCompletionCallback.handleResult
> (QuiescenceDriver.java:258)
> at org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle
> (DirectTransformExecutor.java:190)
> at org.apache.beam.runners.direct.DirectTransformExecutor.run
> (DirectTransformExecutor.java:127)
> at java.util.concurrent.Executors$RunnableAdapter.call
> (Executors.java:511)
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> at java.lang.Thread.run (Thread.java:748)
> Caused by: org.apache.beam.sdk.util.IllegalMutationException: Value KV{null,
> 2015-07-15T00:00:09.999Z shard-00003-of-00025 0 ON_TIME null
> } mutated illegally, new value was KV{null, 2015-07-15T00:00:09.999Z
> shard-00003-of-00025 0 ON_TIME null
> }. Encoding was
> rO0ABXNyADZvcmcuYXBhY2hlLmJlYW0uc2RrLm5leG1hcmsucXVlcmllcy5RdWVyeTEwJE91dHB1dEZpbGUWUg9rZM1SvgIABUoABWluZGV4TAAIZmlsZW5hbWV0ABJMamF2YS9sYW5nL1N0cmluZztMAAxtYXhUaW1lc3RhbXB0ABdMb3JnL2pvZGEvdGltZS9JbnN0YW50O0wABXNoYXJkcQB-AAFMAAZ0aW1pbmd0ADpMb3JnL2FwYWNoZS9iZWFtL3Nkay90cmFuc2Zvcm1zL3dpbmRvd2luZy9QYW5lSW5mbyRUaW1pbmc7eHAAAAAAAAAAAHBzcgAVb3JnLmpvZGEudGltZS5JbnN0YW50Lci-0MYOnM0CAAFKAAdpTWlsbGlzeHAAAAFOjwLrD3QAFHNoYXJkLTAwMDAzLW9mLTAwMDI1fnIAOG9yZy5hcGFjaGUuYmVhbS5zZGsudHJhbnNmb3Jtcy53aW5kb3dpbmcuUGFuZUluZm8kVGltaW5nAAAAAAAAAAASAAB4cgAOamF2YS5sYW5nLkVudW0AAAAAAAAAABIAAHhwdAAHT05fVElNRQ,
> now
> rO0ABXNyADZvcmcuYXBhY2hlLmJlYW0uc2RrLm5leG1hcmsucXVlcmllcy5RdWVyeTEwJE91dHB1dEZpbGUWUg9rZM1SvgIABUoABWluZGV4TAAIZmlsZW5hbWV0ABJMamF2YS9sYW5nL1N0cmluZztMAAxtYXhUaW1lc3RhbXB0ABdMb3JnL2pvZGEvdGltZS9JbnN0YW50O0wABXNoYXJkcQB-AAFMAAZ0aW1pbmd0ADpMb3JnL2FwYWNoZS9iZWFtL3Nkay90cmFuc2Zvcm1zL3dpbmRvd2luZy9QYW5lSW5mbyRUaW1pbmc7eHAAAAAAAAAAAHBzcgAVb3JnLmpvZGEudGltZS5JbnN0YW50Lci-0MYOnM0CAAFKAAdpTWlsbGlzeHAAAAFOjwLrD3QAFHNoYXJkLTAwMDAzLW9mLTAwMDI1fnIAOG9yZy5hcGFjaGUuYmVhbS5zZGsudHJhbnNmb3Jtcy53aW5kb3dpbmcuUGFuZUluZm8kVGltaW5nAAAAAAAAAAASAAB4cgAOamF2YS5sYW5nLkVudW0AAAAAAAAAABIAAHhwdAAHT05fVElNRQ.
> at
> org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.illegalMutation
> (MutationDetectors.java:144)
> at
> org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodifiedThrowingCheckedExceptions
> (MutationDetectors.java:139)
> at
> org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodified
> (MutationDetectors.java:123)
> at
> org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.commit
> (ImmutabilityCheckingBundleFactory.java:124)
> at org.apache.beam.runners.direct.EvaluationContext.commitBundles
> (EvaluationContext.java:212)
> at org.apache.beam.runners.direct.EvaluationContext.handleResult
> (EvaluationContext.java:152)
> at
> org.apache.beam.runners.direct.QuiescenceDriver$TimerIterableCompletionCallback.handleResult
> (QuiescenceDriver.java:258)
> at org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle
> (DirectTransformExecutor.java:190)
> at org.apache.beam.runners.direct.DirectTransformExecutor.run
> (DirectTransformExecutor.java:127)
> at java.util.concurrent.Executors$RunnableAdapter.call
> (Executors.java:511)
> at java.util.concurrent.FutureTask.run (FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker
> (ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:624)
> at java.lang.Thread.run (Thread.java:748)
> Mar 07, 2018 5:01:45 PM org.apache.beam.sdk.nexmark.queries.Query10$3
> processElement
> INFO: Writing shard-00004-of-00025 with record timestamp
> 2015-07-15T00:00:09.999Z, window timestamp 2015-07-15T00:00:09.999Z, pane
> PaneInfo\{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}
> Mar 07, 2018 5:01:45 PM org.apache.beam.sdk.nexmark.queries.Query10$3
> processElement
> INFO: Writing shard-00021-of-00025 with record timestamp
> 2015-07-15T00:00:09.999Z, window timestamp 2015-07-15T00:00:09.999Z, pane
> PaneInfo\{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}
> Mar 07, 2018 5:01:45 PM org.apache.beam.sdk.nexmark.queries.Query10$3
> processElement
> INFO: Writing shard-00018-of-00025 with record timestamp
> 2015-07-15T00:00:09.999Z, window timestamp 2015-07-15T00:00:09.999Z, pane
> PaneInfo\{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}
> Mar 07, 2018 5:01:45 PM org.apache.beam.sdk.nexmark.queries.Query10$3
> processElement
> INFO: Writing shard-00002-of-00025 with record timestamp
> 2015-07-15T00:00:09.999Z, window timestamp 2015-07-15T00:00:09.999Z, pane
> PaneInfo\{isFirst=true, isLast=true, timing=ON_TIME, index=0, onTimeIndex=0}
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 32.560 s
> [INFO] Finished at: 2018-03-07T17:01:45+01:00
> [INFO] Final Memory: 82M/2831M
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java
> (default-cli) on project beam-sdks-java-nexmark: An exception occured while
> executing the Java class. null: InvocationTargetException: PTransform
> Query10/Query10.UploadEvents/ParMultiDo(Anonymous) mutated value KV{null,
> 2015-07-15T00:00:09.999Z shard-00003-of-00025 0 ON_TIME null
> [ERROR] } after it was output (new value was KV{null,
> 2015-07-15T00:00:09.999Z shard-00003-of-00025 0 ON_TIME null
> [ERROR] }). Values must not be mutated in any way after being output. Value
> KV{null, 2015-07-15T00:00:09.999Z shard-00003-of-00025 0 ON_TIME null
> [ERROR] } mutated illegally, new value was KV{null, 2015-07-15T00:00:09.999Z
> shard-00003-of-00025 0 ON_TIME null
> [ERROR] }. Encoding was
> rO0ABXNyADZvcmcuYXBhY2hlLmJlYW0uc2RrLm5leG1hcmsucXVlcmllcy5RdWVyeTEwJE91dHB1dEZpbGUWUg9rZM1SvgIABUoABWluZGV4TAAIZmlsZW5hbWV0ABJMamF2YS9sYW5nL1N0cmluZztMAAxtYXhUaW1lc3RhbXB0ABdMb3JnL2pvZGEvdGltZS9JbnN0YW50O0wABXNoYXJkcQB-AAFMAAZ0aW1pbmd0ADpMb3JnL2FwYWNoZS9iZWFtL3Nkay90cmFuc2Zvcm1zL3dpbmRvd2luZy9QYW5lSW5mbyRUaW1pbmc7eHAAAAAAAAAAAHBzcgAVb3JnLmpvZGEudGltZS5JbnN0YW50Lci-0MYOnM0CAAFKAAdpTWlsbGlzeHAAAAFOjwLrD3QAFHNoYXJkLTAwMDAzLW9mLTAwMDI1fnIAOG9yZy5hcGFjaGUuYmVhbS5zZGsudHJhbnNmb3Jtcy53aW5kb3dpbmcuUGFuZUluZm8kVGltaW5nAAAAAAAAAAASAAB4cgAOamF2YS5sYW5nLkVudW0AAAAAAAAAABIAAHhwdAAHT05fVElNRQ,
> now
> rO0ABXNyADZvcmcuYXBhY2hlLmJlYW0uc2RrLm5leG1hcmsucXVlcmllcy5RdWVyeTEwJE91dHB1dEZpbGUWUg9rZM1SvgIABUoABWluZGV4TAAIZmlsZW5hbWV0ABJMamF2YS9sYW5nL1N0cmluZztMAAxtYXhUaW1lc3RhbXB0ABdMb3JnL2pvZGEvdGltZS9JbnN0YW50O0wABXNoYXJkcQB-AAFMAAZ0aW1pbmd0ADpMb3JnL2FwYWNoZS9iZWFtL3Nkay90cmFuc2Zvcm1zL3dpbmRvd2luZy9QYW5lSW5mbyRUaW1pbmc7eHAAAAAAAAAAAHBzcgAVb3JnLmpvZGEudGltZS5JbnN0YW50Lci-0MYOnM0CAAFKAAdpTWlsbGlzeHAAAAFOjwLrD3QAFHNoYXJkLTAwMDAzLW9mLTAwMDI1fnIAOG9yZy5hcGFjaGUuYmVhbS5zZGsudHJhbnNmb3Jtcy53aW5kb3dpbmcuUGFuZUluZm8kVGltaW5nAAAAAAAAAAASAAB4cgAOamF2YS5sYW5nLkVudW0AAAAAAAAAABIAAHhwdAAHT05fVElNRQ.
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> {quote}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
