[
https://issues.apache.org/jira/browse/BEAM-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619901#comment-16619901
]
Ryan Williams commented on BEAM-4176:
-------------------------------------
As of [#6328|https://github.com/apache/beam/pull/6328] being merged, we're down
to 117 failures:
!https://camo.githubusercontent.com/be8a669f7d4fd09b1cd83e224ab37db2a4c68fd4/68747470733a2f2f636c2e6c792f3465643734343832393236342f53637265656e25323053686f74253230323031382d30392d31382532306174253230362e30342e3232253230504d2e706e67!
I put them in [this google
sheet|https://docs.google.com/spreadsheets/d/1aNH4iwR99s2bSjFpj3vcHhdrVg_9q7XySX_5jKjyK2E/edit?usp=sharing],
where I am also going through and spot-checking why each case failed ([here's
the full test output I'm working
from|https://storage.googleapis.com/runsascoded-tmp/beam-portable-flink-vr-tests/ecd1ac085a/index.html]).
So far, all the cases I've checked have failed due transforms of the form
{{Combine.globally(…)/View.AsIterable/View.CreatePCollectionView}} , e.g.:
{code:java}
[flink-runner-job-server] ERROR
org.apache.beam.runners.flink.FlinkJobInvocation - Error during job invocation
combinetest0accumulationtests0testaccumulatingcombineempty-ryan-0918183535-7620b406_f38db722-b27f-45a6-bbf5-e0bc86261cb2.
java.lang.IllegalArgumentException: Unknown type of URN
beam:transform:create_view:v1 for PTransform with id
Combine.globally(MeanInts)/View.AsIterable/View.CreatePCollectionView.
at
org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.urnNotFound(FlinkBatchPortablePipelineTranslator.java:578)
at
org.apache.beam.runners.flink.FlinkBatchPortablePipelineTranslator.translate(FlinkBatchPortablePipelineTranslator.java:233)
at
org.apache.beam.runners.flink.FlinkJobInvocation.runPipeline(FlinkJobInvocation.java:112)
at
org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
at
org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
{code}
This is promising; getting {{PTransformTranslation.CREATE_VIEW_TRANSFORM_URN}}
into [this list in
FlinkBatchPortablePipelineTranslator|https://github.com/apache/beam/blob/c49a97ecbf815b320926285dcddba993590e3073/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java#L132-L153]
may recover many more cases.
Should FlinkBatchPortablePipelineTranslator handle {{create_view}} URN
directly? Or should such transforms be folded into ExecutableStages? Or
something else?
> Java: Portable batch runner passes all ValidatesRunner tests that
> non-portable runner passes
> --------------------------------------------------------------------------------------------
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Ben Sidhom
> Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot
> 2018-09-03 at 11.07.38 AM.png
>
> Time Spent: 22h 20m
> Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
