[jira] [Commented] (BEAM-1624) Unable to deserialize Coder in DataflowRunner

2017-03-07 Thread Kenneth Knowles (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899974#comment-15899974
 ] 

Kenneth Knowles commented on BEAM-1624:
---

It hasn't been reproduced, so I am closing it. I will follow up to make sure 
our ITs are sufficient.

> Unable to deserialize Coder in DataflowRunner
> -
>
> Key: BEAM-1624
> URL: https://issues.apache.org/jira/browse/BEAM-1624
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Frances Perry
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: Not applicable
>
>
> To repro, sync to head and run the LeaderBoard example with the Dataflow 
> runner
> Does not repro in 0.5.
> Caused by: java.lang.RuntimeException: Unable to deserialize Coder: 
> WindowedValue$FullWindowedValueCoder(KvCoder(BigQueryIO$ShardedKeyCoder(StringUtf8Coder),BigQueryIO$TableRowInfoCoder),IntervalWindow$IntervalWindowCoder).
>  Check that a suitable constructor is defined.  See Coder for details.
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:115)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:655)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:602)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translateOutputs(DataflowPipelineTranslator.java:945)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.access$1200(DataflowPipelineTranslator.java:111)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translateMultiHelper(DataflowPipelineTranslator.java:836)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:826)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:823)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:413)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$400(TransformHierarchy.java:231)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:206)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:321)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:363)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:153)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:505)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:150)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:210)
>   at 
> org.apache.beam.examples.complete.game.GameStats.main(GameStats.java:340)
>   ... 6 more
> Caused by: java.lang.RuntimeException: Unable to deserialize class interface 
> org.apache.beam.sdk.coders.Coder
>   at org.apache.beam.sdk.util.Serializer.deserialize(Serializer.java:102)
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:112)
>   ... 29 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1624) Unable to deserialize Coder in DataflowRunner

2017-03-07 Thread Ahmet Altay (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899880#comment-15899880
 ] 

Ahmet Altay commented on BEAM-1624:
---

Is this fixed?

> Unable to deserialize Coder in DataflowRunner
> -
>
> Key: BEAM-1624
> URL: https://issues.apache.org/jira/browse/BEAM-1624
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Frances Perry
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 0.6.0
>
>
> To repro, sync to head and run the LeaderBoard example with the Dataflow 
> runner
> Does not repro in 0.5.
> Caused by: java.lang.RuntimeException: Unable to deserialize Coder: 
> WindowedValue$FullWindowedValueCoder(KvCoder(BigQueryIO$ShardedKeyCoder(StringUtf8Coder),BigQueryIO$TableRowInfoCoder),IntervalWindow$IntervalWindowCoder).
>  Check that a suitable constructor is defined.  See Coder for details.
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:115)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:655)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:602)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translateOutputs(DataflowPipelineTranslator.java:945)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.access$1200(DataflowPipelineTranslator.java:111)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translateMultiHelper(DataflowPipelineTranslator.java:836)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:826)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:823)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:413)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$400(TransformHierarchy.java:231)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:206)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:321)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:363)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:153)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:505)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:150)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:210)
>   at 
> org.apache.beam.examples.complete.game.GameStats.main(GameStats.java:340)
>   ... 6 more
> Caused by: java.lang.RuntimeException: Unable to deserialize class interface 
> org.apache.beam.sdk.coders.Coder
>   at org.apache.beam.sdk.util.Serializer.deserialize(Serializer.java:102)
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:112)
>   ... 29 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1624) Unable to deserialize Coder in DataflowRunner

2017-03-06 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898462#comment-15898462
 ] 

ASF GitHub Bot commented on BEAM-1624:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2173


> Unable to deserialize Coder in DataflowRunner
> -
>
> Key: BEAM-1624
> URL: https://issues.apache.org/jira/browse/BEAM-1624
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Frances Perry
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 0.6.0
>
>
> To repro, sync to head and run the LeaderBoard example with the Dataflow 
> runner
> Does not repro in 0.5.
> Caused by: java.lang.RuntimeException: Unable to deserialize Coder: 
> WindowedValue$FullWindowedValueCoder(KvCoder(BigQueryIO$ShardedKeyCoder(StringUtf8Coder),BigQueryIO$TableRowInfoCoder),IntervalWindow$IntervalWindowCoder).
>  Check that a suitable constructor is defined.  See Coder for details.
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:115)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:655)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:602)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translateOutputs(DataflowPipelineTranslator.java:945)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.access$1200(DataflowPipelineTranslator.java:111)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translateMultiHelper(DataflowPipelineTranslator.java:836)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:826)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:823)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:413)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$400(TransformHierarchy.java:231)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:206)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:321)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:363)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:153)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:505)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:150)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:210)
>   at 
> org.apache.beam.examples.complete.game.GameStats.main(GameStats.java:340)
>   ... 6 more
> Caused by: java.lang.RuntimeException: Unable to deserialize class interface 
> org.apache.beam.sdk.coders.Coder
>   at org.apache.beam.sdk.util.Serializer.deserialize(Serializer.java:102)
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:112)
>   ... 29 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1624) Unable to deserialize Coder in DataflowRunner

2017-03-06 Thread Kenneth Knowles (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898433#comment-15898433
 ] 

Kenneth Knowles commented on BEAM-1624:
---

Couldn't repro - possible that the jar was corrupted via some 
multiple-jobs-running race condition?

> Unable to deserialize Coder in DataflowRunner
> -
>
> Key: BEAM-1624
> URL: https://issues.apache.org/jira/browse/BEAM-1624
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Frances Perry
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 0.6.0
>
>
> To repro, sync to head and run the LeaderBoard example with the Dataflow 
> runner
> Does not repro in 0.5.
> Caused by: java.lang.RuntimeException: Unable to deserialize Coder: 
> WindowedValue$FullWindowedValueCoder(KvCoder(BigQueryIO$ShardedKeyCoder(StringUtf8Coder),BigQueryIO$TableRowInfoCoder),IntervalWindow$IntervalWindowCoder).
>  Check that a suitable constructor is defined.  See Coder for details.
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:115)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:655)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:602)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translateOutputs(DataflowPipelineTranslator.java:945)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.access$1200(DataflowPipelineTranslator.java:111)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translateMultiHelper(DataflowPipelineTranslator.java:836)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:826)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:823)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:413)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$400(TransformHierarchy.java:231)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:206)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:321)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:363)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:153)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:505)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:150)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:210)
>   at 
> org.apache.beam.examples.complete.game.GameStats.main(GameStats.java:340)
>   ... 6 more
> Caused by: java.lang.RuntimeException: Unable to deserialize class interface 
> org.apache.beam.sdk.coders.Coder
>   at org.apache.beam.sdk.util.Serializer.deserialize(Serializer.java:102)
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:112)
>   ... 29 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1624) Unable to deserialize Coder in DataflowRunner

2017-03-06 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898417#comment-15898417
 ] 

ASF GitHub Bot commented on BEAM-1624:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2173

[BEAM-1624] Add tests for serialization of BigQueryIO.TableRowInfoCoder

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

I have not yet reproduced the issue in BEAM-1624 (my LeaderBoard pipeline 
seems happy), but these tests are a good idea anyhow.

R: @tgroh 
CC: @francesperry 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam BQIO-Coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2173.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2173


commit 57886246a08e82a2cff0d6b95816b76a5b912595
Author: Kenneth Knowles 
Date:   2017-03-07T00:02:54Z

Add tests for serialization of BigQueryIO.TableRowInfoCoder




> Unable to deserialize Coder in DataflowRunner
> -
>
> Key: BEAM-1624
> URL: https://issues.apache.org/jira/browse/BEAM-1624
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Frances Perry
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 0.6.0
>
>
> To repro, sync to head and run the LeaderBoard example with the Dataflow 
> runner
> Does not repro in 0.5.
> Caused by: java.lang.RuntimeException: Unable to deserialize Coder: 
> WindowedValue$FullWindowedValueCoder(KvCoder(BigQueryIO$ShardedKeyCoder(StringUtf8Coder),BigQueryIO$TableRowInfoCoder),IntervalWindow$IntervalWindowCoder).
>  Check that a suitable constructor is defined.  See Coder for details.
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:115)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:655)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:602)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translateOutputs(DataflowPipelineTranslator.java:945)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.access$1200(DataflowPipelineTranslator.java:111)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translateMultiHelper(DataflowPipelineTranslator.java:836)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:826)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:823)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:413)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$400(TransformHierarchy.java:231)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:206)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:321)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:363)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:153)
>   at 
>

[jira] [Commented] (BEAM-1624) Unable to deserialize Coder in DataflowRunner

2017-03-06 Thread Kenneth Knowles (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898387#comment-15898387
 ] 

Kenneth Knowles commented on BEAM-1624:
---

False; the coder is serializable. The topmost error is a red herring - the 
inability to deserialize {{Coder}} is more curious.

> Unable to deserialize Coder in DataflowRunner
> -
>
> Key: BEAM-1624
> URL: https://issues.apache.org/jira/browse/BEAM-1624
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Frances Perry
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 0.6.0
>
>
> To repro, sync to head and run the LeaderBoard example with the Dataflow 
> runner
> Does not repro in 0.5.
> Caused by: java.lang.RuntimeException: Unable to deserialize Coder: 
> WindowedValue$FullWindowedValueCoder(KvCoder(BigQueryIO$ShardedKeyCoder(StringUtf8Coder),BigQueryIO$TableRowInfoCoder),IntervalWindow$IntervalWindowCoder).
>  Check that a suitable constructor is defined.  See Coder for details.
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:115)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:655)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:602)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translateOutputs(DataflowPipelineTranslator.java:945)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.access$1200(DataflowPipelineTranslator.java:111)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translateMultiHelper(DataflowPipelineTranslator.java:836)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:826)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:823)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:413)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$400(TransformHierarchy.java:231)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:206)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:321)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:363)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:153)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:505)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:150)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:210)
>   at 
> org.apache.beam.examples.complete.game.GameStats.main(GameStats.java:340)
>   ... 6 more
> Caused by: java.lang.RuntimeException: Unable to deserialize class interface 
> org.apache.beam.sdk.coders.Coder
>   at org.apache.beam.sdk.util.Serializer.deserialize(Serializer.java:102)
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:112)
>   ... 29 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1624) Unable to deserialize Coder in DataflowRunner

2017-03-06 Thread Kenneth Knowles (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898227#comment-15898227
 ] 

Kenneth Knowles commented on BEAM-1624:
---

My guess is wrong: this is actually a legitimate code path to make sure the 
coder is deserializable via its JSON. So it is likely an erroneous coder in 
BigQueryIO missing some Jackson annotations.

> Unable to deserialize Coder in DataflowRunner
> -
>
> Key: BEAM-1624
> URL: https://issues.apache.org/jira/browse/BEAM-1624
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Frances Perry
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 0.6.0
>
>
> To repro, sync to head and run the LeaderBoard example with the Dataflow 
> runner
> Does not repro in 0.5.
> Caused by: java.lang.RuntimeException: Unable to deserialize Coder: 
> WindowedValue$FullWindowedValueCoder(KvCoder(BigQueryIO$ShardedKeyCoder(StringUtf8Coder),BigQueryIO$TableRowInfoCoder),IntervalWindow$IntervalWindowCoder).
>  Check that a suitable constructor is defined.  See Coder for details.
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:115)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:655)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:602)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translateOutputs(DataflowPipelineTranslator.java:945)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.access$1200(DataflowPipelineTranslator.java:111)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translateMultiHelper(DataflowPipelineTranslator.java:836)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:826)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:823)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:413)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$400(TransformHierarchy.java:231)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:206)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:321)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:363)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:153)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:505)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:150)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:210)
>   at 
> org.apache.beam.examples.complete.game.GameStats.main(GameStats.java:340)
>   ... 6 more
> Caused by: java.lang.RuntimeException: Unable to deserialize class interface 
> org.apache.beam.sdk.coders.Coder
>   at org.apache.beam.sdk.util.Serializer.deserialize(Serializer.java:102)
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:112)
>   ... 29 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1624) Unable to deserialize Coder in DataflowRunner

2017-03-06 Thread Kenneth Knowles (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897813#comment-15897813
 ] 

Kenneth Knowles commented on BEAM-1624:
---

Most likely the coder getting caught in an inner class getting serialized. I'll 
take this.

> Unable to deserialize Coder in DataflowRunner
> -
>
> Key: BEAM-1624
> URL: https://issues.apache.org/jira/browse/BEAM-1624
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Frances Perry
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 0.6.0
>
>
> To repro, sync to head and run the LeaderBoard example with the Dataflow 
> runner
> Does not repro in 0.5.
> Caused by: java.lang.RuntimeException: Unable to deserialize Coder: 
> WindowedValue$FullWindowedValueCoder(KvCoder(BigQueryIO$ShardedKeyCoder(StringUtf8Coder),BigQueryIO$TableRowInfoCoder),IntervalWindow$IntervalWindowCoder).
>  Check that a suitable constructor is defined.  See Coder for details.
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:115)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:655)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:602)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translateOutputs(DataflowPipelineTranslator.java:945)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.access$1200(DataflowPipelineTranslator.java:111)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translateMultiHelper(DataflowPipelineTranslator.java:836)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:826)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:823)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:413)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$400(TransformHierarchy.java:231)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:206)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:321)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:363)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:153)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:505)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:150)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:210)
>   at 
> org.apache.beam.examples.complete.game.GameStats.main(GameStats.java:340)
>   ... 6 more
> Caused by: java.lang.RuntimeException: Unable to deserialize class interface 
> org.apache.beam.sdk.coders.Coder
>   at org.apache.beam.sdk.util.Serializer.deserialize(Serializer.java:102)
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:112)
>   ... 29 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1624) Unable to deserialize Coder in DataflowRunner

2017-03-04 Thread Frances Perry (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896112#comment-15896112
 ] 

Frances Perry commented on BEAM-1624:
-

[~altay] FYI -- considering this 0.6 release blocking until triaged.


> Unable to deserialize Coder in DataflowRunner
> -
>
> Key: BEAM-1624
> URL: https://issues.apache.org/jira/browse/BEAM-1624
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Frances Perry
>Assignee: Davor Bonaci
>Priority: Blocker
> Fix For: 0.6.0
>
>
> To repro, sync to head and run the LeaderBoard example with the Dataflow 
> runner
> Does not repro in 0.5.
> Caused by: java.lang.RuntimeException: Unable to deserialize Coder: 
> WindowedValue$FullWindowedValueCoder(KvCoder(BigQueryIO$ShardedKeyCoder(StringUtf8Coder),BigQueryIO$TableRowInfoCoder),IntervalWindow$IntervalWindowCoder).
>  Check that a suitable constructor is defined.  See Coder for details.
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:115)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:655)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$StepTranslator.addOutput(DataflowPipelineTranslator.java:602)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translateOutputs(DataflowPipelineTranslator.java:945)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.access$1200(DataflowPipelineTranslator.java:111)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translateMultiHelper(DataflowPipelineTranslator.java:836)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:826)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$6.translate(DataflowPipelineTranslator.java:823)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:413)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:486)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:481)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$400(TransformHierarchy.java:231)
>   at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:206)
>   at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:321)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:363)
>   at 
> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:153)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:505)
>   at 
> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:150)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:210)
>   at 
> org.apache.beam.examples.complete.game.GameStats.main(GameStats.java:340)
>   ... 6 more
> Caused by: java.lang.RuntimeException: Unable to deserialize class interface 
> org.apache.beam.sdk.coders.Coder
>   at org.apache.beam.sdk.util.Serializer.deserialize(Serializer.java:102)
>   at 
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(SerializableUtils.java:112)
>   ... 29 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)