[jira] [Assigned] (BEAM-9932) Add documentation describing cross-language test pipelines
[ https://issues.apache.org/jira/browse/BEAM-9932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath reassigned BEAM-9932: --- Assignee: Kevin Sijo Puthusseri (was: Chamikara Madhusanka Jayalath) > Add documentation describing cross-language test pipelines > -- > > Key: BEAM-9932 > URL: https://issues.apache.org/jira/browse/BEAM-9932 > Project: Beam > Issue Type: Improvement > Components: cross-language >Reporter: Chamikara Madhusanka Jayalath >Assignee: Kevin Sijo Puthusseri >Priority: P2 > > We designed cross-language test pipelines [1][2] based on the discussion in > [3]. > Adding some pydocs and Java docs regarding rational behind each pipeline will > be helpful. > [1] > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/validate_runner_xlang_test.py] > [2] > [https://github.com/apache/beam/blob/master/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java] > [3] > [https://docs.google.com/document/d/1xQp0ElIV84b8OCVz8CD2hvbiWdR8w4BvWxPTZJZA6NA/edit?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7163) Godoc for passert.Sum seems wrong
[ https://issues.apache.org/jira/browse/BEAM-7163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath reassigned BEAM-7163: --- Assignee: Kevin Sijo Puthusseri > Godoc for passert.Sum seems wrong > - > > Key: BEAM-7163 > URL: https://issues.apache.org/jira/browse/BEAM-7163 > Project: Beam > Issue Type: Improvement > Components: sdk-go >Reporter: Damien Desfontaines >Assignee: Kevin Sijo Puthusseri >Priority: P2 > Labels: stale-P2 > > From > [https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/testing/passert#Sum]: > > Sum validates that the incoming PCollection is a singleton with the > > given value. > Why is there a size argument if the input is supposed to be a singleton? > From the implementation, it seems that Sum sums all elements from the input > PCollection (which must be int, apparently?), and checks both the number of > elements and their sum. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-10186) Send error response to runner when split or progress requests are received out of order
[ https://issues.apache.org/jira/browse/BEAM-10186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath reassigned BEAM-10186: Assignee: Chamikara Madhusanka Jayalath > Send error response to runner when split or progress requests are received > out of order > --- > > Key: BEAM-10186 > URL: https://issues.apache.org/jira/browse/BEAM-10186 > Project: Beam > Issue Type: Improvement > Components: sdk-java-harness >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > > We need to send an error to the runner when split requests or progress > updates are received by the SDK harness before receiving the corresponding > bundle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10186) Send error response to runner when split or progress requests are received out of order
Chamikara Madhusanka Jayalath created BEAM-10186: Summary: Send error response to runner when split or progress requests are received out of order Key: BEAM-10186 URL: https://issues.apache.org/jira/browse/BEAM-10186 Project: Beam Issue Type: Improvement Components: sdk-java-harness Reporter: Chamikara Madhusanka Jayalath We need to send an error to the runner when split requests or progress updates are received by the SDK harness before receiving the corresponding bundle. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10151) Document how to run a Java (xlang) pipeline on the Spark portable runner
[ https://issues.apache.org/jira/browse/BEAM-10151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-10151: - Component/s: cross-language > Document how to run a Java (xlang) pipeline on the Spark portable runner > > > Key: BEAM-10151 > URL: https://issues.apache.org/jira/browse/BEAM-10151 > Project: Beam > Issue Type: New Feature > Components: cross-language, runner-spark, website >Reporter: Kyle Weaver >Priority: P2 > Labels: portability-spark > > While users will probably still prefer the classic Spark runner for most Java > pipelines, the Spark portable runner enables cross-language transforms. We > can start by providing instructions for plain Java, then add cross-language > (Java -> Python). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-3083) BigQueryIO.write() with DynamicDestinations should not call getSchema() on every element
[ https://issues.apache.org/jira/browse/BEAM-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-3083: Labels: testlabel (was: ) > BigQueryIO.write() with DynamicDestinations should not call getSchema() on > every element > > > Key: BEAM-3083 > URL: https://issues.apache.org/jira/browse/BEAM-3083 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Eugene Kirpichov >Priority: P2 > Labels: testlabel > > https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/CreateTables.java#L103 > calls getSchema() on every element. This might be very expensive if the user > expects that this will be called about once per destination. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9921) Add Go SDK tests to cross-language Flink ValidatesRunner test suite
[ https://issues.apache.org/jira/browse/BEAM-9921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9921: Description: Test suite is here: [https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_XVR_Flink/] > Add Go SDK tests to cross-language Flink ValidatesRunner test suite > --- > > Key: BEAM-9921 > URL: https://issues.apache.org/jira/browse/BEAM-9921 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > > Test suite is here: > [https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_XVR_Flink/] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9921) Add Go SDK tests to cross-language Flink ValidatesRunner test suite
[ https://issues.apache.org/jira/browse/BEAM-9921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9921: Summary: Add Go SDK tests to cross-language Flink ValidatesRunner test suite (was: Go SDK supports executing cross-language transforms on Flink runner) > Add Go SDK tests to cross-language Flink ValidatesRunner test suite > --- > > Key: BEAM-9921 > URL: https://issues.apache.org/jira/browse/BEAM-9921 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9922) Add Go SDK tests to cross-language Spark ValidatesRunner test suite
[ https://issues.apache.org/jira/browse/BEAM-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9922: Description: Test suite is here: [https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_XVR_Spark/] > Add Go SDK tests to cross-language Spark ValidatesRunner test suite > --- > > Key: BEAM-9922 > URL: https://issues.apache.org/jira/browse/BEAM-9922 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > > Test suite is here: > [https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_XVR_Spark/] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9922) Add Go SDK tests to cross-language Spark ValidatesRunner test suite
[ https://issues.apache.org/jira/browse/BEAM-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9922: Summary: Add Go SDK tests to cross-language Spark ValidatesRunner test suite (was: Go SDK supports executing cross-language transforms on Spark runner) > Add Go SDK tests to cross-language Spark ValidatesRunner test suite > --- > > Key: BEAM-9922 > URL: https://issues.apache.org/jira/browse/BEAM-9922 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9919) Add an External transform API to Go SDK
[ https://issues.apache.org/jira/browse/BEAM-9919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9919: Summary: Add an External transform API to Go SDK (was: Add an External transform to Go SDK) > Add an External transform API to Go SDK > --- > > Key: BEAM-9919 > URL: https://issues.apache.org/jira/browse/BEAM-9919 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > > This transform will be able to expand cross-language transforms by connecting > to a remote expansion service through Beam expansion API. > [https://github.com/apache/beam/blob/master/model/job-management/src/main/proto/beam_expansion_api.proto] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9923) Add an expansion service to Go SDK
[ https://issues.apache.org/jira/browse/BEAM-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9923: Parent: BEAM-9918 Issue Type: Sub-task (was: Bug) > Add an expansion service to Go SDK > -- > > Key: BEAM-9923 > URL: https://issues.apache.org/jira/browse/BEAM-9923 > Project: Beam > Issue Type: Sub-task > Components: sdk-go >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > > This will allow Go SDK to offer it's transforms to other SDKs. > This should support Beam [transform expansion > API|[https://github.com/apache/beam/blob/master/model/job-management/src/main/proto/beam_expansion_api.proto]] > and should have support for registering transforms to be offered through the > transform expansion API. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9918) Cross-language transforms support for Go SDK
[ https://issues.apache.org/jira/browse/BEAM-9918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9918: Component/s: cross-language > Cross-language transforms support for Go SDK > > > Key: BEAM-9918 > URL: https://issues.apache.org/jira/browse/BEAM-9918 > Project: Beam > Issue Type: New Feature > Components: cross-language, sdk-go >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > > This is an uber issue for tasks related to cross-language transforms support > for Go SDK. We can create sub-tasks as needed. > cc: [~lostluck] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9348) Enable CrossLanguageValidateRunner test for direct runner
[ https://issues.apache.org/jira/browse/BEAM-9348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9348: Component/s: cross-language > Enable CrossLanguageValidateRunner test for direct runner > - > > Key: BEAM-9348 > URL: https://issues.apache.org/jira/browse/BEAM-9348 > Project: Beam > Issue Type: Improvement > Components: cross-language, java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > > Enable CrossLanguageValidateRunner test for direct runner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10078) uniquify Dataflow specific jars when staging
[ https://issues.apache.org/jira/browse/BEAM-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-10078: - Issue Type: Improvement (was: Bug) > uniquify Dataflow specific jars when staging > > > Key: BEAM-10078 > URL: https://issues.apache.org/jira/browse/BEAM-10078 > Project: Beam > Issue Type: Improvement > Components: cross-language, runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > After BEAM-9383, Dataflow specific jars (dataflow-worker.jar, windmill_main) > could be overwritten when two or more jobs share the same staging location. > Since they 1) should have specific predefined names AND 2) should have unique > location for avoiding collision, they need special handling when staging > artifacts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9238) Cross-language pipeline dependency management
[ https://issues.apache.org/jira/browse/BEAM-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9238: Component/s: cross-language > Cross-language pipeline dependency management > - > > Key: BEAM-9238 > URL: https://issues.apache.org/jira/browse/BEAM-9238 > Project: Beam > Issue Type: Improvement > Components: cross-language, java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > > Meta-issue for tracking cross-language pipeline dependency management -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10125) adding cross-language KafkaIO integration test
[ https://issues.apache.org/jira/browse/BEAM-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-10125: - Component/s: cross-language > adding cross-language KafkaIO integration test > -- > > Key: BEAM-10125 > URL: https://issues.apache.org/jira/browse/BEAM-10125 > Project: Beam > Issue Type: Improvement > Components: cross-language, io-java-kafka >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Time Spent: 2h 20m > Remaining Estimate: 0h > > adding cross-language KafkaIO integration test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9719) Cross-language test suites failing due to mossing nose plugin
[ https://issues.apache.org/jira/browse/BEAM-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath resolved BEAM-9719. - Fix Version/s: Not applicable Resolution: Fixed > Cross-language test suites failing due to mossing nose plugin > - > > Key: BEAM-9719 > URL: https://issues.apache.org/jira/browse/BEAM-9719 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness, test-failures >Reporter: Chamikara Madhusanka Jayalath >Assignee: Heejong Lee >Priority: P2 > Fix For: Not applicable > > Time Spent: 20m > Remaining Estimate: 0h > > Seems like due to [https://github.com/apache/beam/pull/11307] > > [https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_XVR_Flink/] > > [https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_XVR_Spark/] > > *16:00:54* File > "/usr/local/lib/python2.7/site-packages/apache_beam/transforms/validate_runner_xlang_test.py", > line 24, in *16:00:54* from nose.plugins.attrib import > attr*16:00:54* ImportError: No module named nose.plugins.attrib > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10078) uniquify Dataflow specific jars when staging
[ https://issues.apache.org/jira/browse/BEAM-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-10078: - Component/s: cross-language > uniquify Dataflow specific jars when staging > > > Key: BEAM-10078 > URL: https://issues.apache.org/jira/browse/BEAM-10078 > Project: Beam > Issue Type: Bug > Components: cross-language, runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > After BEAM-9383, Dataflow specific jars (dataflow-worker.jar, windmill_main) > could be overwritten when two or more jobs share the same staging location. > Since they 1) should have specific predefined names AND 2) should have unique > location for avoiding collision, they need special handling when staging > artifacts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10134) Add Cross-language wrappers for Java IOs
[ https://issues.apache.org/jira/browse/BEAM-10134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-10134: - Component/s: cross-language > Add Cross-language wrappers for Java IOs > > > Key: BEAM-10134 > URL: https://issues.apache.org/jira/browse/BEAM-10134 > Project: Beam > Issue Type: Improvement > Components: cross-language, sdk-py-core >Affects Versions: Not applicable >Reporter: Piotr Szuberski >Priority: P2 > Labels: portability > Fix For: Not applicable > > > Add cross-language wrappers for Java IOs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9932) Add documentation describing cross-language test pipelines
[ https://issues.apache.org/jira/browse/BEAM-9932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9932: Component/s: (was: testing) website cross-language > Add documentation describing cross-language test pipelines > -- > > Key: BEAM-9932 > URL: https://issues.apache.org/jira/browse/BEAM-9932 > Project: Beam > Issue Type: Improvement > Components: cross-language, website >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > > We designed cross-language test pipelines [1][2] based on the discussion in > [3]. > Adding some pydocs and Java docs regarding rational behind each pipeline will > be helpful. > [1] > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/validate_runner_xlang_test.py] > [2] > [https://github.com/apache/beam/blob/master/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java] > [3] > [https://docs.google.com/document/d/1xQp0ElIV84b8OCVz8CD2hvbiWdR8w4BvWxPTZJZA6NA/edit?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9932) Add documentation describing cross-language test pipelines
[ https://issues.apache.org/jira/browse/BEAM-9932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath reassigned BEAM-9932: --- Assignee: (was: Chamikara Madhusanka Jayalath) > Add documentation describing cross-language test pipelines > -- > > Key: BEAM-9932 > URL: https://issues.apache.org/jira/browse/BEAM-9932 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > > We designed cross-language test pipelines [1][2] based on the discussion in > [3]. > Adding some pydocs and Java docs regarding rational behind each pipeline will > be helpful. > [1] > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/validate_runner_xlang_test.py] > [2] > [https://github.com/apache/beam/blob/master/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java] > [3] > [https://docs.google.com/document/d/1xQp0ElIV84b8OCVz8CD2hvbiWdR8w4BvWxPTZJZA6NA/edit?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9932) Add documentation describing cross-language test pipelines
[ https://issues.apache.org/jira/browse/BEAM-9932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9932: Component/s: (was: website) > Add documentation describing cross-language test pipelines > -- > > Key: BEAM-9932 > URL: https://issues.apache.org/jira/browse/BEAM-9932 > Project: Beam > Issue Type: Improvement > Components: cross-language >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > > We designed cross-language test pipelines [1][2] based on the discussion in > [3]. > Adding some pydocs and Java docs regarding rational behind each pipeline will > be helpful. > [1] > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/validate_runner_xlang_test.py] > [2] > [https://github.com/apache/beam/blob/master/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java] > [3] > [https://docs.google.com/document/d/1xQp0ElIV84b8OCVz8CD2hvbiWdR8w4BvWxPTZJZA6NA/edit?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9932) Add documentation describing cross-language test pipelines
[ https://issues.apache.org/jira/browse/BEAM-9932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath reassigned BEAM-9932: --- Assignee: Chamikara Madhusanka Jayalath > Add documentation describing cross-language test pipelines > -- > > Key: BEAM-9932 > URL: https://issues.apache.org/jira/browse/BEAM-9932 > Project: Beam > Issue Type: Improvement > Components: cross-language >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > > We designed cross-language test pipelines [1][2] based on the discussion in > [3]. > Adding some pydocs and Java docs regarding rational behind each pipeline will > be helpful. > [1] > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/validate_runner_xlang_test.py] > [2] > [https://github.com/apache/beam/blob/master/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java] > [3] > [https://docs.google.com/document/d/1xQp0ElIV84b8OCVz8CD2hvbiWdR8w4BvWxPTZJZA6NA/edit?usp=sharing] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-3557) Converting to runner api protos and back results in parent pointers being dropped
[ https://issues.apache.org/jira/browse/BEAM-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath resolved BEAM-3557. - Fix Version/s: Not applicable Resolution: Fixed > Converting to runner api protos and back results in parent pointers being > dropped > - > > Key: BEAM-3557 > URL: https://issues.apache.org/jira/browse/BEAM-3557 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > Labels: triaged > Fix For: Not applicable > > > We currently convert transform graph to runner API protos and back at > following location. Seems like this results in parent pointers being dropped > for all AppliedTransform objects of the graph. Looks like this doesn't have > an impact in pipeline execution for Direct and Dataflow runners though (given > that all tests pass). > [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pipeline.py#L332] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-6486) Cross-language transform API
[ https://issues.apache.org/jira/browse/BEAM-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath resolved BEAM-6486. - Fix Version/s: Not applicable Resolution: Fixed > Cross-language transform API > > > Key: BEAM-6486 > URL: https://issues.apache.org/jira/browse/BEAM-6486 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > Fix For: Not applicable > > > We should be able to refer to a transform defined in SDK X from a different > SDK Y. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-6486) Cross-language transform API
[ https://issues.apache.org/jira/browse/BEAM-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119808#comment-17119808 ] Chamikara Madhusanka Jayalath commented on BEAM-6486: - This is already available. Closing. > Cross-language transform API > > > Key: BEAM-6486 > URL: https://issues.apache.org/jira/browse/BEAM-6486 > Project: Beam > Issue Type: New Feature > Components: beam-model, runner-core, sdk-java-core, sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > > We should be able to refer to a transform defined in SDK X from a different > SDK Y. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-6683: Component/s: cross-language > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: cross-language, testing >Reporter: Chamikara Madhusanka Jayalath >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.15.0 > > Time Spent: 23.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-8019: Component/s: cross-language > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: cross-language, sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 24h > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8603) Add Python SqlTransform MVP
[ https://issues.apache.org/jira/browse/BEAM-8603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-8603: Component/s: cross-language > Add Python SqlTransform MVP > --- > > Key: BEAM-8603 > URL: https://issues.apache.org/jira/browse/BEAM-8603 > Project: Beam > Issue Type: Improvement > Components: cross-language, dsl-sql, sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 12h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7738) Support PubSubIO to be configured externally for use with other SDKs
[ https://issues.apache.org/jira/browse/BEAM-7738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-7738: Component/s: cross-language > Support PubSubIO to be configured externally for use with other SDKs > > > Key: BEAM-7738 > URL: https://issues.apache.org/jira/browse/BEAM-7738 > Project: Beam > Issue Type: New Feature > Components: cross-language, io-java-gcp, runner-flink, sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: P2 > Labels: portability > Time Spent: 9h 50m > Remaining Estimate: 0h > > Now that KafkaIO is supported via the external transform API (BEAM-7029) we > should add support for PubSub. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7029) Support KafkaIO to be configured externally for use with other SDKs
[
https://issues.apache.org/jira/browse/BEAM-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chamikara Madhusanka Jayalath updated BEAM-7029:
Component/s: cross-language
> Support KafkaIO to be configured externally for use with other SDKs
> ---
>
> Key: BEAM-7029
> URL: https://issues.apache.org/jira/browse/BEAM-7029
> Project: Beam
> Issue Type: New Feature
> Components: cross-language, io-java-kafka, runner-flink, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: P2
> Fix For: 2.13.0
>
> Time Spent: 16h 20m
> Remaining Estimate: 0h
>
> As of BEAM-6730, we can externally configure existing transforms from SDKs.
> We should add more useful transforms then just {{GenerateSequence}}.
> {{KafkaIO}} is a good candidate.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-7886) Make row coder a standard coder and implement in python
[ https://issues.apache.org/jira/browse/BEAM-7886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-7886: Component/s: cross-language > Make row coder a standard coder and implement in python > --- > > Key: BEAM-7886 > URL: https://issues.apache.org/jira/browse/BEAM-7886 > Project: Beam > Issue Type: Improvement > Components: beam-model, cross-language, sdk-java-core, sdk-py-core >Reporter: Brian Hulette >Assignee: Brian Hulette >Priority: P2 > Fix For: 2.18.0 > > Time Spent: 16h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7029) Support KafkaIO to be configured externally for use with other SDKs
[
https://issues.apache.org/jira/browse/BEAM-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118975#comment-17118975
]
Chamikara Madhusanka Jayalath commented on BEAM-7029:
-
Will do. Thanks.
> Support KafkaIO to be configured externally for use with other SDKs
> ---
>
> Key: BEAM-7029
> URL: https://issues.apache.org/jira/browse/BEAM-7029
> Project: Beam
> Issue Type: New Feature
> Components: io-java-kafka, runner-flink, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: P2
> Fix For: 2.13.0
>
> Time Spent: 16h 20m
> Remaining Estimate: 0h
>
> As of BEAM-6730, we can externally configure existing transforms from SDKs.
> We should add more useful transforms then just {{GenerateSequence}}.
> {{KafkaIO}} is a good candidate.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-10077) using filename + hash instead of UUID for staging name
[ https://issues.apache.org/jira/browse/BEAM-10077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath resolved BEAM-10077. -- Resolution: Fixed > using filename + hash instead of UUID for staging name > -- > > Key: BEAM-10077 > URL: https://issues.apache.org/jira/browse/BEAM-10077 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Recent change BEAM-9383 disabled the artifact caching logic for GCS by object > names. Changing staging name generation from UUID to filename + hash will > re-enable the artifact caching so we can avoid re-uploading same artifact. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7029) Support KafkaIO to be configured externally for use with other SDKs
[
https://issues.apache.org/jira/browse/BEAM-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118945#comment-17118945
]
Chamikara Madhusanka Jayalath commented on BEAM-7029:
-
Dataflow will support cross-language KafkaIO starting Beam 2.22.0. Will add a
document better describing cross-language KafkaIO in general.
> Support KafkaIO to be configured externally for use with other SDKs
> ---
>
> Key: BEAM-7029
> URL: https://issues.apache.org/jira/browse/BEAM-7029
> Project: Beam
> Issue Type: New Feature
> Components: io-java-kafka, runner-flink, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: P2
> Fix For: 2.13.0
>
> Time Spent: 16h 20m
> Remaining Estimate: 0h
>
> As of BEAM-6730, we can externally configure existing transforms from SDKs.
> We should add more useful transforms then just {{GenerateSequence}}.
> {{KafkaIO}} is a good candidate.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (BEAM-10077) using filename + hash instead of UUID for staging name
[ https://issues.apache.org/jira/browse/BEAM-10077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath reopened BEAM-10077: -- > using filename + hash instead of UUID for staging name > -- > > Key: BEAM-10077 > URL: https://issues.apache.org/jira/browse/BEAM-10077 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Recent change BEAM-9383 disabled the artifact caching logic for GCS by object > names. Changing staging name generation from UUID to filename + hash will > re-enable the artifact caching so we can avoid re-uploading same artifact. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10077) using filename + hash instead of UUID for staging name
[ https://issues.apache.org/jira/browse/BEAM-10077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118176#comment-17118176 ] Chamikara Madhusanka Jayalath commented on BEAM-10077: -- Seems like [https://github.com/apache/beam/pull/11813] without [https://github.com/apache/beam/pull/11771] breaks Dataflow for some cases. Sent out cherry-pick [https://github.com/apache/beam/pull/11843] > using filename + hash instead of UUID for staging name > -- > > Key: BEAM-10077 > URL: https://issues.apache.org/jira/browse/BEAM-10077 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Recent change BEAM-9383 disabled the artifact caching logic for GCS by object > names. Changing staging name generation from UUID to filename + hash will > re-enable the artifact caching so we can avoid re-uploading same artifact. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3788) Implement a Kafka IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117906#comment-17117906 ] Chamikara Madhusanka Jayalath commented on BEAM-3788: - KafkaIO is expected to be available as a cross-language transforms for Dataflow with Beam 2.22. Flink/Spark already support this. So closing the JIRA. We can track further improvements to the cross-language KafkaIO in other JIRAs. > Implement a Kafka IO for Python SDK > --- > > Key: BEAM-3788 > URL: https://issues.apache.org/jira/browse/BEAM-3788 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > Fix For: 2.22.0 > > > Java KafkaIO will be made available to Python users as a cross-language > transform. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-3788) Implement a Kafka IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath resolved BEAM-3788. - Resolution: Fixed > Implement a Kafka IO for Python SDK > --- > > Key: BEAM-3788 > URL: https://issues.apache.org/jira/browse/BEAM-3788 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > Fix For: 2.22.0 > > > Java KafkaIO will be made available to Python users as a cross-language > transform. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-3788) Implement a Kafka IO for Python SDK
[ https://issues.apache.org/jira/browse/BEAM-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-3788: Fix Version/s: 2.22.0 > Implement a Kafka IO for Python SDK > --- > > Key: BEAM-3788 > URL: https://issues.apache.org/jira/browse/BEAM-3788 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > Fix For: 2.22.0 > > > Java KafkaIO will be made available to Python users as a cross-language > transform. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116875#comment-17116875 ] Chamikara Madhusanka Jayalath commented on BEAM-9383: - Steve, why would two workers jars be staged at the same location ? Also, I'm not sure why we started picking up new jars from JRE libraries for staging. Weren't we staging everything in the CLASSPATH anyways ? > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 12h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113628#comment-17113628 ] Chamikara Madhusanka Jayalath commented on BEAM-8019: - Marking this as fixed since the basic framework is in place. > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P1 > Fix For: 2.22.0 > > Time Spent: 22h > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-8019: Priority: P2 (was: P1) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 22h > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath resolved BEAM-8019. - Resolution: Fixed > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P1 > Fix For: 2.22.0 > > Time Spent: 22h > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-8019: Fix Version/s: 2.22.0 > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P1 > Fix For: 2.22.0 > > Time Spent: 22h > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9383: Fix Version/s: 2.22.0 > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 12h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath resolved BEAM-9383. - Resolution: Fixed > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 12h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10057) Failure when getting watermark "getWatermark is never meant to be invoked."
[ https://issues.apache.org/jira/browse/BEAM-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-10057: - Priority: P1 (was: P0) > Failure when getting watermark "getWatermark is never meant to be invoked." > --- > > Key: BEAM-10057 > URL: https://issues.apache.org/jira/browse/BEAM-10057 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Luke Cwik >Priority: P1 > Fix For: 2.22.0 > > > generic::unknown: org.apache.beam.sdk.util.UserCodeException: > java.lang.UnsupportedOperationException: getWatermark is never meant to be > invoked. at > org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36) at > org.apache.beam.sdk.io.Read$UnboundedSourceAsSDFWrapperFn$DoFnInvoker.invokeProcessElement(Unknown > Source) at > org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForElementAndRestriction(FnApiDoFnRunner.java:838) > at > org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForSizedElementAndRestriction(FnApiDoFnRunner.java:808) > at > org.apache.beam.fn.harness.FnApiDoFnRunner.access$200(FnApiDoFnRunner.java:132) > at > org.apache.beam.fn.harness.FnApiDoFnRunner$Factory$2.accept(FnApiDoFnRunner.java:226) > at > org.apache.beam.fn.harness.FnApiDoFnRunner$Factory$2.accept(FnApiDoFnRunner.java:223) > at > org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:216) > at > org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:179) > at > org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer(BeamFnDataReadRunner.java:204) > at > org.apache.beam.fn.harness.data.QueueingBeamFnDataClient.drainAndBlock(QueueingBeamFnDataClient.java:106) > at > org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:295) > at > org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:173) > at > org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:157) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Seems to be a breakage in SDF due to a recent change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10057) Failure when getting watermark "getWatermark is never meant to be invoked."
[ https://issues.apache.org/jira/browse/BEAM-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113479#comment-17113479 ] Chamikara Madhusanka Jayalath commented on BEAM-10057: -- Not sure. Hit this when running x-lang Kafka on the release branch. Luke is looking into it. > Failure when getting watermark "getWatermark is never meant to be invoked." > --- > > Key: BEAM-10057 > URL: https://issues.apache.org/jira/browse/BEAM-10057 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Luke Cwik >Priority: P0 > Fix For: 2.22.0 > > > generic::unknown: org.apache.beam.sdk.util.UserCodeException: > java.lang.UnsupportedOperationException: getWatermark is never meant to be > invoked. at > org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36) at > org.apache.beam.sdk.io.Read$UnboundedSourceAsSDFWrapperFn$DoFnInvoker.invokeProcessElement(Unknown > Source) at > org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForElementAndRestriction(FnApiDoFnRunner.java:838) > at > org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForSizedElementAndRestriction(FnApiDoFnRunner.java:808) > at > org.apache.beam.fn.harness.FnApiDoFnRunner.access$200(FnApiDoFnRunner.java:132) > at > org.apache.beam.fn.harness.FnApiDoFnRunner$Factory$2.accept(FnApiDoFnRunner.java:226) > at > org.apache.beam.fn.harness.FnApiDoFnRunner$Factory$2.accept(FnApiDoFnRunner.java:223) > at > org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:216) > at > org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:179) > at > org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer(BeamFnDataReadRunner.java:204) > at > org.apache.beam.fn.harness.data.QueueingBeamFnDataClient.drainAndBlock(QueueingBeamFnDataClient.java:106) > at > org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:295) > at > org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:173) > at > org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:157) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Seems to be a breakage in SDF due to a recent change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10057) Failure when getting watermark "getWatermark is never meant to be invoked."
[ https://issues.apache.org/jira/browse/BEAM-10057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113473#comment-17113473 ] Chamikara Madhusanka Jayalath commented on BEAM-10057: -- cc: [~bhulette] > Failure when getting watermark "getWatermark is never meant to be invoked." > --- > > Key: BEAM-10057 > URL: https://issues.apache.org/jira/browse/BEAM-10057 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Luke Cwik >Priority: P0 > Fix For: 2.22.0 > > > generic::unknown: org.apache.beam.sdk.util.UserCodeException: > java.lang.UnsupportedOperationException: getWatermark is never meant to be > invoked. at > org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36) at > org.apache.beam.sdk.io.Read$UnboundedSourceAsSDFWrapperFn$DoFnInvoker.invokeProcessElement(Unknown > Source) at > org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForElementAndRestriction(FnApiDoFnRunner.java:838) > at > org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForSizedElementAndRestriction(FnApiDoFnRunner.java:808) > at > org.apache.beam.fn.harness.FnApiDoFnRunner.access$200(FnApiDoFnRunner.java:132) > at > org.apache.beam.fn.harness.FnApiDoFnRunner$Factory$2.accept(FnApiDoFnRunner.java:226) > at > org.apache.beam.fn.harness.FnApiDoFnRunner$Factory$2.accept(FnApiDoFnRunner.java:223) > at > org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:216) > at > org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:179) > at > org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer(BeamFnDataReadRunner.java:204) > at > org.apache.beam.fn.harness.data.QueueingBeamFnDataClient.drainAndBlock(QueueingBeamFnDataClient.java:106) > at > org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:295) > at > org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:173) > at > org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:157) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Seems to be a breakage in SDF due to a recent change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10057) Failure when getting watermark "getWatermark is never meant to be invoked."
Chamikara Madhusanka Jayalath created BEAM-10057: Summary: Failure when getting watermark "getWatermark is never meant to be invoked." Key: BEAM-10057 URL: https://issues.apache.org/jira/browse/BEAM-10057 Project: Beam Issue Type: Bug Components: sdk-java-core Reporter: Chamikara Madhusanka Jayalath Assignee: Luke Cwik Fix For: 2.22.0 generic::unknown: org.apache.beam.sdk.util.UserCodeException: java.lang.UnsupportedOperationException: getWatermark is never meant to be invoked. at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36) at org.apache.beam.sdk.io.Read$UnboundedSourceAsSDFWrapperFn$DoFnInvoker.invokeProcessElement(Unknown Source) at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForElementAndRestriction(FnApiDoFnRunner.java:838) at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForSizedElementAndRestriction(FnApiDoFnRunner.java:808) at org.apache.beam.fn.harness.FnApiDoFnRunner.access$200(FnApiDoFnRunner.java:132) at org.apache.beam.fn.harness.FnApiDoFnRunner$Factory$2.accept(FnApiDoFnRunner.java:226) at org.apache.beam.fn.harness.FnApiDoFnRunner$Factory$2.accept(FnApiDoFnRunner.java:223) at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:216) at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:179) at org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer(BeamFnDataReadRunner.java:204) at org.apache.beam.fn.harness.data.QueueingBeamFnDataClient.drainAndBlock(QueueingBeamFnDataClient.java:106) at org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:295) at org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:173) at org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:157) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Seems to be a breakage in SDF due to a recent change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9383: Fix Version/s: (was: 2.22.0) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Time Spent: 12h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113468#comment-17113468 ] Chamikara Madhusanka Jayalath commented on BEAM-9383: - Removing this from the blockers list. > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 12h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9383: Priority: P2 (was: P0) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > Fix For: 2.22.0 > > Time Spent: 12h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112678#comment-17112678 ] Chamikara Madhusanka Jayalath commented on BEAM-9383: - Seems like it's picking up jars from the Java runtime. nashorn.jar ldrdata.jar jfxrt.jar dnsns.jar localedata.jar MRJToolkit.jar beam-sdks-java-io-expansion-service-2.22.0-SNAPSHOT.jar Also part of the problem is that beam-sdks-java-io-expansion-service-2.22.0-SNAPSHOT.jar is 51MB and takes a long time to stage. We stage two of each of the above since we have both Kafka read and write transforms in the pipeline. Can we somehow exclude jars from the Java runtime here ? [https://github.com/apache/beam/blob/master/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java#L324] > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 12h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112649#comment-17112649 ] Chamikara Madhusanka Jayalath commented on BEAM-9383: - Note that I'm running from Beam HEAD without specifying additional dependencies or an expansion service. Pipeline is here: [https://paste.ofcode.org/32sxtbEGuzqbw4d7PKMiC6V] > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 12h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112645#comment-17112645 ] Chamikara Madhusanka Jayalath edited comment on BEAM-9383 at 5/20/20, 10:51 PM: I tried running a Kafka pipeline on Dataflow and I see a lot of jars being staged during pipeline submission. INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/pipeline.pb in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/596ab8b3-840a-43ff-accb-8f6815e1a302.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/596ab8b3-840a-43ff-accb-8f6815e1a302.jar in 24 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/6f80255b-453f-4ad8-aa28-7e40fdfeedac.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/6f80255b-453f-4ad8-aa28-7e40fdfeedac.jar in 22 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/2704b169-8874-4163-9f3c-ab8765f3c330.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/2704b169-8874-4163-9f3c-ab8765f3c330.jar in 69 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/40bd912f-ce2f-45a8-9625-019b85c46cc7.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/40bd912f-ce2f-45a8-9625-019b85c46cc7.jar in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/9d1bfb42-518d-4cc7-9a3a-7a8ea792ce6f.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/9d1bfb42-518d-4cc7-9a3a-7a8ea792ce6f.jar in 8 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/7e1e7095-32d6-4ea6-b9a0-aa5e2ffdbb31.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/7e1e7095-32d6-4ea6-b9a0-aa5e2ffdbb31.jar in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/926b735c-552f-4a3a-9e81-f0fe8162ce26.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/926b735c-552f-4a3a-9e81-f0fe8162ce26.jar in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/e9f829ba-eadf-4ae4-98c4-492238cb9998.jar... ... Ideally there should be only one jar, beam-sdks-java-io-expansion-service-2.22.0-SNAPSHOT.jar Any idea where additional jars are coming from. Also can we use names of jars instread of URLs so that we can easily identify what these are ? cc: [~robertwb] [~lcwik] was (Author: chamikara): I tried running a Kafka pipeline on Dataflow and I see a lot of jars being staged during pipeline submission. INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/pipeline.pb in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/596ab8b3-840a-43ff-accb-8f6815e1a302.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/596ab8b3-840a-43ff-accb-8f6815e1a302.jar in 24 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/6f80255b-453f-4ad8-aa28-7e40fdfeedac.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to
[jira] [Reopened] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath reopened BEAM-9383: - > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 12h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112645#comment-17112645 ] Chamikara Madhusanka Jayalath commented on BEAM-9383: - I tried running a Kafka pipeline on Dataflow and I see a lot of jars being staged during pipeline submission. INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/pipeline.pb in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/596ab8b3-840a-43ff-accb-8f6815e1a302.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/596ab8b3-840a-43ff-accb-8f6815e1a302.jar in 24 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/6f80255b-453f-4ad8-aa28-7e40fdfeedac.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/6f80255b-453f-4ad8-aa28-7e40fdfeedac.jar in 22 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/2704b169-8874-4163-9f3c-ab8765f3c330.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/2704b169-8874-4163-9f3c-ab8765f3c330.jar in 69 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/40bd912f-ce2f-45a8-9625-019b85c46cc7.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/40bd912f-ce2f-45a8-9625-019b85c46cc7.jar in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/9d1bfb42-518d-4cc7-9a3a-7a8ea792ce6f.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/9d1bfb42-518d-4cc7-9a3a-7a8ea792ce6f.jar in 8 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/7e1e7095-32d6-4ea6-b9a0-aa5e2ffdbb31.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/7e1e7095-32d6-4ea6-b9a0-aa5e2ffdbb31.jar in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/926b735c-552f-4a3a-9e81-f0fe8162ce26.jar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/926b735c-552f-4a3a-9e81-f0fe8162ce26.jar in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://clouddfe-chamikara/staging/python-222-wc-chamikara.1590014685.641805/e9f829ba-eadf-4ae4-98c4-492238cb9998.jar... ... Ideally there should be only one jar, beam-sdks-java-io-expansion-service-2.22.0-SNAPSHOT.ja Any idea where additional jars are coming from. Also can we use names of jars instread of URLs so that we can easily identify what these are ? cc: [~robertwb] [~lcwik] > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 12h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-10045) Reduce logging related to ratelimitExceeded error for BQ sink when performing streaming inserts
[ https://issues.apache.org/jira/browse/BEAM-10045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath reassigned BEAM-10045: Assignee: Chamikara Madhusanka Jayalath > Reduce logging related to ratelimitExceeded error for BQ sink when performing > streaming inserts > --- > > Key: BEAM-10045 > URL: https://issues.apache.org/jira/browse/BEAM-10045 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > > These errors are usually temporary and pipelines may recover. > > We can consider not logging till we backoff for a certain amount of time here. > [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L792] > > cc: [~reuvenlax] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10045) Reduce logging related to ratelimitExceeded error for BQ sink when performing streaming inserts
Chamikara Madhusanka Jayalath created BEAM-10045: Summary: Reduce logging related to ratelimitExceeded error for BQ sink when performing streaming inserts Key: BEAM-10045 URL: https://issues.apache.org/jira/browse/BEAM-10045 Project: Beam Issue Type: Improvement Components: io-java-gcp Reporter: Chamikara Madhusanka Jayalath These errors are usually temporary and pipelines may recover. We can consider not logging till we backoff for a certain amount of time here. [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L792] cc: [~reuvenlax] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-8019: Fix Version/s: (was: 2.22.0) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P1 > Time Spent: 19h 40m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111546#comment-17111546 ] Chamikara Madhusanka Jayalath commented on BEAM-8019: - Yeah, removing from the blockers list. Thanks. > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P1 > Fix For: 2.22.0 > > Time Spent: 19h 40m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111376#comment-17111376 ] Chamikara Madhusanka Jayalath commented on BEAM-8019: - Temporarily making this a blocker for 2.22 since we need to get [https://github.com/apache/beam/pull/11740] in. cc: [~bhulette] > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P1 > Fix For: 2.22.0 > > Time Spent: 19h 10m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-8019: Fix Version/s: 2.22.0 > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P1 > Fix For: 2.22.0 > > Time Spent: 19h 10m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110523#comment-17110523 ] Chamikara Madhusanka Jayalath commented on BEAM-9383: - Work after [https://github.com/apache/beam/pull/11039] (updating Dataflow to separate dependencies for multiple environments) is not a blocker. > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 9h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110521#comment-17110521 ] Chamikara Madhusanka Jayalath commented on BEAM-9383: - Changing to a blocker to get [https://github.com/apache/beam/pull/11039] into Beam 2.22.0. > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 9h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-8019: Priority: P1 (was: P0) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P1 > Fix For: 2.22.0 > > Time Spent: 18h 10m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-8019: Fix Version/s: (was: 2.22.0) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P1 > Time Spent: 18h 10m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9383: Priority: P0 (was: P2) > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Time Spent: 9h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9383) Staging Dataflow artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-9383: Fix Version/s: 2.22.0 > Staging Dataflow artifacts from environment > --- > > Key: BEAM-9383 > URL: https://issues.apache.org/jira/browse/BEAM-9383 > Project: Beam > Issue Type: Sub-task > Components: java-fn-execution >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 9h > Remaining Estimate: 0h > > Staging Dataflow artifacts from environment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath reassigned BEAM-8019: --- Assignee: Chamikara Madhusanka Jayalath (was: Heejong Lee) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 18h 10m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-8019: Comment: was deleted (was: Changing to a blocker to get [https://github.com/apache/beam/pull/11039] into Beam 2.22.0.) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 18h 10m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-8019: Fix Version/s: 2.22.0 > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Heejong Lee >Priority: P0 > Fix For: 2.22.0 > > Time Spent: 17h 50m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath updated BEAM-8019: Priority: P0 (was: P2) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Heejong Lee >Priority: P0 > Time Spent: 17h 50m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110386#comment-17110386 ] Chamikara Madhusanka Jayalath commented on BEAM-8019: - Changing to a blocker to get [https://github.com/apache/beam/pull/11039] into Beam 2.22.0. > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Heejong Lee >Priority: P2 > Time Spent: 17h 50m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8019) Support cross-language transforms for DataflowRunner
[ https://issues.apache.org/jira/browse/BEAM-8019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath reassigned BEAM-8019: --- Assignee: Heejong Lee (was: Chamikara Madhusanka Jayalath) > Support cross-language transforms for DataflowRunner > > > Key: BEAM-8019 > URL: https://issues.apache.org/jira/browse/BEAM-8019 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Heejong Lee >Priority: P2 > Time Spent: 17h 50m > Remaining Estimate: 0h > > This is to capture the Beam changes needed for this task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chamikara Madhusanka Jayalath updated BEAM-9745:
Fix Version/s: 2.22.0
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Kenneth Knowles
>Priority: P0
> Labels: currently-failing
> Fix For: 2.22.0
>
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of tracking down a specific breaking change.
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See [Apache Beam
> Post-Commit
> Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109176#comment-17109176
]
Chamikara Madhusanka Jayalath edited comment on BEAM-9745 at 5/16/20, 7:04 PM:
---
I noticed that we add harness jar to the container in addition to jars received
through the retrieval service.
[https://github.com/apache/beam/blob/master/sdks/java/container/boot.go#L122]
Could it be that we end up with conflicting versions of the same set of classes
due to this ?
was (Author: chamikara):
I noticed that we add harness jar to the container in addition to jars received
through the retrieval service.
[https://github.com/apache/beam/blob/master/sdks/java/container/boot.go#L122]
Could it be that we end up with conflicting versions of the same set of classes
for Dataflow due to this ?
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Kenneth Knowles
>Priority: P0
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of
[jira] [Commented] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109176#comment-17109176
]
Chamikara Madhusanka Jayalath commented on BEAM-9745:
-
I noticed that we add harness jar to the container in addition to jars received
through the retrieval service.
[https://github.com/apache/beam/blob/master/sdks/java/container/boot.go#L122]
Could it be that we end up with conflicting versions of the same set of classes
for Dataflow due to this ?
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Kenneth Knowles
>Priority: P0
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of tracking down a specific breaking change.
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping
[jira] [Updated] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chamikara Madhusanka Jayalath updated BEAM-9745:
Priority: P0 (was: P2)
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Kenneth Knowles
>Priority: P0
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of tracking down a specific breaking change.
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See [Apache Beam
> Post-Commit
> Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109113#comment-17109113
]
Chamikara Madhusanka Jayalath commented on BEAM-9745:
-
I've seen this when running x-lang KafkaIO tests on Dataflow and I think this
might be an issue with the SDK Harness is setup not an issue with BigQuery as
Boyuan mentioned. Given that we hope to use Java SDK Harness with with UW for
x-lang in Beam 2.22 we should make this a blocker for Beam 2.22. [~kenn] and
[~altay] can we prioritize this ?
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Pablo Estrada
>Priority: P2
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of tracking down a specific breaking change.
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
>
[jira] [Assigned] (BEAM-9745) [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to deserialize Custom DoFns and Custom Coders.
[
https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chamikara Madhusanka Jayalath reassigned BEAM-9745:
---
Assignee: Kenneth Knowles (was: Pablo Estrada)
> [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to
> deserialize Custom DoFns and Custom Coders.
> -
>
> Key: BEAM-9745
> URL: https://issues.apache.org/jira/browse/BEAM-9745
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, java-fn-execution, sdk-java-harness,
> test-failures
>Reporter: Daniel Oliveira
>Assignee: Kenneth Knowles
>Priority: P2
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/]
> * [Gradle Build
> Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project]
> Initial investigation:
> The bug appears to be popping up on BigQuery tests mostly, but also a
> BigTable and a Datastore test.
> Here's an example stacktrace of the two errors, showing _only_ the error
> messages themselves. Source:
> [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe]
> {noformat}
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -191:
> java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
> Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -191: java.lang.IllegalArgumentException: unable to deserialize
> Custom DoFn With Execution Info
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error
> received from SDK harness for instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> Caused by: java.lang.RuntimeException: Error received from SDK harness for
> instruction -206:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
>
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by:
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes
> ...
> Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom
> Coder Bytes
> ...
> Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder
> ...
> {noformat}
> Update: Looks like this has been failing as far back as [Apr
> 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/]
> after a long period where the test was consistently timing out since [Mar
> 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/].
> So it's hard to narrow down what commit may have caused this. Plus, the test
> was failing due to a completely different BigQuery failure before anyway, so
> it seems like this test will need to be completely fixed from scratch,
> instead of tracking down a specific breaking change.
>
> _After you've filled out the above details, please [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See [Apache Beam
> Post-Commit
> Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-10012) Update Python SDK to construct Dataflow job requests from Beam runner API protos
Chamikara Madhusanka Jayalath created BEAM-10012: Summary: Update Python SDK to construct Dataflow job requests from Beam runner API protos Key: BEAM-10012 URL: https://issues.apache.org/jira/browse/BEAM-10012 Project: Beam Issue Type: New Feature Components: sdk-py-core Reporter: Chamikara Madhusanka Jayalath Currently, portable runners are expected to do following when constructing a runner specific job. SDK specific job graph -> Beam runner API proto -> Runner specific job request Portable Spark and Flink follow this model. Dataflow does following. SDK specific job graph -> Runner specific job request Beam runner API proto -> Upload to GCS -> Download at workers We should update Dataflow to follow the prior path which is expected to be followed by all portable runners. This will simplify the cross-language transforms job construction logic for Dataflow. We can probably start this by just implementing this for Python SDK for portions of pipeline received by expanding external transforms. cc: [~lcwik] [~robertwb] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9932) Add documentation describing cross-language test pipelines
Chamikara Madhusanka Jayalath created BEAM-9932: --- Summary: Add documentation describing cross-language test pipelines Key: BEAM-9932 URL: https://issues.apache.org/jira/browse/BEAM-9932 Project: Beam Issue Type: Improvement Components: testing Reporter: Chamikara Madhusanka Jayalath Assignee: Chamikara Madhusanka Jayalath We designed cross-language test pipelines [1][2] based on the discussion in [3]. Adding some pydocs and Java docs regarding rational behind each pipeline will be helpful. [1] [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/validate_runner_xlang_test.py] [2] [https://github.com/apache/beam/blob/master/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java] [3] [https://docs.google.com/document/d/1xQp0ElIV84b8OCVz8CD2hvbiWdR8w4BvWxPTZJZA6NA/edit?usp=sharing] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9907) apache_beam.transforms.external_test.ExternalTransformTest.test_nested flaky
[
https://issues.apache.org/jira/browse/BEAM-9907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101938#comment-17101938
]
Chamikara Madhusanka Jayalath commented on BEAM-9907:
-
Thanks Brian. Assigning to Jira to you for now.
> apache_beam.transforms.external_test.ExternalTransformTest.test_nested flaky
>
>
> Key: BEAM-9907
> URL: https://issues.apache.org/jira/browse/BEAM-9907
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Ning Kang
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
>
> Example test failures:
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/12682/
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/12684/
> A stacktrace
> {code:bash}
> apache_beam.transforms.external_test.ExternalTransformTest.test_nested (from
> py37-cloud)
> Failing for the past 1 build (Since Failed#12682 )
> Took 54 ms.
> Error Message
> google.protobuf.json_format.ParseError: Unexpected type for Value message.
> Stacktrace
> self = testMethod=test_nested>
> def test_nested(self):
> with beam.Pipeline() as p:
> > assert_that(p | FibTransform(6), equal_to([8]))
> apache_beam/transforms/external_test.py:250:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> apache_beam/transforms/ptransform.py:562: in __ror__
> result = p.apply(self, pvalueish, label)
> apache_beam/pipeline.py:651: in apply
> pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
> apache_beam/runners/runner.py:198: in apply
> return m(transform, input, options)
> apache_beam/runners/runner.py:228: in apply_PTransform
> return transform.expand(input)
> apache_beam/runners/portability/expansion_service_test.py:257: in expand
> expansion_service.ExpansionServiceServicer())
> apache_beam/pvalue.py:140: in __or__
> return self.pipeline.apply(ptransform, self)
> apache_beam/pipeline.py:598: in apply
> transform.transform, pvalueish, label or transform.label)
> apache_beam/pipeline.py:608: in apply
> return self.apply(transform, pvalueish)
> apache_beam/pipeline.py:651: in apply
> pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
> apache_beam/runners/runner.py:198: in apply
> return m(transform, input, options)
> apache_beam/runners/runner.py:228: in apply_PTransform
> return transform.expand(input)
> apache_beam/transforms/external.py:322: in expand
> pipeline_options=job_utils.pipeline_options_dict_to_struct(options))
> apache_beam/runners/job/utils.py:38: in pipeline_options_dict_to_struct
> v in options.items() if v is not None
> apache_beam/runners/job/utils.py:44: in dict_to_struct
> return json_format.ParseDict(dict_obj, struct_pb2.Struct())
> target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:450:
> in ParseDict
> parser.ConvertMessage(js_dict, message)
> target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:479:
> in ConvertMessage
> methodcaller(_WKTJSONMETHODS[full_name][1], value, message)(self)
> target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:667:
> in _ConvertStructMessage
> self._ConvertValueMessage(value[key], message.fields[key])
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self =
> value = 0x7f35a4c00390>
> message =
> def _ConvertValueMessage(self, value, message):
> """Convert a JSON representation into Value message."""
> if isinstance(value, dict):
> self._ConvertStructMessage(value, message.struct_value)
> elif isinstance(value, list):
> self. _ConvertListValueMessage(value, message.list_value)
> elif value is None:
> message.null_value = 0
> elif isinstance(value, bool):
> message.bool_value = value
> elif isinstance(value, six.string_types):
> message.string_value = value
> elif isinstance(value, _INT_OR_FLOAT):
> message.number_value = value
> else:
> > raise ParseError('Unexpected type for Value message.')
> E google.protobuf.json_format.ParseError: Unexpected type for Value
> message.
> target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:647:
> ParseError
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9907) apache_beam.transforms.external_test.ExternalTransformTest.test_nested flaky
[
https://issues.apache.org/jira/browse/BEAM-9907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chamikara Madhusanka Jayalath reassigned BEAM-9907:
---
Assignee: Brian Hulette (was: Chamikara Madhusanka Jayalath)
> apache_beam.transforms.external_test.ExternalTransformTest.test_nested flaky
>
>
> Key: BEAM-9907
> URL: https://issues.apache.org/jira/browse/BEAM-9907
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Ning Kang
>Assignee: Brian Hulette
>Priority: Major
>
> Example test failures:
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/12682/
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/12684/
> A stacktrace
> {code:bash}
> apache_beam.transforms.external_test.ExternalTransformTest.test_nested (from
> py37-cloud)
> Failing for the past 1 build (Since Failed#12682 )
> Took 54 ms.
> Error Message
> google.protobuf.json_format.ParseError: Unexpected type for Value message.
> Stacktrace
> self = testMethod=test_nested>
> def test_nested(self):
> with beam.Pipeline() as p:
> > assert_that(p | FibTransform(6), equal_to([8]))
> apache_beam/transforms/external_test.py:250:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> apache_beam/transforms/ptransform.py:562: in __ror__
> result = p.apply(self, pvalueish, label)
> apache_beam/pipeline.py:651: in apply
> pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
> apache_beam/runners/runner.py:198: in apply
> return m(transform, input, options)
> apache_beam/runners/runner.py:228: in apply_PTransform
> return transform.expand(input)
> apache_beam/runners/portability/expansion_service_test.py:257: in expand
> expansion_service.ExpansionServiceServicer())
> apache_beam/pvalue.py:140: in __or__
> return self.pipeline.apply(ptransform, self)
> apache_beam/pipeline.py:598: in apply
> transform.transform, pvalueish, label or transform.label)
> apache_beam/pipeline.py:608: in apply
> return self.apply(transform, pvalueish)
> apache_beam/pipeline.py:651: in apply
> pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
> apache_beam/runners/runner.py:198: in apply
> return m(transform, input, options)
> apache_beam/runners/runner.py:228: in apply_PTransform
> return transform.expand(input)
> apache_beam/transforms/external.py:322: in expand
> pipeline_options=job_utils.pipeline_options_dict_to_struct(options))
> apache_beam/runners/job/utils.py:38: in pipeline_options_dict_to_struct
> v in options.items() if v is not None
> apache_beam/runners/job/utils.py:44: in dict_to_struct
> return json_format.ParseDict(dict_obj, struct_pb2.Struct())
> target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:450:
> in ParseDict
> parser.ConvertMessage(js_dict, message)
> target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:479:
> in ConvertMessage
> methodcaller(_WKTJSONMETHODS[full_name][1], value, message)(self)
> target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:667:
> in _ConvertStructMessage
> self._ConvertValueMessage(value[key], message.fields[key])
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self =
> value = 0x7f35a4c00390>
> message =
> def _ConvertValueMessage(self, value, message):
> """Convert a JSON representation into Value message."""
> if isinstance(value, dict):
> self._ConvertStructMessage(value, message.struct_value)
> elif isinstance(value, list):
> self. _ConvertListValueMessage(value, message.list_value)
> elif value is None:
> message.null_value = 0
> elif isinstance(value, bool):
> message.bool_value = value
> elif isinstance(value, six.string_types):
> message.string_value = value
> elif isinstance(value, _INT_OR_FLOAT):
> message.number_value = value
> else:
> > raise ParseError('Unexpected type for Value message.')
> E google.protobuf.json_format.ParseError: Unexpected type for Value
> message.
> target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:647:
> ParseError
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9907) apache_beam.transforms.external_test.ExternalTransformTest.test_nested flaky
[
https://issues.apache.org/jira/browse/BEAM-9907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101912#comment-17101912
]
Chamikara Madhusanka Jayalath commented on BEAM-9907:
-
Looking into this.
> apache_beam.transforms.external_test.ExternalTransformTest.test_nested flaky
>
>
> Key: BEAM-9907
> URL: https://issues.apache.org/jira/browse/BEAM-9907
> Project: Beam
> Issue Type: Test
> Components: sdk-py-core
>Reporter: Ning Kang
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
>
> Example test failures:
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/12682/
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/12684/
> A stacktrace
> {code:bash}
> apache_beam.transforms.external_test.ExternalTransformTest.test_nested (from
> py37-cloud)
> Failing for the past 1 build (Since Failed#12682 )
> Took 54 ms.
> Error Message
> google.protobuf.json_format.ParseError: Unexpected type for Value message.
> Stacktrace
> self = testMethod=test_nested>
> def test_nested(self):
> with beam.Pipeline() as p:
> > assert_that(p | FibTransform(6), equal_to([8]))
> apache_beam/transforms/external_test.py:250:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> apache_beam/transforms/ptransform.py:562: in __ror__
> result = p.apply(self, pvalueish, label)
> apache_beam/pipeline.py:651: in apply
> pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
> apache_beam/runners/runner.py:198: in apply
> return m(transform, input, options)
> apache_beam/runners/runner.py:228: in apply_PTransform
> return transform.expand(input)
> apache_beam/runners/portability/expansion_service_test.py:257: in expand
> expansion_service.ExpansionServiceServicer())
> apache_beam/pvalue.py:140: in __or__
> return self.pipeline.apply(ptransform, self)
> apache_beam/pipeline.py:598: in apply
> transform.transform, pvalueish, label or transform.label)
> apache_beam/pipeline.py:608: in apply
> return self.apply(transform, pvalueish)
> apache_beam/pipeline.py:651: in apply
> pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
> apache_beam/runners/runner.py:198: in apply
> return m(transform, input, options)
> apache_beam/runners/runner.py:228: in apply_PTransform
> return transform.expand(input)
> apache_beam/transforms/external.py:322: in expand
> pipeline_options=job_utils.pipeline_options_dict_to_struct(options))
> apache_beam/runners/job/utils.py:38: in pipeline_options_dict_to_struct
> v in options.items() if v is not None
> apache_beam/runners/job/utils.py:44: in dict_to_struct
> return json_format.ParseDict(dict_obj, struct_pb2.Struct())
> target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:450:
> in ParseDict
> parser.ConvertMessage(js_dict, message)
> target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:479:
> in ConvertMessage
> methodcaller(_WKTJSONMETHODS[full_name][1], value, message)(self)
> target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:667:
> in _ConvertStructMessage
> self._ConvertValueMessage(value[key], message.fields[key])
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _
> self =
> value = 0x7f35a4c00390>
> message =
> def _ConvertValueMessage(self, value, message):
> """Convert a JSON representation into Value message."""
> if isinstance(value, dict):
> self._ConvertStructMessage(value, message.struct_value)
> elif isinstance(value, list):
> self. _ConvertListValueMessage(value, message.list_value)
> elif value is None:
> message.null_value = 0
> elif isinstance(value, bool):
> message.bool_value = value
> elif isinstance(value, six.string_types):
> message.string_value = value
> elif isinstance(value, _INT_OR_FLOAT):
> message.number_value = value
> else:
> > raise ParseError('Unexpected type for Value message.')
> E google.protobuf.json_format.ParseError: Unexpected type for Value
> message.
> target/.tox-py37-cloud/py37-cloud/lib/python3.7/site-packages/google/protobuf/json_format.py:647:
> ParseError
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9923) Add an expansion service to Go SDK
Chamikara Madhusanka Jayalath created BEAM-9923: --- Summary: Add an expansion service to Go SDK Key: BEAM-9923 URL: https://issues.apache.org/jira/browse/BEAM-9923 Project: Beam Issue Type: Bug Components: sdk-go Reporter: Chamikara Madhusanka Jayalath This will allow Go SDK to offer it's transforms to other SDKs. This should support Beam [transform expansion API|[https://github.com/apache/beam/blob/master/model/job-management/src/main/proto/beam_expansion_api.proto]] and should have support for registering transforms to be offered through the transform expansion API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9922) Go SDK supports executing cross-language transforms on Spark runner
Chamikara Madhusanka Jayalath created BEAM-9922: --- Summary: Go SDK supports executing cross-language transforms on Spark runner Key: BEAM-9922 URL: https://issues.apache.org/jira/browse/BEAM-9922 Project: Beam Issue Type: Sub-task Components: sdk-go Reporter: Chamikara Madhusanka Jayalath -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9921) Go SDK supports executing cross-language transforms on Flink runner
Chamikara Madhusanka Jayalath created BEAM-9921: --- Summary: Go SDK supports executing cross-language transforms on Flink runner Key: BEAM-9921 URL: https://issues.apache.org/jira/browse/BEAM-9921 Project: Beam Issue Type: Sub-task Components: sdk-go Reporter: Chamikara Madhusanka Jayalath -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9920) Add support for identifying and staging artifacts received through transform expansion protocol
Chamikara Madhusanka Jayalath created BEAM-9920: --- Summary: Add support for identifying and staging artifacts received through transform expansion protocol Key: BEAM-9920 URL: https://issues.apache.org/jira/browse/BEAM-9920 Project: Beam Issue Type: Sub-task Components: sdk-go Reporter: Chamikara Madhusanka Jayalath A expanded remote transform may provide additional dependencies that has to be handled/staged by the calling SDK. [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L1310] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9919) Add an External transform to Go SDK
Chamikara Madhusanka Jayalath created BEAM-9919: --- Summary: Add an External transform to Go SDK Key: BEAM-9919 URL: https://issues.apache.org/jira/browse/BEAM-9919 Project: Beam Issue Type: Sub-task Components: sdk-go Reporter: Chamikara Madhusanka Jayalath This transform will be able to expand cross-language transforms by connecting to a remote expansion service through Beam expansion API. [https://github.com/apache/beam/blob/master/model/job-management/src/main/proto/beam_expansion_api.proto] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9918) Cross-language transforms support for Go SDK
Chamikara Madhusanka Jayalath created BEAM-9918: --- Summary: Cross-language transforms support for Go SDK Key: BEAM-9918 URL: https://issues.apache.org/jira/browse/BEAM-9918 Project: Beam Issue Type: New Feature Components: sdk-go Reporter: Chamikara Madhusanka Jayalath This is an uber issue for tasks related to cross-language transforms support for Go SDK. We can create sub-tasks as needed. cc: [~lostluck] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9913) Cross-language ValidatesRunner tests are failing due to failure of ':sdks:java:container:pullLicenses'
[ https://issues.apache.org/jira/browse/BEAM-9913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101288#comment-17101288 ] Chamikara Madhusanka Jayalath commented on BEAM-9913: - cc: [~robertwb] > Cross-language ValidatesRunner tests are failing due to failure of > ':sdks:java:container:pullLicenses' > -- > > Key: BEAM-9913 > URL: https://issues.apache.org/jira/browse/BEAM-9913 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Chamikara Madhusanka Jayalath >Assignee: Hannah Jiang >Priority: Major > > Both beam_PostCommit_XVR_Flink and beam_PostCommit_XVR_Spark are perma red. > For example, > [https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_XVR_Flink/2487/] > [https://scans.gradle.com/s/rydkawcamxtm4/console-log?task=:sdks:java:container:pullLicenses] > > Caused by: > org.gradle.process.internal.ExecException > Process 'command './sdks/java/container/license_scripts/license_script.sh'' > finished with non-zero exit value 2 > > at > org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNormalExitValue(DefaultExecHandle.java:396) > at > org.gradle.process.internal.DefaultExecAction.execute(DefaultExecAction.java:37) > > Probably due to [https://github.com/apache/beam/pull/11548] > > Hannah, can you please take a look ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9913) Cross-language ValidatesRunner tests are failing due to failure of ':sdks:java:container:pullLicenses'
Chamikara Madhusanka Jayalath created BEAM-9913: --- Summary: Cross-language ValidatesRunner tests are failing due to failure of ':sdks:java:container:pullLicenses' Key: BEAM-9913 URL: https://issues.apache.org/jira/browse/BEAM-9913 Project: Beam Issue Type: Bug Components: test-failures Reporter: Chamikara Madhusanka Jayalath Assignee: Hannah Jiang Both beam_PostCommit_XVR_Flink and beam_PostCommit_XVR_Spark are perma red. For example, [https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_XVR_Flink/2487/] [https://scans.gradle.com/s/rydkawcamxtm4/console-log?task=:sdks:java:container:pullLicenses] Caused by: org.gradle.process.internal.ExecException Process 'command './sdks/java/container/license_scripts/license_script.sh'' finished with non-zero exit value 2 at org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNormalExitValue(DefaultExecHandle.java:396) at org.gradle.process.internal.DefaultExecAction.execute(DefaultExecAction.java:37) Probably due to [https://github.com/apache/beam/pull/11548] Hannah, can you please take a look ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9482) beam_PerformanceTests_Kafka_IO failing due to " provided port is already allocated"
[ https://issues.apache.org/jira/browse/BEAM-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100309#comment-17100309 ] Chamikara Madhusanka Jayalath commented on BEAM-9482: - Haven't been able to look into this. ccing some folks who might be able to comment regarding access. cc: [~alanmyrvold] [~yifanzou] > beam_PerformanceTests_Kafka_IO failing due to " provided port is already > allocated" > --- > > Key: BEAM-9482 > URL: https://issues.apache.org/jira/browse/BEAM-9482 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Chamikara Madhusanka Jayalath >Priority: Major > > For example, > [https://builds.apache.org/view/A-D/view/Beam/view/PerformanceTests/job/beam_PerformanceTests_Kafka_IO/514/console] > > 18:55:33 Error from server (Invalid): error when creating > "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Kafka_IO/src/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-0.yml": > Service "outside-0" is invalid: spec.ports[0].nodePort: Invalid value: > 32400: provided port is already allocated > 18:55:33 Error from server (Invalid): error when creating > "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Kafka_IO/src/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-1.yml": > Service "outside-1" is invalid: spec.ports[0].nodePort: Invalid value: > 32401: provided port is already allocated > 18:55:33 Error from server (Invalid): error when creating > "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Kafka_IO/src/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-2.yml": > Service "outside-2" is invalid: spec.ports[0].nodePort: Invalid value: > 32402: provided port is already allocated > 1 > > Seems like we tried three ports but they were being used. Probably we should > update code to find an unused port dynamically. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9482) beam_PerformanceTests_Kafka_IO failing due to " provided port is already allocated"
[ https://issues.apache.org/jira/browse/BEAM-9482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chamikara Madhusanka Jayalath reassigned BEAM-9482: --- Assignee: (was: Chamikara Madhusanka Jayalath) > beam_PerformanceTests_Kafka_IO failing due to " provided port is already > allocated" > --- > > Key: BEAM-9482 > URL: https://issues.apache.org/jira/browse/BEAM-9482 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Chamikara Madhusanka Jayalath >Priority: Major > > For example, > [https://builds.apache.org/view/A-D/view/Beam/view/PerformanceTests/job/beam_PerformanceTests_Kafka_IO/514/console] > > 18:55:33 Error from server (Invalid): error when creating > "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Kafka_IO/src/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-0.yml": > Service "outside-0" is invalid: spec.ports[0].nodePort: Invalid value: > 32400: provided port is already allocated > 18:55:33 Error from server (Invalid): error when creating > "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Kafka_IO/src/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-1.yml": > Service "outside-1" is invalid: spec.ports[0].nodePort: Invalid value: > 32401: provided port is already allocated > 18:55:33 Error from server (Invalid): error when creating > "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Kafka_IO/src/.test-infra/kubernetes/kafka-cluster/04-outside-services/outside-2.yml": > Service "outside-2" is invalid: spec.ports[0].nodePort: Invalid value: > 32402: provided port is already allocated > 1 > > Seems like we tried three ports but they were being used. Probably we should > update code to find an unused port dynamically. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9779) HL7v2IOWriteIT is flaky
[ https://issues.apache.org/jira/browse/BEAM-9779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099986#comment-17099986 ] Chamikara Madhusanka Jayalath commented on BEAM-9779: - I think so. Test suite looks pretty stable now: [https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_Java/] We can resolve this. > HL7v2IOWriteIT is flaky > --- > > Key: BEAM-9779 > URL: https://issues.apache.org/jira/browse/BEAM-9779 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, test-failures >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Critical > Time Spent: 4h 10m > Remaining Estimate: 0h > > There seems to be a race condition somewhere in HL7v2IOWriteIT that causes > flakiness. > https://builds.apache.org/job/beam_PostCommit_Java/5947/ > https://builds.apache.org/job/beam_PostCommit_Java/5943/ > https://builds.apache.org/job/beam_PostCommit_Java/5942/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
