[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=290007=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-290007 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Aug/19 21:28 Start Date: 06/Aug/19 21:28 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 290007) Time Spent: 23.5h (was: 23h 20m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 23.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=290005=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-290005 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Aug/19 21:27 Start Date: 06/Aug/19 21:27 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-518850800 LGTM. Thanks. I'll go ahead and merge since a JIRA was created for the Python3 issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 290005) Time Spent: 23h 20m (was: 23h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 23h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289981 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Aug/19 20:42 Start Date: 06/Aug/19 20:42 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-518836109 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289981) Time Spent: 23h 10m (was: 23h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 23h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289980 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Aug/19 20:38 Start Date: 06/Aug/19 20:38 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-518834437 @tvalentyn This PR has nothing to do with `sdks/python/build.gradle`, thus using the same setting for both Python 2 and Python 3 is not easy. I've created a separate Jira issue: https://issues.apache.org/jira/browse/BEAM-7914 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289980) Time Spent: 23h (was: 22h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 23h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289970 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Aug/19 20:15 Start Date: 06/Aug/19 20:15 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-518826637 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289970) Time Spent: 22h 50m (was: 22h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 22h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289969 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Aug/19 20:15 Start Date: 06/Aug/19 20:15 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-518826637 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289969) Time Spent: 22h 40m (was: 22.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 22h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289923=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289923 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Aug/19 18:29 Start Date: 06/Aug/19 18:29 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-518789412 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289923) Time Spent: 22.5h (was: 22h 20m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 22.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289922=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289922 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Aug/19 18:29 Start Date: 06/Aug/19 18:29 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-518789412 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289922) Time Spent: 22h 20m (was: 22h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 22h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289919=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289919 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Aug/19 18:27 Start Date: 06/Aug/19 18:27 Worklog Time Spent: 10m Work Description: ihji commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r311210973 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -787,6 +788,40 @@ def as_deterministic_coder(self, step_label, error_message=None): return self +AVRO_CODER_URN = "beam:coder:avro:v1" + + +class AvroCoder(FastCoder): + """A coder used for AvroRecord values.""" + + def __init__(self, schema): +self.schema = schema + + def _create_impl(self): +return coder_impl.AvroCoderImpl(self.schema) + + def is_deterministic(self): +# TODO: need to confirm if it's deterministic Review comment: created https://issues.apache.org/jira/browse/BEAM-7903 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289919) Time Spent: 22h 10m (was: 22h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 22h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289779 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Aug/19 15:19 Start Date: 06/Aug/19 15:19 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-518716347 Please make sure Python scenarios are replicated on at least 1 Python 3 minor version. If you have to pick one, I suggest to pick Python 3.5. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289779) Time Spent: 22h (was: 21h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 22h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289373 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Aug/19 01:48 Start Date: 06/Aug/19 01:48 Worklog Time Spent: 10m Work Description: ihji commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r310853553 ## File path: sdks/python/apache_beam/io/external/generate_sequence.py ## @@ -50,11 +50,12 @@ class GenerateSequence(ptransform.PTransform): Experimental; no backwards compatibility guarantees. """ + URN = 'beam:external:java:generate_sequence:v1' Review comment: There are other places where ‘beam:external:java’ format is used. It would be better to address all of them in one commit later if we have to. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289373) Time Spent: 21h 50m (was: 21h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 21h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289250=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289250 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 05/Aug/19 23:07 Start Date: 05/Aug/19 23:07 Worklog Time Spent: 10m Work Description: ihji commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r310825758 ## File path: sdks/python/apache_beam/examples/wordcount_xlang.py ## @@ -70,7 +70,7 @@ def run(p, input_file, output_file): | 'split' >> (beam.ParDo(WordExtractingDoFn()) .with_output_types(bytes)) | 'count' >> beam.ExternalTransform( -'pytest:beam:transforms:count', None, EXPANSION_SERVICE_ADDR)) +'beam:transforms:xlang:count', None, EXPANSION_SERVICE_ADDR)) Review comment: I've created Jira issue: https://issues.apache.org/jira/browse/BEAM-7904 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289250) Time Spent: 21h 40m (was: 21.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 21h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289245=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289245
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 05/Aug/19 22:56
Start Date: 05/Aug/19 22:56
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #8174: [BEAM-6683] add
createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r310823429
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -297,6 +297,29 @@ class BeamModulePlugin implements Plugin {
}
}
+ // A class defining the configuration for CrossLanguageValidatesRunner.
+ class CrossLanguageValidatesRunnerConfiguration {
+// Task name for cross-language validate runner case.
+String name = 'validatesCrossLanguageRunner'
+// Fully qualified JobServerClass name to use.
+String jobServerDriver
+// A string representing the jobServer Configuration.
+String jobServerConfig
+// Number of parallel test runs.
+Integer numParallelTests = 1
+// Extra options to pass to TestPipeline
+String[] pipelineOpts = []
+// Categories for tests to run.
+Closure testCategories = {
+ includeCategories
'org.apache.beam.sdk.testing.UsesCrossLanguageTransforms'
+ // Use the following to include / exclude categories:
+ // includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
Review comment:
I think it might be useful for someone who wants to modify test categories.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 289245)
Time Spent: 21.5h (was: 21h 20m)
> Add an integration test suite for cross-language transforms for Flink runner
>
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 21.5h
> Remaining Estimate: 0h
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289237=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289237
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 05/Aug/19 22:35
Start Date: 05/Aug/19 22:35
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #8174: [BEAM-6683] add
createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r310818656
##
File path: sdks/python/apache_beam/io/external/generate_sequence_test.py
##
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for cross-language generate sequence."""
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import logging
+import os
+import re
+import unittest
+
+from nose.plugins.attrib import attr
+
+from apache_beam.io.external.generate_sequence import GenerateSequence
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@attr('UsesCrossLanguageTransforms')
+@unittest.skipUnless(
+os.environ.get('EXPANSION_PORT'),
+"EXPANSION_PORT environment var is not provided.")
+class XlangGenerateSequenceTest(unittest.TestCase):
+ def test_generate_sequence(self):
+test_pipeline = TestPipeline()
+port = os.environ.get('EXPANSION_PORT')
+address = 'localhost:%s' % port
+
+try:
+ with test_pipeline as p:
+res = (
+p
+| GenerateSequence(start=1, stop=10,
+ expansion_service=address)
+)
+
+assert_that(res, equal_to([i for i in range(1, 10)]))
+except RuntimeError as e:
+ if re.search(GenerateSequence.URN, str(e)):
+print("looks like URN not implemented in expansion service, skipping.")
Review comment:
`generate_sequence_test` is not only used with Java cross-language
transforms but also with Python (and possibly Go) cross-language transforms. If
we don't handle those errors, the test will fail in Python-using-Python,
Python-using-Go test combinations.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 289237)
Time Spent: 21h 20m (was: 21h 10m)
> Add an integration test suite for cross-language transforms for Flink runner
>
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 21h 20m
> Remaining Estimate: 0h
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289235=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289235 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 05/Aug/19 22:26 Start Date: 05/Aug/19 22:26 Worklog Time Spent: 10m Work Description: ihji commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r310816457 ## File path: sdks/python/apache_beam/transforms/combiners.py ## @@ -133,10 +133,12 @@ class PerElement(ptransform.PTransform): def expand(self, pcoll): paired_with_void_type = typehints.Tuple[pcoll.element_type, Any] + output_type = typehints.KV[pcoll.element_type, int] Review comment: Yes, it is. Cross-language `PerElement` doesn't work without explicit output type. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289235) Time Spent: 21h 10m (was: 21h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 21h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289118=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289118 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 05/Aug/19 18:31 Start Date: 05/Aug/19 18:31 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r310731131 ## File path: sdks/python/apache_beam/examples/wordcount_xlang.py ## @@ -70,7 +70,7 @@ def run(p, input_file, output_file): | 'split' >> (beam.ParDo(WordExtractingDoFn()) .with_output_types(bytes)) | 'count' >> beam.ExternalTransform( -'pytest:beam:transforms:count', None, EXPANSION_SERVICE_ADDR)) +'beam:transforms:xlang:count', None, EXPANSION_SERVICE_ADDR)) Review comment: Please create a JIRA to create a standard for cross-language transform URNs. I think having a recommended standard will ensure (hopefully) that users do not end up creating cross-language transforms with conflicting URNs. cc: @robertwb @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289118) Time Spent: 20h 50m (was: 20h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 20h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289113 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 05/Aug/19 18:31 Start Date: 05/Aug/19 18:31 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r310731455 ## File path: sdks/python/apache_beam/io/external/generate_sequence.py ## @@ -50,11 +50,12 @@ class GenerateSequence(ptransform.PTransform): Experimental; no backwards compatibility guarantees. """ + URN = 'beam:external:java:generate_sequence:v1' Review comment: Should be of the format "beam:transforms:xlang..." ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289113) Time Spent: 20h 10m (was: 20h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 20h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289114=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289114 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 05/Aug/19 18:31 Start Date: 05/Aug/19 18:31 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r310734774 ## File path: sdks/python/apache_beam/transforms/combiners.py ## @@ -133,10 +133,12 @@ class PerElement(ptransform.PTransform): def expand(self, pcoll): paired_with_void_type = typehints.Tuple[pcoll.element_type, Any] + output_type = typehints.KV[pcoll.element_type, int] Review comment: Is the change here still needed (just to confirm) ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289114) Time Spent: 20h 20m (was: 20h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 20h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289112=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289112 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 05/Aug/19 18:31 Start Date: 05/Aug/19 18:31 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r310730528 ## File path: sdks/python/apache_beam/coders/coders.py ## @@ -787,6 +788,40 @@ def as_deterministic_coder(self, step_label, error_message=None): return self +AVRO_CODER_URN = "beam:coder:avro:v1" + + +class AvroCoder(FastCoder): + """A coder used for AvroRecord values.""" + + def __init__(self, schema): +self.schema = schema + + def _create_impl(self): +return coder_impl.AvroCoderImpl(self.schema) + + def is_deterministic(self): +# TODO: need to confirm if it's deterministic Review comment: Please resolve this or add a JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289112) Time Spent: 20h (was: 19h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 20h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289116=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289116
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 05/Aug/19 18:31
Start Date: 05/Aug/19 18:31
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #8174:
[BEAM-6683] add createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r309410867
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -297,6 +297,29 @@ class BeamModulePlugin implements Plugin {
}
}
+ // A class defining the configuration for CrossLanguageValidatesRunner.
+ class CrossLanguageValidatesRunnerConfiguration {
+// Task name for cross-language validate runner case.
+String name = 'validatesCrossLanguageRunner'
+// Fully qualified JobServerClass name to use.
+String jobServerDriver
+// A string representing the jobServer Configuration.
+String jobServerConfig
+// Number of parallel test runs.
+Integer numParallelTests = 1
+// Extra options to pass to TestPipeline
+String[] pipelineOpts = []
+// Categories for tests to run.
+Closure testCategories = {
+ includeCategories
'org.apache.beam.sdk.testing.UsesCrossLanguageTransforms'
+ // Use the following to include / exclude categories:
+ // includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
Review comment:
Did you intend to leave these commented out lines ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 289116)
Time Spent: 20h 40m (was: 20.5h)
> Add an integration test suite for cross-language transforms for Flink runner
>
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 20h 40m
> Remaining Estimate: 0h
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289115=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289115
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 05/Aug/19 18:31
Start Date: 05/Aug/19 18:31
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #8174:
[BEAM-6683] add createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r310732987
##
File path: sdks/python/apache_beam/io/external/generate_sequence_test.py
##
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for cross-language generate sequence."""
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import logging
+import os
+import re
+import unittest
+
+from nose.plugins.attrib import attr
+
+from apache_beam.io.external.generate_sequence import GenerateSequence
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@attr('UsesCrossLanguageTransforms')
+@unittest.skipUnless(
+os.environ.get('EXPANSION_PORT'),
+"EXPANSION_PORT environment var is not provided.")
+class XlangGenerateSequenceTest(unittest.TestCase):
+ def test_generate_sequence(self):
+test_pipeline = TestPipeline()
+port = os.environ.get('EXPANSION_PORT')
+address = 'localhost:%s' % port
+
+try:
+ with test_pipeline as p:
+res = (
+p
+| GenerateSequence(start=1, stop=10,
+ expansion_service=address)
+)
+
+assert_that(res, equal_to([i for i in range(1, 10)]))
+except RuntimeError as e:
+ if re.search(GenerateSequence.URN, str(e)):
+print("looks like URN not implemented in expansion service, skipping.")
Review comment:
When would this happen ? Looks like GenerateSequence.java is a part of
java/core so this should always be available.
Please avoid skipping the test here if possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 289115)
Time Spent: 20.5h (was: 20h 20m)
> Add an integration test suite for cross-language transforms for Flink runner
>
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 20.5h
> Remaining Estimate: 0h
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289119=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289119 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 05/Aug/19 18:31 Start Date: 05/Aug/19 18:31 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r310729923 ## File path: sdks/python/apache_beam/coders/avro_record.py ## @@ -0,0 +1,38 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +"""AvroRecord for AvroCoder.""" Review comment: Please add experiment annotations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289119) Time Spent: 21h (was: 20h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 21h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=289117=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289117 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 05/Aug/19 18:31 Start Date: 05/Aug/19 18:31 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r310729685 ## File path: sdks/python/apache_beam/coders/avro_coder.py ## @@ -1,99 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -#http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -"""Coder for AvroRecord serialization/deserialization.""" - -from __future__ import absolute_import - -import json -from io import BytesIO - -from fastavro import parse_schema -from fastavro import schemaless_reader -from fastavro import schemaless_writer - -from apache_beam.coders.coder_impl import SimpleCoderImpl -from apache_beam.coders.coders import Coder -from apache_beam.coders.coders import FastCoder - -AVRO_CODER_URN = "beam:coder:avro:v1" - -__all__ = ['AvroCoder', 'AvroRecord'] Review comment: FYI: I think it's OK to remove this since it was just added two months ago but for future, please make sure to add the experiment annotations till APIs are stabilized. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289117) Time Spent: 20h 50m (was: 20h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 20h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=282830=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282830 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 25/Jul/19 18:13 Start Date: 25/Jul/19 18:13 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-515155756 @chamikaramj PTAL. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282830) Time Spent: 19h 50m (was: 19h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 19h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=282815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282815 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 25/Jul/19 17:49 Start Date: 25/Jul/19 17:49 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-515145560 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282815) Time Spent: 19h 40m (was: 19.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 19h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=282814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282814 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 25/Jul/19 17:49 Start Date: 25/Jul/19 17:49 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-515140992 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282814) Time Spent: 19.5h (was: 19h 20m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 19.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=282801=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282801 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 25/Jul/19 17:38 Start Date: 25/Jul/19 17:38 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-515140992 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282801) Time Spent: 18h 50m (was: 18h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 18h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=282803=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282803 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 25/Jul/19 17:38 Start Date: 25/Jul/19 17:38 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-515141193 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282803) Time Spent: 19h 10m (was: 19h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 19h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=282802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282802 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 25/Jul/19 17:38 Start Date: 25/Jul/19 17:38 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-514853752 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282802) Time Spent: 19h (was: 18h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 19h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=282804=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282804 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 25/Jul/19 17:38 Start Date: 25/Jul/19 17:38 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-515141193 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282804) Time Spent: 19h 20m (was: 19h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 19h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=282368=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282368 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 25/Jul/19 01:02 Start Date: 25/Jul/19 01:02 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-514853752 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282368) Time Spent: 18h 40m (was: 18.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 18h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=282367=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282367 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 25/Jul/19 01:02 Start Date: 25/Jul/19 01:02 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-514836928 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282367) Time Spent: 18.5h (was: 18h 20m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 18.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=282366=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282366 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 25/Jul/19 01:02 Start Date: 25/Jul/19 01:02 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-514839151 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282366) Time Spent: 18h 20m (was: 18h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 18h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=282357=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282357 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 24/Jul/19 23:42 Start Date: 24/Jul/19 23:42 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-514839151 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282357) Time Spent: 18h 10m (was: 18h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 18h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=282352=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-282352 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 24/Jul/19 23:32 Start Date: 24/Jul/19 23:32 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-514836928 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 282352) Time Spent: 18h (was: 17h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 18h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=264246=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264246 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 20/Jun/19 23:58 Start Date: 20/Jun/19 23:58 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-504240101 Java ParquetIO write transform is still broken so it's temporarily removed from the test. Will fix this issue in a separate PR (https://issues.apache.org/jira/browse/BEAM-7612) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 264246) Time Spent: 17h 50m (was: 17h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 17h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=264222=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264222 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 20/Jun/19 23:17 Start Date: 20/Jun/19 23:17 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-504232195 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 264222) Time Spent: 17h 20m (was: 17h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 17h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=264224=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264224 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 20/Jun/19 23:17 Start Date: 20/Jun/19 23:17 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-504232195 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 264224) Time Spent: 17h 40m (was: 17.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 17h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=264223=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264223 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 20/Jun/19 23:17 Start Date: 20/Jun/19 23:17 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-504226525 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 264223) Time Spent: 17.5h (was: 17h 20m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 17.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=264206=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264206 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 20/Jun/19 22:50 Start Date: 20/Jun/19 22:50 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-504226713 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 264206) Time Spent: 17h (was: 16h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 17h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=264207=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264207 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 20/Jun/19 22:50 Start Date: 20/Jun/19 22:50 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-504226713 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 264207) Time Spent: 17h 10m (was: 17h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 17h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=264204=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264204 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 20/Jun/19 22:49 Start Date: 20/Jun/19 22:49 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-503808085 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 264204) Time Spent: 16h 40m (was: 16.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 16h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=264205=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264205 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 20/Jun/19 22:49 Start Date: 20/Jun/19 22:49 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-504226525 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 264205) Time Spent: 16h 50m (was: 16h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 16h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=264202=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264202 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 20/Jun/19 22:48 Start Date: 20/Jun/19 22:48 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-504226435 run python precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 264202) Time Spent: 16h 20m (was: 16h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 16h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=264203=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264203 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 20/Jun/19 22:48 Start Date: 20/Jun/19 22:48 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-504226435 run python precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 264203) Time Spent: 16.5h (was: 16h 20m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 16.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=264121=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-264121 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 20/Jun/19 21:11 Start Date: 20/Jun/19 21:11 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-504184379 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 264121) Time Spent: 16h 10m (was: 16h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 16h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=263461=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-263461 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 20/Jun/19 01:47 Start Date: 20/Jun/19 01:47 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-503808085 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 263461) Time Spent: 15h 50m (was: 15h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 15h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=263457=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-263457 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 20/Jun/19 01:28 Start Date: 20/Jun/19 01:28 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-503785657 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 263457) Time Spent: 15h 40m (was: 15.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 15h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=263413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-263413 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 19/Jun/19 23:42 Start Date: 19/Jun/19 23:42 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-503785657 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 263413) Time Spent: 15.5h (was: 15h 20m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 15.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=263393=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-263393 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 19/Jun/19 23:11 Start Date: 19/Jun/19 23:11 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-503779549 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 263393) Time Spent: 15h 20m (was: 15h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 15h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=263391=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-263391 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 19/Jun/19 23:11 Start Date: 19/Jun/19 23:11 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-503779176 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 263391) Time Spent: 15h (was: 14h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 15h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=263392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-263392 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 19/Jun/19 23:11 Start Date: 19/Jun/19 23:11 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-503779549 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 263392) Time Spent: 15h 10m (was: 15h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 15h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=263389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-263389 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 19/Jun/19 23:09 Start Date: 19/Jun/19 23:09 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-503779176 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 263389) Time Spent: 14h 50m (was: 14h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 14h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=260870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260870 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 15/Jun/19 11:39 Start Date: 15/Jun/19 11:39 Worklog Time Spent: 10m Work Description: mxm commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-502359225 Any update here? Would be great to get this in. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 260870) Time Spent: 14h 40m (was: 14.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 14h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=255032=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-255032 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Jun/19 10:47 Start Date: 06/Jun/19 10:47 Worklog Time Spent: 10m Work Description: mxm commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-499445538 I'm just collecting all the issues here. If they do not appear on Jenkins, we can address them separately. After fixing the Docker executable path manually, I've got the tests running. Problematic test is `xlang_parquetio_test.py#test_write_and_read`. I wonder, is it not problematic to read and write to a parquet file from the same job? Especially because Flink does not usually do staged execution but "pipelines" the data through the entire DAG. I think the change in #8693 might have made that problem visible. Could we first execute the write pipeline and subsequently another read pipeline? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 255032) Time Spent: 14.5h (was: 14h 20m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 14.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=255027=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-255027 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Jun/19 10:14 Start Date: 06/Jun/19 10:14 Worklog Time Spent: 10m Work Description: mxm commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-499435604 Now I'm getting this when running the tests: ``` == ERROR: test_java_expansion_portable_runner (apache_beam.transforms.external_test.ExternalTransformTest) -- Traceback (most recent call last): File "/Users/max/Dev/beam/sdks/python/apache_beam/transforms/external_test.py", line 153, in test_java_expansion_portable_runner ExternalTransformTest.run_pipeline_with_portable_runner(None) File "/Users/max/Dev/beam/sdks/python/apache_beam/transforms/external_test.py", line 186, in run_pipeline_with_portable_runner pipeline_options, ExternalTransformTest.expansion_service_port, True) File "/Users/max/Dev/beam/sdks/python/apache_beam/transforms/external_test.py", line 219, in run_pipeline result = p.run() File "/Users/max/Dev/beam/sdks/python/apache_beam/testing/test_pipeline.py", line 107, in run else test_runner_api)) File "/Users/max/Dev/beam/sdks/python/apache_beam/pipeline.py", line 419, in run return self.runner.run_pipeline(self, self._options) File "/Users/max/Dev/beam/sdks/python/apache_beam/runners/portability/portable_runner.py", line 161, in run_pipeline self.init_dockerized_job_server() File "/Users/max/Dev/beam/sdks/python/apache_beam/runners/portability/portable_runner.py", line 148, in init_dockerized_job_server self._job_endpoint = docker.start() File "/Users/max/Dev/beam/sdks/python/apache_beam/runners/portability/job_server.py", line 56, in start docker_path = check_output(['which', 'docker']).strip() File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 216, in check_output process = Popen(stdout=PIPE, *popenargs, **kwargs) File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 394, in __init__ errread, errwrite) File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1047, in _execute_child raise child_exception OSError: [Errno 9] Bad file descriptor ``` Strange because `which docker` returns the correct Docker executable on my machine. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 255027) Time Spent: 14h 20m (was: 14h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 14h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=255026=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-255026 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 06/Jun/19 10:13 Start Date: 06/Jun/19 10:13 Worklog Time Spent: 10m Work Description: mxm commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-499435362 Python 3 test fail for me: ``` == ERROR: Failure: SyntaxError (invalid syntax (transforms_keyword_only_args_py3_only_test.py, line 39)) -- Traceback (most recent call last): File "/Users/max/Dev/beam/build/gradleenv/192237/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName addr.filename, addr.module) File "/Users/max/Dev/beam/build/gradleenv/192237/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath return self.importFromDir(dir_path, fqname) File "/Users/max/Dev/beam/build/gradleenv/192237/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/Users/max/Dev/beam/sdks/python/apache_beam/transforms/transforms_keyword_only_args_py3_only_test.py", line 39 def sort_with_side_inputs(x, *s, reverse=False): ^ SyntaxError: invalid syntax ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 255026) Time Spent: 14h 10m (was: 14h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 14h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=255022=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-255022
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 06/Jun/19 09:57
Start Date: 06/Jun/19 09:57
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8174: [BEAM-6683] add
createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r291102612
##
File path: runners/flink/job-server/flink_job_server.gradle
##
@@ -165,23 +165,13 @@ task validatesPortableRunner() {
dependsOn validatesPortableRunnerStreaming
}
-project.ext.validatesCrossLanguageTransforms =
- createPortableValidatesRunnerTask(
-name: "validatesCrossLanguageTransforms",
-jobServerDriver: "org.apache.beam.runners.flink.FlinkJobServerDriver",
-jobServerConfig:
"--clean-artifacts-per-job,--job-host=localhost,--job-port=0,--artifact-port=0,--expansion-port=0",
-testClasspathConfiguration: configurations.validatesPortableRunner,
-numParallelTests: 1,
-pipelineOpts: [
- // Limit resource consumption via parallelism
- "--parallelism=2",
- "--shutdownSourcesOnFinalWatermark",
-],
-testCategories: {
- // Only include cross-language transform tests. Avoid to retest
everything on Docker environment.
- includeCategories
'org.apache.beam.sdk.testing.UsesCrossLanguageTransforms'
-},
- )
-project.evaluationDependsOn ':sdks:python'
-validatesCrossLanguageTransforms.dependsOn ':sdks:python:setupVirtualenv'
-validatesCrossLanguageTransforms.systemProperty "pythonTestExpansionCommand",
". ${project(':sdks:python').envdir}/bin/activate && pip install -e
${project(':sdks:python').projectDir}[test] && python -m
apache_beam.runners.portability.expansion_service_test"
+project.ext.validatesCrossLanguageRunner =
createCrossLanguageValidatesRunnerTask(
+ jobServerDriver: "org.apache.beam.runners.flink.FlinkJobServerDriver",
Review comment:
At least for Python we have to depend on
`:runners:flink:1.5:job-server-container:docker` or bring up the
FlinkJobServerDriver like we do in `flink_runner_test.py`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 255022)
Time Spent: 14h (was: 13h 50m)
> Add an integration test suite for cross-language transforms for Flink runner
>
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 14h
> Remaining Estimate: 0h
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=254557=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-254557 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 05/Jun/19 18:23 Start Date: 05/Jun/19 18:23 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-499199513 @mxm That's strange. `expansion-port` is a valid option: https://github.com/apache/beam/blob/e6104a2fcb2bd8e25e06d06428cb7e6223d30083/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobServerDriver.java#L82 BTW, I couldn't find the line `FlinkJobServerDriver.java:131` as your error message shows. Looks like https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java only has 113 lines. Maybe, we're seeing a different version of the file? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 254557) Time Spent: 13h 50m (was: 13h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 13h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=254468=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-254468 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 05/Jun/19 16:24 Start Date: 05/Jun/19 16:24 Worklog Time Spent: 10m Work Description: mxm commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-499156048 When I get rid of the errors above, I get this: ``` [main] ERROR org.apache.beam.runners.flink.FlinkJobServerDriver - Unable to parse command line arguments. org.kohsuke.args4j.CmdLineException: "--expansion-port" is not a valid option at org.kohsuke.args4j.CmdLineParser.parseArgument(CmdLineParser.java:497) at org.apache.beam.runners.flink.FlinkJobServerDriver.fromParams(FlinkJobServerDriver.java:127) at org.apache.beam.runners.flink.FlinkJobServerDriver.main(FlinkJobServerDriver.java:113) Usage: java FlinkJobServerDriver arguments... --artifact-port N : The artifact service port. 0 to use a dynamic port. (Default: 8098) (default: 49219) --artifacts-dir VAL: The location to store staged artifact files (default: /tmp/beam-artifact-staging) --clean-artifacts-per-job : When true, remove each job's staged artifacts when it completes (default: false) --flink-conf-dir VAL : Directory containing Flink YAML configuration files. These properties will be set to all jobs submitted to Flink and take precedence over configurations in FLINK_CONF_DIR. --flink-master-url VAL : Flink master url to submit job. (default: [auto]) --job-host VAL : The job server host name (default: localhost) --job-port N : The job service port. 0 to use a dynamic port. (Default: 8099) (default: 49218) --sdk-worker-parallelism N : Default parallelism for SDK worker processes (see portable pipeline options) (default: 1) Exception in thread "main" java.lang.IllegalArgumentException: Unable to parse command line arguments. at org.apache.beam.runners.flink.FlinkJobServerDriver.fromParams(FlinkJobServerDriver.java:131) at org.apache.beam.runners.flink.FlinkJobServerDriver.main(FlinkJobServerDriver.java:113) Caused by: org.kohsuke.args4j.CmdLineException: "--expansion-port" is not a valid option at org.kohsuke.args4j.CmdLineParser.parseArgument(CmdLineParser.java:497) at org.apache.beam.runners.flink.FlinkJobServerDriver.fromParams(FlinkJobServerDriver.java:127) ... 1 more ``` Quite strange. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 254468) Time Spent: 13h 40m (was: 13.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 13h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=254313=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-254313
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 05/Jun/19 11:39
Start Date: 05/Jun/19 11:39
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8174: [BEAM-6683] add
createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r290676217
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1633,6 +1653,109 @@ class BeamModulePlugin implements Plugin {
/**
***/
+// Method to create the crossLanguageValidatesRunnerTask.
+// The method takes crossLanguageValidatesRunnerConfiguration as parameter.
+project.ext.createCrossLanguageValidatesRunnerTask = {
+ def config = it ? it as CrossLanguageValidatesRunnerConfiguration : new
CrossLanguageValidatesRunnerConfiguration()
+
+ project.evaluationDependsOn(":sdks:python")
+ project.evaluationDependsOn(":sdks:java:testing:expansion-service")
+ project.evaluationDependsOn(":runners:core-construction-java")
+
+ // Task for launching expansion services
+ def envDir = project.project(":sdks:python").envdir
+ def pythonDir = project.project(":sdks:python").projectDir
+ def javaPort = startingExpansionPortNumber.getAndDecrement()
+ def pythonPort = startingExpansionPortNumber.getAndDecrement()
+ def expansionJar =
project.project(':sdks:java:testing:expansion-service').buildTestExpansionServiceJar.archivePath
+ def expansionServiceOpts = [
+"group_id": project.name,
+"java_expansion_service_jar": expansionJar,
+"java_port": javaPort,
+"python_virtualenv_dir": envDir,
+"python_expansion_service_module":
"apache_beam.runners.portability.expansion_service_test",
+"python_port": pythonPort
+ ]
+ def serviceArgs =
project.project(':sdks:python').mapToArgString(expansionServiceOpts)
+ def setupTask = project.tasks.create(name: config.name+"Setup", type:
Exec) {
+dependsOn ':sdks:java:container:docker'
+dependsOn ':sdks:python:container:docker'
+dependsOn
':sdks:java:testing:expansion-service:buildTestExpansionServiceJar'
+dependsOn ":sdks:python:installGcpTest"
+// setup test env
+executable 'sh'
Review comment:
Even if I change this to `bash` I get the following on my machine:
```
> Task :runners:flink:1.5:job-server:validatesCrossLanguageRunnerSetup FAILED
/Users/max/Dev/beam/sdks/python/scripts/run_expansion_services.sh: line 80:
[: -v: unary operator expected
/Users/max/Dev/beam/sdks/python/scripts/run_expansion_services.sh: line 136:
flock: command not found
> Task :runners:flink:1.5:job-server:validatesCrossLanguageRunnerCleanup
FAILED
/Users/max/Dev/beam/sdks/python/scripts/run_expansion_services.sh: line 80:
[: -v: unary operator expected
/Users/max/Dev/beam/sdks/python/scripts/run_expansion_services.sh: line 136:
flock: command not found
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 254313)
Time Spent: 13.5h (was: 13h 20m)
> Add an integration test suite for cross-language transforms for Flink runner
>
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 13.5h
> Remaining Estimate: 0h
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=254314=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-254314
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 05/Jun/19 11:39
Start Date: 05/Jun/19 11:39
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #8174: [BEAM-6683] add
createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r290676240
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1633,6 +1653,109 @@ class BeamModulePlugin implements Plugin {
/**
***/
+// Method to create the crossLanguageValidatesRunnerTask.
+// The method takes crossLanguageValidatesRunnerConfiguration as parameter.
+project.ext.createCrossLanguageValidatesRunnerTask = {
+ def config = it ? it as CrossLanguageValidatesRunnerConfiguration : new
CrossLanguageValidatesRunnerConfiguration()
+
+ project.evaluationDependsOn(":sdks:python")
+ project.evaluationDependsOn(":sdks:java:testing:expansion-service")
+ project.evaluationDependsOn(":runners:core-construction-java")
+
+ // Task for launching expansion services
+ def envDir = project.project(":sdks:python").envdir
+ def pythonDir = project.project(":sdks:python").projectDir
+ def javaPort = startingExpansionPortNumber.getAndDecrement()
+ def pythonPort = startingExpansionPortNumber.getAndDecrement()
+ def expansionJar =
project.project(':sdks:java:testing:expansion-service').buildTestExpansionServiceJar.archivePath
+ def expansionServiceOpts = [
+"group_id": project.name,
+"java_expansion_service_jar": expansionJar,
+"java_port": javaPort,
+"python_virtualenv_dir": envDir,
+"python_expansion_service_module":
"apache_beam.runners.portability.expansion_service_test",
+"python_port": pythonPort
+ ]
+ def serviceArgs =
project.project(':sdks:python').mapToArgString(expansionServiceOpts)
+ def setupTask = project.tasks.create(name: config.name+"Setup", type:
Exec) {
+dependsOn ':sdks:java:container:docker'
+dependsOn ':sdks:python:container:docker'
+dependsOn
':sdks:java:testing:expansion-service:buildTestExpansionServiceJar'
+dependsOn ":sdks:python:installGcpTest"
+// setup test env
+executable 'sh'
+args '-c', "$pythonDir/scripts/run_expansion_services.sh stop
--group_id ${project.name} && $pythonDir/scripts/run_expansion_services.sh
start $serviceArgs"
+ }
+
+ def mainTask = project.tasks.create(name: config.name) {
+group = "Verification"
+description = "Validates cross-language capability of runner"
+ }
+
+ def cleanupTask = project.tasks.create(name: config.name+'Cleanup',
type: Exec) {
+// teardown test env
+executable 'sh'
Review comment:
Same as above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 254314)
Time Spent: 13.5h (was: 13h 20m)
> Add an integration test suite for cross-language transforms for Flink runner
>
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 13.5h
> Remaining Estimate: 0h
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=254033=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-254033 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 04/Jun/19 21:17 Start Date: 04/Jun/19 21:17 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-498847354 @mxm Sure. It would be great if you can look into it. I also wonder why #8693 breaks it since the change #8693 introduced looks pretty trivial. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 254033) Time Spent: 13h 20m (was: 13h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 13h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=253978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253978 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 04/Jun/19 19:27 Start Date: 04/Jun/19 19:27 Worklog Time Spent: 10m Work Description: mxm commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-498809192 @ihji I can take a look if you want. Curious why this breaks because #8693 didn't break any other integration tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 253978) Time Spent: 13h 10m (was: 13h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 13h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=253899=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253899 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 04/Jun/19 18:11 Start Date: 04/Jun/19 18:11 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-498782464 Found that XVR_Flink PostCommit is broken because of https://github.com/apache/beam/pull/8693 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 253899) Time Spent: 13h (was: 12h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 13h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=253497=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253497 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 04/Jun/19 02:20 Start Date: 04/Jun/19 02:20 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-498493430 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 253497) Time Spent: 12h 50m (was: 12h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 12h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=253496=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253496 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 04/Jun/19 02:20 Start Date: 04/Jun/19 02:20 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-498493416 LGTM. Thanks. Will merge after tests pass. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 253496) Time Spent: 12h 40m (was: 12.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 12h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=253489=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253489 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 04/Jun/19 01:56 Start Date: 04/Jun/19 01:56 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-498489176 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 253489) Time Spent: 12.5h (was: 12h 20m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 12.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=253488=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253488 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 04/Jun/19 01:55 Start Date: 04/Jun/19 01:55 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-498488953 Run Flink CrossLanguageValidatesRunner Tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 253488) Time Spent: 12h 20m (was: 12h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 12h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=253483=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-253483 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 04/Jun/19 01:18 Start Date: 04/Jun/19 01:18 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-498482529 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 253483) Time Spent: 12h 10m (was: 12h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 12h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=250454=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-250454 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 29/May/19 20:25 Start Date: 29/May/19 20:25 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-497095893 @chamikaramj Rebased on latest master. PTAL. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 250454) Time Spent: 12h (was: 11h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 12h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=250390=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-250390 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 29/May/19 19:05 Start Date: 29/May/19 19:05 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-497069483 run python precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 250390) Time Spent: 11h 50m (was: 11h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 11h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=250389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-250389 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 29/May/19 19:05 Start Date: 29/May/19 19:05 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-497069437 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 250389) Time Spent: 11h 40m (was: 11.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 11h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=250387=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-250387 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 29/May/19 19:05 Start Date: 29/May/19 19:05 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-497069437 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 250387) Time Spent: 11h 20m (was: 11h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 11h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=250388=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-250388 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 29/May/19 19:05 Start Date: 29/May/19 19:05 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-497069483 run python precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 250388) Time Spent: 11.5h (was: 11h 20m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 11.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=249737=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-249737
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 29/May/19 00:21
Start Date: 29/May/19 00:21
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #8174: [BEAM-6683] add
createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r288351808
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1578,6 +1603,110 @@ class BeamModulePlugin implements Plugin {
/**
***/
+// Method to create the crossLanguageValidatesRunnerTask.
+// The method takes crossLanguageValidatesRunnerConfiguration as parameter.
+project.ext.createCrossLanguageValidatesRunnerTask = {
+ def config = it ? it as CrossLanguageValidatesRunnerConfiguration : new
CrossLanguageValidatesRunnerConfiguration()
+
+ project.evaluationDependsOn(":beam-sdks-python")
+ project.evaluationDependsOn(":beam-sdks-java-test-expansion-service")
+ project.evaluationDependsOn(":beam-sdks-java-io-parquet")
+ project.evaluationDependsOn(":beam-runners-core-construction-java")
+
+ // Task for launching expansion services
+ def envDir = project.project(":beam-sdks-python").envdir
+ def pythonDir = project.project(":beam-sdks-python").projectDir
+ def javaPort = startingExpansionPortNumber.getAndDecrement()
+ def pythonPort = startingExpansionPortNumber.getAndDecrement()
+ def expansionServiceOpts = [
+"group_id": project.name,
+"java_expansion_service_jar":
"${project.project(':beam-sdks-java-test-expansion-service').buildTestExpansionServiceJar.archivePath}",
+"java_port": javaPort,
+"python_virtualenv_dir": envDir,
+"python_expansion_service_module":
"apache_beam.runners.portability.expansion_service_test",
+"python_port": pythonPort
+ ]
+ def serviceArgs =
project.project(':beam-sdks-python').mapToArgString(expansionServiceOpts)
+ def setupTask = project.tasks.create(name: config.name+"Setup", type:
Exec) {
+dependsOn ':beam-sdks-java-container:docker'
+dependsOn ':beam-sdks-python-container:docker'
+dependsOn
':beam-sdks-java-test-expansion-service:buildTestExpansionServiceJar'
+dependsOn ":beam-sdks-java-io-parquet:shadowJarWithDependencies"
+dependsOn ":beam-sdks-python:installGcpTest"
+// setup test env
+executable 'sh'
+args '-c', "$pythonDir/scripts/run_expansion_services.sh start
$serviceArgs"
+ }
+
+ def mainTask = project.tasks.create(name: config.name) {
+group = "Verification"
+description = "Validates cross-language capability of runner"
+ }
+
+ def cleanupTask = project.tasks.create(name: config.name+'Cleanup',
type: Exec) {
+// teardown test env
+executable 'sh'
+args '-c', "$pythonDir/scripts/run_expansion_services.sh stop
--group_id ${project.name}"
+ }
+ setupTask.finalizedBy cleanupTask
+
+ // Task for running testcases in Java SDK
+ def beamJavaTestPipelineOptions = [
+
"--runner=org.apache.beam.runners.reference.testing.TestPortableRunner",
+"--jobServerDriver=${config.jobServerDriver}",
+"--environmentCacheMillis=1"
+ ]
+ beamJavaTestPipelineOptions.addAll(config.pipelineOpts)
+ if (config.jobServerConfig) {
+
beamJavaTestPipelineOptions.add("--jobServerConfig=${config.jobServerConfig}")
+ }
+ ['Java': javaPort, 'Python': pythonPort].each { sdk, port ->
+def javaTask = project.tasks.create(name: config.name+"JavaUsing"+sdk,
type: Test) {
+ group = "Verification"
+ description = "Validates runner for cross-language capability of
using ${sdk} transforms from Java SDK"
+ systemProperty "beamTestPipelineOptions",
JsonOutput.toJson(beamJavaTestPipelineOptions)
+ classpath = config.testClasspathConfiguration
+ testClassesDirs =
project.files(project.project(":beam-runners-core-construction-java").sourceSets.test.output.classesDirs)
+ maxParallelForks config.numParallelTests
+ useJUnit(config.testCategories)
+ // increase maxHeapSize as this is directly correlated to direct
memory,
+ // see https://issues.apache.org/jira/browse/BEAM-6698
+ maxHeapSize = '4g'
+ dependsOn setupTask
+}
+mainTask.dependsOn javaTask
+cleanupTask.mustRunAfter javaTask
+
+// Task for running testcases in Python SDK
+def testOpts = [
+ "--attr=UsesCrossLanguageTransforms"
+
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243727=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243727 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 23:35 Start Date: 16/May/19 23:35 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-493266079 run python precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243727) Time Spent: 11h (was: 10h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 11h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243726=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243726 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 23:34 Start Date: 16/May/19 23:34 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-493266079 run python precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243726) Time Spent: 10h 50m (was: 10h 40m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 10h 50m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243715 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 23:00 Start Date: 16/May/19 23:00 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-493259550 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243715) Time Spent: 10h 40m (was: 10.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 10h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243714 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 23:00 Start Date: 16/May/19 23:00 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-493259550 run java precommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243714) Time Spent: 10.5h (was: 10h 20m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 10.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243666
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 16/May/19 21:35
Start Date: 16/May/19 21:35
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #8174: [BEAM-6683] add
createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r284907859
##
File path: sdks/python/apache_beam/io/external/generate_sequence_test.py
##
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for cross-language generate sequence."""
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import logging
+import os
+import re
+import unittest
+
+from nose.plugins.attrib import attr
+
+from apache_beam.io.external.generate_sequence import GenerateSequence
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@attr('UsesCrossLanguageTransforms')
+class XlangGenerateSequenceTest(unittest.TestCase):
+ def test_generate_sequence(self):
+test_pipeline = TestPipeline()
+port = os.environ.get('EXPANSION_PORT')
Review comment:
We don't need to stage the expansion service jar here since
`GenerateSequence` doesn't depend on extra dependencies.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243666)
Time Spent: 10h 20m (was: 10h 10m)
> Add an integration test suite for cross-language transforms for Flink runner
>
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243643=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243643 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 21:18 Start Date: 16/May/19 21:18 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-493235414 run xvr_flink postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243643) Time Spent: 10h 10m (was: 10h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 10h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243642=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243642 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 21:18 Start Date: 16/May/19 21:18 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-493235414 run xvr_flink postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243642) Time Spent: 10h (was: 9h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 10h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243604=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243604
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 16/May/19 19:33
Start Date: 16/May/19 19:33
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #8174: [BEAM-6683] add
createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r284864864
##
File path: sdks/python/apache_beam/transforms/combiners.py
##
@@ -129,10 +129,13 @@ class PerElement(ptransform.PTransform):
def expand(self, pcoll):
paired_with_void_type = KV[pcoll.element_type, Any]
- return (pcoll
- | ('%s:PairWithVoid' % self.label >> core.Map(lambda x: (x,
None))
- .with_output_types(paired_with_void_type))
- | core.CombinePerKey(CountCombineFn()))
+ output_type = KV[pcoll.element_type, int]
Review comment:
The output type needs to be as narrow as possible in order to avoid python
pickled coder. The test was failed because of it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243604)
Time Spent: 9h 50m (was: 9h 40m)
> Add an integration test suite for cross-language transforms for Flink runner
>
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243597=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243597
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 16/May/19 19:20
Start Date: 16/May/19 19:20
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #8174: [BEAM-6683] add
createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r284860203
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1625,6 +1645,110 @@ class BeamModulePlugin implements Plugin {
/**
***/
+// Method to create the crossLanguageValidatesRunnerTask.
+// The method takes crossLanguageValidatesRunnerConfiguration as parameter.
+project.ext.createCrossLanguageValidatesRunnerTask = {
Review comment:
added the postcommit test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243597)
Time Spent: 9h 40m (was: 9.5h)
> Add an integration test suite for cross-language transforms for Flink runner
>
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243587=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243587 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 19:07 Start Date: 16/May/19 19:07 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-493194159 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243587) Time Spent: 9h 20m (was: 9h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 9h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243588=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243588 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 19:07 Start Date: 16/May/19 19:07 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-493194159 Run XVR_Flink PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243588) Time Spent: 9.5h (was: 9h 20m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 9.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243585=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243585
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 16/May/19 19:05
Start Date: 16/May/19 19:05
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #8174: [BEAM-6683] add
createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r284854976
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ExternalTest.java
##
@@ -86,26 +84,27 @@ public static void tearDown() {
@Test
@Category({ValidatesRunner.class, UsesCrossLanguageTransforms.class})
public void expandSingleTest() {
-PCollection col =
+PCollection col =
testPipeline
-.apply(Create.of(1, 2, 3))
+.apply(Create.of("1", "2", "3"))
Review comment:
The test is modified as close to Python external_test. It was just adding 1
but now concatenating `Simple(%s)`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243585)
Time Spent: 9h 10m (was: 9h)
> Add an integration test suite for cross-language transforms for Flink runner
>
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243584=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243584 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 19:03 Start Date: 16/May/19 19:03 Worklog Time Spent: 10m Work Description: ihji commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r284854091 ## File path: sdks/python/apache_beam/runners/worker/sdk_worker_main.py ## @@ -197,6 +198,19 @@ def _get_worker_count(pipeline_options): return 12 +def _load_avro_coder(pipeline_options): + experiments = pipeline_options.view_as(DebugOptions).experiments + + experiments = experiments if experiments else [] + + for experiment in experiments: +# There should only be 1 match so returning from the loop +if re.match(r'xlang_test', experiment): Review comment: same here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243584) Time Spent: 9h (was: 8h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 9h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243582=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243582 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 19:02 Start Date: 16/May/19 19:02 Worklog Time Spent: 10m Work Description: ihji commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r284853783 ## File path: sdks/python/apache_beam/runners/worker/sdk_worker_main.py ## @@ -136,6 +136,7 @@ def main(unused_argv): service_descriptor = endpoints_pb2.ApiServiceDescriptor() text_format.Merge(os.environ['CONTROL_API_SERVICE_DESCRIPTOR'], service_descriptor) +_load_avro_coder(sdk_pipeline_options) Review comment: Thanks for pointing this out. Will fix this (this is from old design when I thought that Avro coder is only for testing xlang). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243582) Time Spent: 8h 40m (was: 8.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 8h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243583=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243583
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 16/May/19 19:02
Start Date: 16/May/19 19:02
Worklog Time Spent: 10m
Work Description: ihji commented on pull request #8174: [BEAM-6683] add
createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r284853870
##
File path:
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
##
@@ -1625,6 +1645,110 @@ class BeamModulePlugin implements Plugin {
/**
***/
+// Method to create the crossLanguageValidatesRunnerTask.
+// The method takes crossLanguageValidatesRunnerConfiguration as parameter.
+project.ext.createCrossLanguageValidatesRunnerTask = {
+ def config = it ? it as CrossLanguageValidatesRunnerConfiguration : new
CrossLanguageValidatesRunnerConfiguration()
+
+ project.evaluationDependsOn(":sdks:python")
+ project.evaluationDependsOn(":sdks:java:testing:expansion-service")
+ project.evaluationDependsOn(":runners:core-construction-java")
+
+ // Task for launching expansion services
+ def envDir = project.project(":sdks:python").envdir
+ def pythonDir = project.project(":sdks:python").projectDir
+ def javaPort = startingExpansionPortNumber.getAndDecrement()
+ def pythonPort = startingExpansionPortNumber.getAndDecrement()
+ def expansionJar =
project.project(':sdks:java:testing:expansion-service').buildTestExpansionServiceJar.archivePath
+ def expansionServiceOpts = [
+"group_id": project.name,
+"java_expansion_service_jar": expansionJar,
+"java_port": javaPort,
+"python_virtualenv_dir": envDir,
+"python_expansion_service_module":
"apache_beam.runners.portability.expansion_service_test",
+"python_port": pythonPort
+ ]
+ def serviceArgs =
project.project(':sdks:python').mapToArgString(expansionServiceOpts)
+ def setupTask = project.tasks.create(name: config.name+"Setup", type:
Exec) {
+dependsOn ':sdks:java:container:docker'
+dependsOn ':sdks:python:container:docker'
+dependsOn
':sdks:java:testing:expansion-service:buildTestExpansionServiceJar'
+dependsOn ":sdks:python:installGcpTest"
+// setup test env
+executable 'sh'
+args '-c', "$pythonDir/scripts/run_expansion_services.sh stop
--group_id ${project.name} && $pythonDir/scripts/run_expansion_services.sh
start $serviceArgs"
+ }
+
+ def mainTask = project.tasks.create(name: config.name) {
+group = "Verification"
+description = "Validates cross-language capability of runner"
+ }
+
+ def cleanupTask = project.tasks.create(name: config.name+'Cleanup',
type: Exec) {
+// teardown test env
+executable 'sh'
+args '-c', "$pythonDir/scripts/run_expansion_services.sh stop
--group_id ${project.name}"
+ }
+ setupTask.finalizedBy cleanupTask
+
+ // Task for running testcases in Java SDK
+ def beamJavaTestPipelineOptions = [
+
"--runner=org.apache.beam.runners.reference.testing.TestPortableRunner",
+"--jobServerDriver=${config.jobServerDriver}",
+"--environmentCacheMillis=1"
+ ]
+ beamJavaTestPipelineOptions.addAll(config.pipelineOpts)
+ if (config.jobServerConfig) {
+
beamJavaTestPipelineOptions.add("--jobServerConfig=${config.jobServerConfig}")
+ }
+ ['Java': javaPort, 'Python': pythonPort].each { sdk, port ->
+def javaTask = project.tasks.create(name: config.name+"JavaUsing"+sdk,
type: Test) {
+ group = "Verification"
+ description = "Validates runner for cross-language capability of
using ${sdk} transforms from Java SDK"
+ systemProperty "beamTestPipelineOptions",
JsonOutput.toJson(beamJavaTestPipelineOptions)
+ systemProperty "expansionPort", port
+ classpath = config.testClasspathConfiguration
+ testClassesDirs =
project.files(project.project(":runners:core-construction-java").sourceSets.test.output.classesDirs)
+ maxParallelForks config.numParallelTests
+ useJUnit(config.testCategories)
+ // increase maxHeapSize as this is directly correlated to direct
memory,
+ // see https://issues.apache.org/jira/browse/BEAM-6698
+ maxHeapSize = '4g'
+ dependsOn setupTask
+}
+mainTask.dependsOn javaTask
+cleanupTask.mustRunAfter javaTask
+
+// Task for running testcases in Python SDK
+def testOpts = [
+ "--attr=UsesCrossLanguageTransforms"
+]
+def
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243577=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243577 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 18:57 Start Date: 16/May/19 18:57 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-493190945 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243577) Time Spent: 8h 20m (was: 8h 10m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 8h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243578=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243578 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 18:57 Start Date: 16/May/19 18:57 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-493190945 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243578) Time Spent: 8.5h (was: 8h 20m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243576 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 18:57 Start Date: 16/May/19 18:57 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-493185048 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243576) Time Spent: 8h 10m (was: 8h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 8h 10m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243560=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243560 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 18:40 Start Date: 16/May/19 18:40 Worklog Time Spent: 10m Work Description: ihji commented on issue #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#issuecomment-493185048 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243560) Time Spent: 8h (was: 7h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 8h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243556=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243556 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 18:36 Start Date: 16/May/19 18:36 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r284839573 ## File path: sdks/python/apache_beam/runners/worker/sdk_worker_main.py ## @@ -136,6 +136,7 @@ def main(unused_argv): service_descriptor = endpoints_pb2.ApiServiceDescriptor() text_format.Merge(os.environ['CONTROL_API_SERVICE_DESCRIPTOR'], service_descriptor) +_load_avro_coder(sdk_pipeline_options) Review comment: Can you explain why we need to import the AvroCoder here (but not the other coders). Can we load coders in a uniform way ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243556) Time Spent: 7h 40m (was: 7.5h) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 7h 40m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243554=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243554 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 18:36 Start Date: 16/May/19 18:36 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r284840403 ## File path: sdks/python/apache_beam/runners/worker/sdk_worker_main.py ## @@ -197,6 +198,19 @@ def _get_worker_count(pipeline_options): return 12 +def _load_avro_coder(pipeline_options): + experiments = pipeline_options.view_as(DebugOptions).experiments + + experiments = experiments if experiments else [] + + for experiment in experiments: +# There should only be 1 match so returning from the loop +if re.match(r'xlang_test', experiment): Review comment: I think having to do logic specific to a test in sdk_worker_main.py is pretty hacky. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243554) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 7h 20m > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243553=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243553
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 16/May/19 18:36
Start Date: 16/May/19 18:36
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #8174:
[BEAM-6683] add createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r284838231
##
File path: sdks/python/apache_beam/io/external/generate_sequence_test.py
##
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for cross-language generate sequence."""
+
+from __future__ import absolute_import
+from __future__ import print_function
+
+import logging
+import os
+import re
+import unittest
+
+from nose.plugins.attrib import attr
+
+from apache_beam.io.external.generate_sequence import GenerateSequence
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+
+@attr('UsesCrossLanguageTransforms')
+class XlangGenerateSequenceTest(unittest.TestCase):
+ def test_generate_sequence(self):
+test_pipeline = TestPipeline()
+port = os.environ.get('EXPANSION_PORT')
Review comment:
Do we also need to get the expansion jar from the environment similar to
following test ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243553)
Time Spent: 7h 20m (was: 7h 10m)
> Add an integration test suite for cross-language transforms for Flink runner
>
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[
https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243552
]
ASF GitHub Bot logged work on BEAM-6683:
Author: ASF GitHub Bot
Created on: 16/May/19 18:36
Start Date: 16/May/19 18:36
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #8174:
[BEAM-6683] add createCrossLanguageValidatesRunner task
URL: https://github.com/apache/beam/pull/8174#discussion_r284833818
##
File path:
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ExternalTest.java
##
@@ -86,26 +84,27 @@ public static void tearDown() {
@Test
@Category({ValidatesRunner.class, UsesCrossLanguageTransforms.class})
public void expandSingleTest() {
-PCollection col =
+PCollection col =
testPipeline
-.apply(Create.of(1, 2, 3))
+.apply(Create.of("1", "2", "3"))
Review comment:
Was there any reason for changing the type here (from Integer to string) ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 243552)
Time Spent: 7h 10m (was: 7h)
> Add an integration test suite for cross-language transforms for Flink runner
>
>
> Key: BEAM-6683
> URL: https://issues.apache.org/jira/browse/BEAM-6683
> Project: Beam
> Issue Type: Test
> Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Heejong Lee
>Priority: Major
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> We should add an integration test suite that covers following.
> (1) Currently available Java IO connectors that do not use UDFs work for
> Python SDK on Flink runner.
> (2) Currently available Python IO connectors that do not use UDFs work for
> Java SDK on Flink runner.
> (3) Currently available Java/Python pipelines work in a scalable manner for
> cross-language pipelines (for example, try 10GB, 100GB input for
> textio/avroio for Java and Python).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6683) Add an integration test suite for cross-language transforms for Flink runner
[ https://issues.apache.org/jira/browse/BEAM-6683?focusedWorklogId=243551=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-243551 ] ASF GitHub Bot logged work on BEAM-6683: Author: ASF GitHub Bot Created on: 16/May/19 18:36 Start Date: 16/May/19 18:36 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8174: [BEAM-6683] add createCrossLanguageValidatesRunner task URL: https://github.com/apache/beam/pull/8174#discussion_r284834510 ## File path: runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/expansion/TestExpansionService.java ## @@ -1,58 +0,0 @@ -/* Review comment: Please mention this file move in the PR description and commit message. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 243551) Time Spent: 7h (was: 6h 50m) > Add an integration test suite for cross-language transforms for Flink runner > > > Key: BEAM-6683 > URL: https://issues.apache.org/jira/browse/BEAM-6683 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Chamikara Jayalath >Assignee: Heejong Lee >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > > We should add an integration test suite that covers following. > (1) Currently available Java IO connectors that do not use UDFs work for > Python SDK on Flink runner. > (2) Currently available Python IO connectors that do not use UDFs work for > Java SDK on Flink runner. > (3) Currently available Java/Python pipelines work in a scalable manner for > cross-language pipelines (for example, try 10GB, 100GB input for > textio/avroio for Java and Python). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
