[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=319296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319296 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 27/Sep/19 00:54 Start Date: 27/Sep/19 00:54 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9466: [DO NOT MERGE] [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/9466 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319296) Time Spent: 6h 10m (was: 6h) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-py-avro >Reporter: Valentyn Tymofieiev >Assignee: Valentyn Tymofieiev >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=316131=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316131 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 21/Sep/19 12:54 Start Date: 21/Sep/19 12:54 Worklog Time Spent: 10m Work Description: stale[bot] commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-533795908 This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 316131) Time Spent: 5h 50m (was: 5h 40m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-py-avro >Reporter: Valentyn Tymofieiev >Assignee: Valentyn Tymofieiev >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=316132=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316132 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 21/Sep/19 12:54 Start Date: 21/Sep/19 12:54 Worklog Time Spent: 10m Work Description: stale[bot] commented on pull request #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 316132) Time Spent: 6h (was: 5h 50m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-py-avro >Reporter: Valentyn Tymofieiev >Assignee: Valentyn Tymofieiev >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=312538=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312538 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 14/Sep/19 12:07 Start Date: 14/Sep/19 12:07 Worklog Time Spent: 10m Work Description: stale[bot] commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-531474515 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the d...@beam.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 312538) Time Spent: 5h 40m (was: 5.5h) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-py-avro >Reporter: Valentyn Tymofieiev >Assignee: Valentyn Tymofieiev >Priority: Major > Time Spent: 5h 40m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=305736=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305736 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 03/Sep/19 17:54 Start Date: 03/Sep/19 17:54 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #9466: [DO NOT MERGE] [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/9466#issuecomment-527568447 Run Python 3.5 PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 305736) Time Spent: 5.5h (was: 5h 20m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-py-avro >Reporter: Valentyn Tymofieiev >Assignee: Valentyn Tymofieiev >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=305732=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-305732 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 03/Sep/19 17:52 Start Date: 03/Sep/19 17:52 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #9466: [DO NOT MERGE] [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/9466 Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | [![Build
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=277455=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-277455 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 16/Jul/19 11:55 Start Date: 16/Jul/19 11:55 Worklog Time Spent: 10m Work Description: fredo838 commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-511784607 @tvalentyn PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 277455) Time Spent: 5h 10m (was: 5h) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 5h 10m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[
https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=277446=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-277446
]
ASF GitHub Bot logged work on BEAM-7455:
Author: ASF GitHub Bot
Created on: 16/Jul/19 11:44
Start Date: 16/Jul/19 11:44
Worklog Time Spent: 10m
Work Description: fredo838 commented on pull request #8818: [BEAM-7455]
Improve Avro IO integration test coverage on Python 3.
URL: https://github.com/apache/beam/pull/8818#discussion_r303862791
##
File path: sdks/python/apache_beam/io/avroio_it_test.py
##
@@ -0,0 +1,237 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""End-to-end test for Avro IO's.
+
+Integration test for Avro IO. Using both the avro and fastavro
+libraries, this test writes avro files, reads them back in,
+and combines the different fields.
+
+Usage:
+
+ DataFlowRunner:
+python setup.py nosetests --tests apache_beam.io.avroio_it_test\
+--test-pipeline-options="
+ --runner=TestDataflowRunner
+ --project=...
+ --staging_location=gs://...
+ --temp_location=gs://...
+ --output=gs://...
+ --sdk_location=...
+"
+
+ DirectRunner:
+python setup.py nosetests --tests apache_beam.io.avroio_it_test \
+ --test-pipeline-options="
+--output=/tmp
+ "
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import json
+import logging
+import os
+import sys
+import unittest
+import uuid
+
+from fastavro import parse_schema
+from nose.plugins.attrib import attr
+
+from apache_beam.io.avroio import ReadFromAvro
+from apache_beam.io.avroio import WriteToAvro
+from apache_beam.runners.runner import PipelineState
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.test_utils import delete_files
+from apache_beam.testing.util import BeamAssertException
+from apache_beam.transforms import CombineGlobally
+from apache_beam.transforms.combiners import Count
+from apache_beam.transforms.combiners import Mean
+from apache_beam.transforms.combiners import ToList
+from apache_beam.transforms.core import Create
+from apache_beam.transforms.core import FlatMap
+from apache_beam.transforms.core import Map
+
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+ from avro.schema import Parse # avro-python3 library for python3
+except ImportError:
+ from avro.schema import parse as Parse # avro library for python2
+# pylint: enable=wrong-import-order, wrong-import-position
+
+LABELS = ['abc', 'def', 'ghi', 'jkl', 'mno', 'pqr', 'stu', 'vwx']
+COLORS = ['RED', 'ORANGE', 'YELLOW', 'GREEN', 'BLUE', 'PURPLE', None]
+
+
+def record(i):
+ return {
+ 'label': LABELS[i % len(LABELS)],
+ 'number': i,
+ 'number_str': str(i),
+ 'color': COLORS[i % len(COLORS)]
+ }
+
+
+class AvroITTestBase(object):
+
+ def setUp(self):
+self.SCHEMA_STRING = '''
+ {"namespace": "example.avro",
+ "type": "record",
+ "name": "User",
+ "fields": [
+ {"name": "label", "type": "string"},
+ {"name": "number", "type": ["int", "null"]},
+ {"name": "number_str", "type": ["string", "null"]},
+ {"name": "color", "type": ["string", "null"]}
+ ]
+ }
+ '''
+self.uuid = str(uuid.uuid4())
+
+ @attr('IT')
+ def avro_it_test(self):
+write_pipeline = TestPipeline(is_integration_test=True)
+read_pipeline = TestPipeline(is_integration_test=True)
+output = '/'.join([write_pipeline.get_option('output'),
+ self.uuid, 'avro-files'])
+files_output = '/'.join([output, 'avro'])
+
+num_records = write_pipeline.get_option('records')
+num_records = int(num_records) if num_records else 1
+
+# Seed a `PCollection` with indices that will each be FlatMap'd into
+# `batch_size` records, to avoid having a too-large list in memory at
+# the outset
+batch_size = write_pipeline.get_option('batch-size')
+batch_size = int(batch_size) if batch_size else 1000
+
+assert batch_size < num_records
+# pylint:
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=277445=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-277445 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 16/Jul/19 11:44 Start Date: 16/Jul/19 11:44 Worklog Time Spent: 10m Work Description: fredo838 commented on pull request #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#discussion_r303859539 ## File path: sdks/python/apache_beam/io/avroio_it_test.py ## @@ -0,0 +1,237 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +"""End-to-end test for Avro IO's. + +Integration test for Avro IO. Using both the avro and fastavro +libraries, this test writes avro files, reads them back in, Review comment: Old comment valentyn: Please update the test docstring. Resolution: Updated docstring. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 277445) Time Spent: 4h 40m (was: 4.5h) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[
https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=277447=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-277447
]
ASF GitHub Bot logged work on BEAM-7455:
Author: ASF GitHub Bot
Created on: 16/Jul/19 11:44
Start Date: 16/Jul/19 11:44
Worklog Time Spent: 10m
Work Description: fredo838 commented on pull request #8818: [BEAM-7455]
Improve Avro IO integration test coverage on Python 3.
URL: https://github.com/apache/beam/pull/8818#discussion_r30386
##
File path: sdks/python/apache_beam/io/avroio_it_test.py
##
@@ -0,0 +1,237 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""End-to-end test for Avro IO's.
+
+Integration test for Avro IO. Using both the avro and fastavro
+libraries, this test writes avro files, reads them back in,
+and combines the different fields.
+
+Usage:
+
+ DataFlowRunner:
+python setup.py nosetests --tests apache_beam.io.avroio_it_test\
+--test-pipeline-options="
+ --runner=TestDataflowRunner
+ --project=...
+ --staging_location=gs://...
+ --temp_location=gs://...
+ --output=gs://...
+ --sdk_location=...
+"
+
+ DirectRunner:
+python setup.py nosetests --tests apache_beam.io.avroio_it_test \
+ --test-pipeline-options="
+--output=/tmp
+ "
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import json
+import logging
+import os
+import sys
+import unittest
+import uuid
+
+from fastavro import parse_schema
+from nose.plugins.attrib import attr
+
+from apache_beam.io.avroio import ReadFromAvro
+from apache_beam.io.avroio import WriteToAvro
+from apache_beam.runners.runner import PipelineState
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.test_utils import delete_files
+from apache_beam.testing.util import BeamAssertException
+from apache_beam.transforms import CombineGlobally
+from apache_beam.transforms.combiners import Count
+from apache_beam.transforms.combiners import Mean
+from apache_beam.transforms.combiners import ToList
+from apache_beam.transforms.core import Create
+from apache_beam.transforms.core import FlatMap
+from apache_beam.transforms.core import Map
+
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+ from avro.schema import Parse # avro-python3 library for python3
+except ImportError:
+ from avro.schema import parse as Parse # avro library for python2
+# pylint: enable=wrong-import-order, wrong-import-position
+
+LABELS = ['abc', 'def', 'ghi', 'jkl', 'mno', 'pqr', 'stu', 'vwx']
+COLORS = ['RED', 'ORANGE', 'YELLOW', 'GREEN', 'BLUE', 'PURPLE', None]
+
+
+def record(i):
+ return {
+ 'label': LABELS[i % len(LABELS)],
+ 'number': i,
+ 'number_str': str(i),
+ 'color': COLORS[i % len(COLORS)]
+ }
+
+
+class AvroITTestBase(object):
+
+ def setUp(self):
+self.SCHEMA_STRING = '''
+ {"namespace": "example.avro",
+ "type": "record",
+ "name": "User",
+ "fields": [
+ {"name": "label", "type": "string"},
+ {"name": "number", "type": ["int", "null"]},
+ {"name": "number_str", "type": ["string", "null"]},
+ {"name": "color", "type": ["string", "null"]}
+ ]
+ }
+ '''
+self.uuid = str(uuid.uuid4())
+
+ @attr('IT')
+ def avro_it_test(self):
+write_pipeline = TestPipeline(is_integration_test=True)
+read_pipeline = TestPipeline(is_integration_test=True)
+output = '/'.join([write_pipeline.get_option('output'),
+ self.uuid, 'avro-files'])
+files_output = '/'.join([output, 'avro'])
+
+num_records = write_pipeline.get_option('records')
+num_records = int(num_records) if num_records else 1
+
+# Seed a `PCollection` with indices that will each be FlatMap'd into
+# `batch_size` records, to avoid having a too-large list in memory at
+# the outset
+batch_size = write_pipeline.get_option('batch-size')
+batch_size = int(batch_size) if batch_size else 1000
+
+assert batch_size < num_records
+# pylint:
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[
https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=273390=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273390
]
ASF GitHub Bot logged work on BEAM-7455:
Author: ASF GitHub Bot
Created on: 08/Jul/19 17:02
Start Date: 08/Jul/19 17:02
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8818: [BEAM-7455]
Improve Avro IO integration test coverage on Python 3.
URL: https://github.com/apache/beam/pull/8818#discussion_r299590315
##
File path: sdks/python/apache_beam/io/avroio_it_test.py
##
@@ -0,0 +1,236 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""End-to-end test for Avro IO's.
+
+Integration test for Avro IO. Using both the avro and fastavro
+libraries, this test writes avro files, reads them back in,
+and combines the different fields.
+
+Usage:
+
+ DataFlowRunner:
+python setup.py nosetests --tests apache_beam.io.avro_it_test\
+--test-pipeline-options="
+ --runner=TestDataflowRunner
+ --project=...
+ --staging_location=gs://...
+ --temp_location=gs://...
+ --output=gs://...
+ --sdk_location=...
+"
+
+ DirectRunner:
+python setup.py nosetests --tests apache_beam.io.avro_it_test \
+ --test-pipeline-options="
+--output=/tmp
+ "
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import json
+import logging
+import os
+import sys
+import unittest
+import uuid
+
+from fastavro import parse_schema
+from nose.plugins.attrib import attr
+
+from apache_beam.io.avroio import ReadFromAvro
+from apache_beam.io.avroio import WriteToAvro
+from apache_beam.runners.runner import PipelineState
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.test_utils import delete_files
+from apache_beam.testing.util import BeamAssertException
+from apache_beam.transforms import CombineGlobally
+from apache_beam.transforms.combiners import Count
+from apache_beam.transforms.combiners import Mean
+from apache_beam.transforms.combiners import ToList
+from apache_beam.transforms.core import Create
+from apache_beam.transforms.core import FlatMap
+from apache_beam.transforms.core import Map
+
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+ from avro.schema import Parse # avro-python3 library for python3
+except ImportError:
+ from avro.schema import parse as Parse # avro library for python2
+# pylint: enable=wrong-import-order, wrong-import-position
+
+LABELS = ['abc', 'def', 'ghi', 'jkl', 'mno', 'pqr', 'stu', 'vwx']
+COLORS = ['RED', 'ORANGE', 'YELLOW', 'GREEN', 'BLUE', 'PURPLE', None]
+
+
+def record(i):
+ return {
+ 'label': LABELS[i % len(LABELS)],
+ 'number': i,
+ 'number_str': str(i),
+ 'color': COLORS[i % len(COLORS)]
+ }
+
+
+class AvroITTestBase(object):
+
+ def setUp(self):
+self.SCHEMA_STRING = '''
+ {"namespace": "example.avro",
+ "type": "record",
+ "name": "User",
+ "fields": [
+ {"name": "label", "type": "string"},
+ {"name": "number", "type": ["int", "null"]},
+ {"name": "number_str", "type": ["string", "null"]},
+ {"name": "color", "type": ["string", "null"]}
+ ]
+ }
+ '''
+self.uuid = str(uuid.uuid4())
+
+ @attr('IT')
+ def avro_it_test(self):
+write_pipeline = TestPipeline(is_integration_test=True)
+read_pipeline = TestPipeline(is_integration_test=True)
+output = '/'.join([write_pipeline.get_option('output'),
+ self.uuid, 'avro-files'])
+files_output = '/'.join([output, 'avro'])
+
+num_records = write_pipeline.get_option('records')
+num_records = int(num_records) if num_records else 1
+
+# Seed a `PCollection` with indices that will each be FlatMap'd into
+# `batch_size` records, to avoid having a too-large list in memory at
+# the outset
+batch_size = write_pipeline.get_option('batch-size')
+batch_size = int(batch_size) if batch_size else 1000
+
+assert batch_size < num_records
+# pylint:
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=273277=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273277 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 08/Jul/19 12:42 Start Date: 08/Jul/19 12:42 Worklog Time Spent: 10m Work Description: Juta commented on pull request #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#discussion_r301070954 ## File path: sdks/python/apache_beam/io/avroio_it_test.py ## @@ -0,0 +1,236 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +"""End-to-end test for Avro IO's. + +Integration test for Avro IO. Using both the avro and fastavro +libraries, this test writes avro files, reads them back in, +and combines the different fields. + +Usage: + + DataFlowRunner: +python setup.py nosetests --tests apache_beam.io.avro_it_test\ Review comment: s/avro_it_test/avroio_it_test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 273277) Time Spent: 4h 20m (was: 4h 10m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[
https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=271039=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271039
]
ASF GitHub Bot logged work on BEAM-7455:
Author: ASF GitHub Bot
Created on: 02/Jul/19 17:13
Start Date: 02/Jul/19 17:13
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8818: [BEAM-7455]
Improve Avro IO integration test coverage on Python 3.
URL: https://github.com/apache/beam/pull/8818#discussion_r299590315
##
File path: sdks/python/apache_beam/io/avroio_it_test.py
##
@@ -0,0 +1,236 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""End-to-end test for Avro IO's.
+
+Integration test for Avro IO. Using both the avro and fastavro
+libraries, this test writes avro files, reads them back in,
+and combines the different fields.
+
+Usage:
+
+ DataFlowRunner:
+python setup.py nosetests --tests apache_beam.io.avro_it_test\
+--test-pipeline-options="
+ --runner=TestDataflowRunner
+ --project=...
+ --staging_location=gs://...
+ --temp_location=gs://...
+ --output=gs://...
+ --sdk_location=...
+"
+
+ DirectRunner:
+python setup.py nosetests --tests apache_beam.io.avro_it_test \
+ --test-pipeline-options="
+--output=/tmp
+ "
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import json
+import logging
+import os
+import sys
+import unittest
+import uuid
+
+from fastavro import parse_schema
+from nose.plugins.attrib import attr
+
+from apache_beam.io.avroio import ReadFromAvro
+from apache_beam.io.avroio import WriteToAvro
+from apache_beam.runners.runner import PipelineState
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.test_utils import delete_files
+from apache_beam.testing.util import BeamAssertException
+from apache_beam.transforms import CombineGlobally
+from apache_beam.transforms.combiners import Count
+from apache_beam.transforms.combiners import Mean
+from apache_beam.transforms.combiners import ToList
+from apache_beam.transforms.core import Create
+from apache_beam.transforms.core import FlatMap
+from apache_beam.transforms.core import Map
+
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+ from avro.schema import Parse # avro-python3 library for python3
+except ImportError:
+ from avro.schema import parse as Parse # avro library for python2
+# pylint: enable=wrong-import-order, wrong-import-position
+
+LABELS = ['abc', 'def', 'ghi', 'jkl', 'mno', 'pqr', 'stu', 'vwx']
+COLORS = ['RED', 'ORANGE', 'YELLOW', 'GREEN', 'BLUE', 'PURPLE', None]
+
+
+def record(i):
+ return {
+ 'label': LABELS[i % len(LABELS)],
+ 'number': i,
+ 'number_str': str(i),
+ 'color': COLORS[i % len(COLORS)]
+ }
+
+
+class AvroITTestBase(object):
+
+ def setUp(self):
+self.SCHEMA_STRING = '''
+ {"namespace": "example.avro",
+ "type": "record",
+ "name": "User",
+ "fields": [
+ {"name": "label", "type": "string"},
+ {"name": "number", "type": ["int", "null"]},
+ {"name": "number_str", "type": ["string", "null"]},
+ {"name": "color", "type": ["string", "null"]}
+ ]
+ }
+ '''
+self.uuid = str(uuid.uuid4())
+
+ @attr('IT')
+ def avro_it_test(self):
+write_pipeline = TestPipeline(is_integration_test=True)
+read_pipeline = TestPipeline(is_integration_test=True)
+output = '/'.join([write_pipeline.get_option('output'),
+ self.uuid, 'avro-files'])
+files_output = '/'.join([output, 'avro'])
+
+num_records = write_pipeline.get_option('records')
+num_records = int(num_records) if num_records else 1
+
+# Seed a `PCollection` with indices that will each be FlatMap'd into
+# `batch_size` records, to avoid having a too-large list in memory at
+# the outset
+batch_size = write_pipeline.get_option('batch-size')
+batch_size = int(batch_size) if batch_size else 1000
+
+assert batch_size < num_records
+# pylint:
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[
https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=271040=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271040
]
ASF GitHub Bot logged work on BEAM-7455:
Author: ASF GitHub Bot
Created on: 02/Jul/19 17:13
Start Date: 02/Jul/19 17:13
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8818: [BEAM-7455]
Improve Avro IO integration test coverage on Python 3.
URL: https://github.com/apache/beam/pull/8818#discussion_r299586520
##
File path: sdks/python/apache_beam/io/avroio_it_test.py
##
@@ -0,0 +1,236 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""End-to-end test for Avro IO's.
+
+Integration test for Avro IO. Using both the avro and fastavro
+libraries, this test writes avro files, reads them back in,
+and combines the different fields.
+
+Usage:
+
+ DataFlowRunner:
+python setup.py nosetests --tests apache_beam.io.avro_it_test\
+--test-pipeline-options="
+ --runner=TestDataflowRunner
+ --project=...
+ --staging_location=gs://...
+ --temp_location=gs://...
+ --output=gs://...
+ --sdk_location=...
+"
+
+ DirectRunner:
+python setup.py nosetests --tests apache_beam.io.avro_it_test \
+ --test-pipeline-options="
+--output=/tmp
+ "
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import json
+import logging
+import os
+import sys
+import unittest
+import uuid
+
+from fastavro import parse_schema
+from nose.plugins.attrib import attr
+
+from apache_beam.io.avroio import ReadFromAvro
+from apache_beam.io.avroio import WriteToAvro
+from apache_beam.runners.runner import PipelineState
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.test_utils import delete_files
+from apache_beam.testing.util import BeamAssertException
+from apache_beam.transforms import CombineGlobally
+from apache_beam.transforms.combiners import Count
+from apache_beam.transforms.combiners import Mean
+from apache_beam.transforms.combiners import ToList
+from apache_beam.transforms.core import Create
+from apache_beam.transforms.core import FlatMap
+from apache_beam.transforms.core import Map
+
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+ from avro.schema import Parse # avro-python3 library for python3
+except ImportError:
+ from avro.schema import parse as Parse # avro library for python2
+# pylint: enable=wrong-import-order, wrong-import-position
+
+LABELS = ['abc', 'def', 'ghi', 'jkl', 'mno', 'pqr', 'stu', 'vwx']
+COLORS = ['RED', 'ORANGE', 'YELLOW', 'GREEN', 'BLUE', 'PURPLE', None]
+
+
+def record(i):
+ return {
+ 'label': LABELS[i % len(LABELS)],
+ 'number': i,
+ 'number_str': str(i),
+ 'color': COLORS[i % len(COLORS)]
+ }
+
+
+class AvroITTestBase(object):
+
+ def setUp(self):
+self.SCHEMA_STRING = '''
+ {"namespace": "example.avro",
+ "type": "record",
+ "name": "User",
+ "fields": [
+ {"name": "label", "type": "string"},
+ {"name": "number", "type": ["int", "null"]},
+ {"name": "number_str", "type": ["string", "null"]},
+ {"name": "color", "type": ["string", "null"]}
+ ]
+ }
+ '''
+self.uuid = str(uuid.uuid4())
+
+ @attr('IT')
+ def avro_it_test(self):
+write_pipeline = TestPipeline(is_integration_test=True)
+read_pipeline = TestPipeline(is_integration_test=True)
+output = '/'.join([write_pipeline.get_option('output'),
+ self.uuid, 'avro-files'])
+files_output = '/'.join([output, 'avro'])
+
+num_records = write_pipeline.get_option('records')
+num_records = int(num_records) if num_records else 1
+
+# Seed a `PCollection` with indices that will each be FlatMap'd into
+# `batch_size` records, to avoid having a too-large list in memory at
+# the outset
+batch_size = write_pipeline.get_option('batch-size')
+batch_size = int(batch_size) if batch_size else 1000
+
+assert batch_size < num_records
+# pylint:
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[
https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=271038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271038
]
ASF GitHub Bot logged work on BEAM-7455:
Author: ASF GitHub Bot
Created on: 02/Jul/19 17:13
Start Date: 02/Jul/19 17:13
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8818: [BEAM-7455]
Improve Avro IO integration test coverage on Python 3.
URL: https://github.com/apache/beam/pull/8818#discussion_r299589605
##
File path: sdks/python/apache_beam/io/avroio_it_test.py
##
@@ -0,0 +1,236 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""End-to-end test for Avro IO's.
+
+Integration test for Avro IO. Using both the avro and fastavro
+libraries, this test writes avro files, reads them back in,
+and combines the different fields.
+
+Usage:
+
+ DataFlowRunner:
+python setup.py nosetests --tests apache_beam.io.avro_it_test\
+--test-pipeline-options="
+ --runner=TestDataflowRunner
+ --project=...
+ --staging_location=gs://...
+ --temp_location=gs://...
+ --output=gs://...
+ --sdk_location=...
+"
+
+ DirectRunner:
+python setup.py nosetests --tests apache_beam.io.avro_it_test \
+ --test-pipeline-options="
+--output=/tmp
+ "
+"""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import json
+import logging
+import os
+import sys
+import unittest
+import uuid
+
+from fastavro import parse_schema
+from nose.plugins.attrib import attr
+
+from apache_beam.io.avroio import ReadFromAvro
+from apache_beam.io.avroio import WriteToAvro
+from apache_beam.runners.runner import PipelineState
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.test_utils import delete_files
+from apache_beam.testing.util import BeamAssertException
+from apache_beam.transforms import CombineGlobally
+from apache_beam.transforms.combiners import Count
+from apache_beam.transforms.combiners import Mean
+from apache_beam.transforms.combiners import ToList
+from apache_beam.transforms.core import Create
+from apache_beam.transforms.core import FlatMap
+from apache_beam.transforms.core import Map
+
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+ from avro.schema import Parse # avro-python3 library for python3
+except ImportError:
+ from avro.schema import parse as Parse # avro library for python2
+# pylint: enable=wrong-import-order, wrong-import-position
+
+LABELS = ['abc', 'def', 'ghi', 'jkl', 'mno', 'pqr', 'stu', 'vwx']
+COLORS = ['RED', 'ORANGE', 'YELLOW', 'GREEN', 'BLUE', 'PURPLE', None]
+
+
+def record(i):
+ return {
+ 'label': LABELS[i % len(LABELS)],
+ 'number': i,
+ 'number_str': str(i),
+ 'color': COLORS[i % len(COLORS)]
+ }
+
+
+class AvroITTestBase(object):
+
+ def setUp(self):
+self.SCHEMA_STRING = '''
+ {"namespace": "example.avro",
+ "type": "record",
+ "name": "User",
+ "fields": [
+ {"name": "label", "type": "string"},
+ {"name": "number", "type": ["int", "null"]},
+ {"name": "number_str", "type": ["string", "null"]},
+ {"name": "color", "type": ["string", "null"]}
+ ]
+ }
+ '''
+self.uuid = str(uuid.uuid4())
+
+ @attr('IT')
+ def avro_it_test(self):
+write_pipeline = TestPipeline(is_integration_test=True)
+read_pipeline = TestPipeline(is_integration_test=True)
+output = '/'.join([write_pipeline.get_option('output'),
+ self.uuid, 'avro-files'])
+files_output = '/'.join([output, 'avro'])
+
+num_records = write_pipeline.get_option('records')
+num_records = int(num_records) if num_records else 1
+
+# Seed a `PCollection` with indices that will each be FlatMap'd into
+# `batch_size` records, to avoid having a too-large list in memory at
+# the outset
+batch_size = write_pipeline.get_option('batch-size')
+batch_size = int(batch_size) if batch_size else 1000
+
+assert batch_size < num_records
+# pylint:
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=266813=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266813 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 25/Jun/19 16:17 Start Date: 25/Jun/19 16:17 Worklog Time Spent: 10m Work Description: fredo838 commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-505516654 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266813) Time Spent: 3h 50m (was: 3h 40m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=266536=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266536 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 25/Jun/19 11:03 Start Date: 25/Jun/19 11:03 Worklog Time Spent: 10m Work Description: fredo838 commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-505395669 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266536) Time Spent: 3h 40m (was: 3.5h) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=266494=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266494 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 25/Jun/19 09:49 Start Date: 25/Jun/19 09:49 Worklog Time Spent: 10m Work Description: fredo838 commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-505372097 @tvalentyn Could you take a look why the `WordcountBatchTest` in `Portable_Python` is failing? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266494) Time Spent: 3.5h (was: 3h 20m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=266493=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266493 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 25/Jun/19 09:48 Start Date: 25/Jun/19 09:48 Worklog Time Spent: 10m Work Description: fredo838 commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-505371785 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266493) Time Spent: 3h 20m (was: 3h 10m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=266483=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266483 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 25/Jun/19 09:37 Start Date: 25/Jun/19 09:37 Worklog Time Spent: 10m Work Description: fredo838 commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-505367818 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266483) Time Spent: 3h 10m (was: 3h) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=266102=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266102 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 24/Jun/19 20:44 Start Date: 24/Jun/19 20:44 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-505173482 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266102) Time Spent: 3h (was: 2h 50m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=266101=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266101 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 24/Jun/19 20:43 Start Date: 24/Jun/19 20:43 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-505173391 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266101) Time Spent: 2h 50m (was: 2h 40m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=266100=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266100 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 24/Jun/19 20:43 Start Date: 24/Jun/19 20:43 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-505173276 Thanks LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266100) Time Spent: 2h 40m (was: 2.5h) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=265576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265576 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 24/Jun/19 08:08 Start Date: 24/Jun/19 08:08 Worklog Time Spent: 10m Work Description: fredo838 commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-504905231 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 265576) Time Spent: 2h 20m (was: 2h 10m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=265577=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-265577 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 24/Jun/19 08:08 Start Date: 24/Jun/19 08:08 Worklog Time Spent: 10m Work Description: fredo838 commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-504905408 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 265577) Time Spent: 2.5h (was: 2h 20m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=263914=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-263914 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 20/Jun/19 16:58 Start Date: 20/Jun/19 16:58 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-504103832 Please fix lint failures, @fredo838. Gentle reminder to please take a look @chamikaramj . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 263914) Time Spent: 2h 10m (was: 2h) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=263792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-263792 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 20/Jun/19 13:54 Start Date: 20/Jun/19 13:54 Worklog Time Spent: 10m Work Description: fredo838 commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-504034962 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 263792) Time Spent: 2h (was: 1h 50m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=261950=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261950 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 18/Jun/19 01:25 Start Date: 18/Jun/19 01:25 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-502908822 R: @chamikaramj - PTAL, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 261950) Time Spent: 1h 50m (was: 1h 40m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[
https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=261948=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261948
]
ASF GitHub Bot logged work on BEAM-7455:
Author: ASF GitHub Bot
Created on: 18/Jun/19 01:24
Start Date: 18/Jun/19 01:24
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8818: [BEAM-7455]
Improve Avro IO integration test coverage on Python 3.
URL: https://github.com/apache/beam/pull/8818#discussion_r294578076
##
File path: sdks/python/apache_beam/io/avroio_it_test.py
##
@@ -0,0 +1,236 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""End-to-end test for Avro IO's.
+
+Writes a configurable number of records to a temporary location with each of
+{avro,fastavro}, then reads them back in, joins the two read datasets, and
Review comment:
This looks stale, @fredo838
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 261948)
Time Spent: 1.5h (was: 1h 20m)
> Improve Avro IO integration test coverage on Python 3.
> --
>
> Key: BEAM-7455
> URL: https://issues.apache.org/jira/browse/BEAM-7455
> Project: Beam
> Issue Type: Sub-task
> Components: io-python-avro
>Reporter: Valentyn Tymofieiev
>Assignee: Frederik Bode
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> It seems that we don't have an integration test for Avro IO on Python 3:
> fastavro_it_test [1] depends on both avro and fastavro, however avro package
> currently does not work with Beam on Python 3, so we don't have an
> integration test that exercises Avro IO on Python 3.
> We should add an integration test for Avro IO that does not need both
> libraries at the same time, and instead can run using either library.
> [~frederik] is this something you could help with?
> cc: [~chamikara] [~Juta]
> [1]
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[
https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=261949=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261949
]
ASF GitHub Bot logged work on BEAM-7455:
Author: ASF GitHub Bot
Created on: 18/Jun/19 01:24
Start Date: 18/Jun/19 01:24
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8818: [BEAM-7455]
Improve Avro IO integration test coverage on Python 3.
URL: https://github.com/apache/beam/pull/8818#discussion_r294578076
##
File path: sdks/python/apache_beam/io/avroio_it_test.py
##
@@ -0,0 +1,236 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""End-to-end test for Avro IO's.
+
+Writes a configurable number of records to a temporary location with each of
+{avro,fastavro}, then reads them back in, joins the two read datasets, and
Review comment:
Please update the test docstring.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 261949)
Time Spent: 1h 40m (was: 1.5h)
> Improve Avro IO integration test coverage on Python 3.
> --
>
> Key: BEAM-7455
> URL: https://issues.apache.org/jira/browse/BEAM-7455
> Project: Beam
> Issue Type: Sub-task
> Components: io-python-avro
>Reporter: Valentyn Tymofieiev
>Assignee: Frederik Bode
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> It seems that we don't have an integration test for Avro IO on Python 3:
> fastavro_it_test [1] depends on both avro and fastavro, however avro package
> currently does not work with Beam on Python 3, so we don't have an
> integration test that exercises Avro IO on Python 3.
> We should add an integration test for Avro IO that does not need both
> libraries at the same time, and instead can run using either library.
> [~frederik] is this something you could help with?
> cc: [~chamikara] [~Juta]
> [1]
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[
https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=261947=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261947
]
ASF GitHub Bot logged work on BEAM-7455:
Author: ASF GitHub Bot
Created on: 18/Jun/19 01:23
Start Date: 18/Jun/19 01:23
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8818: [BEAM-7455]
Improve Avro IO integration test coverage on Python 3.
URL: https://github.com/apache/beam/pull/8818#discussion_r294578005
##
File path: sdks/python/apache_beam/io/avroio_it_test.py
##
@@ -132,73 +128,54 @@ def batch_indices(start):
return range(start * batch_size, (start + 1) * batch_size)
# A `PCollection` with `num_records` avro records
-records_pcoll = \
-self.test_pipeline \
-| 'create-batches' >> Create(batches) \
-| 'expand-batches' >> FlatMap(batch_indices) \
-| 'create-records' >> Map(record)
-
-fastavro_output = '/'.join([self.output, 'fastavro'])
-avro_output = '/'.join([self.output, 'avro'])
-
-# pylint: disable=expression-not-assigned
-records_pcoll \
-| 'write_fastavro' >> WriteToAvro(
-fastavro_output,
-parse_schema(json.loads(self.SCHEMA_STRING)),
-use_fastavro=True
-)
-
-# pylint: disable=expression-not-assigned
-records_pcoll \
-| 'write_avro' >> WriteToAvro(
-avro_output,
-Parse(self.SCHEMA_STRING),
-use_fastavro=False
-)
-
-result = self.test_pipeline.run()
-result.wait_until_finish()
-assert result.state == PipelineState.DONE
-
-fastavro_read_pipeline = TestPipeline(is_integration_test=True)
-
-fastavro_records = \
-fastavro_read_pipeline \
-| 'create-fastavro' >> Create(['%s*' % fastavro_output]) \
-| 'read-fastavro' >> ReadAllFromAvro(use_fastavro=True) \
-| Map(lambda rec: (rec['number'], rec))
-
-avro_records = \
-fastavro_read_pipeline \
-| 'create-avro' >> Create(['%s*' % avro_output]) \
-| 'read-avro' >> ReadAllFromAvro(use_fastavro=False) \
-| Map(lambda rec: (rec['number'], rec))
-
-def check(elem):
- v = elem[1]
-
- def assertEqual(l, r):
-if l != r:
- raise BeamAssertException('Assertion failed: %s == %s' % (l, r))
-
- assertEqual(v.keys(), ['avro', 'fastavro'])
- avro_values = v['avro']
- fastavro_values = v['fastavro']
- assertEqual(avro_values, fastavro_values)
- assertEqual(len(avro_values), 1)
-
-# pylint: disable=expression-not-assigned
-{
-'avro': avro_records,
-'fastavro': fastavro_records
-} \
-| CoGroupByKey() \
-| Map(check)
-
-self.addCleanup(delete_files, [self.output])
-fastavro_read_pipeline.run().wait_until_finish()
-assert result.state == PipelineState.DONE
+_ = (write_pipeline
+ | 'CreateBatches' >> Create(batches)
+ | 'ExpandBatches' >> FlatMap(batch_indices)
+ | 'CreateRecords' >> Map(record)
+ | 'WriteAvro' >> WriteToAvro(files_output,
+ self.SCHEMA,
+ file_name_suffix=".avro",
+ use_fastavro=self.use_fastavro)
+)
+
+write_result = write_pipeline.run()
+write_result.wait_until_finish()
+assert write_result.state == PipelineState.DONE
+
+def check(x, mean):
+ if not x == mean:
+raise BeamAssertException('Assertion failed: %s == %s' % (x, mean))
+
+_ = (read_pipeline
+ | "ReadAvro" >> ReadFromAvro(files_output + '*')
+ | 'ExtractNumber' >> Map(lambda x: x['number'])
+ | 'CalculateMean' >> Mean.Globally()
+ | 'MeanIsCorrect' >> Map(
Review comment:
resolved in new pr iteration.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 261947)
Time Spent: 1h 20m (was: 1h 10m)
> Improve Avro IO integration test coverage on Python 3.
> --
>
> Key: BEAM-7455
> URL: https://issues.apache.org/jira/browse/BEAM-7455
> Project: Beam
> Issue Type: Sub-task
> Components: io-python-avro
>Reporter: Valentyn Tymofieiev
>Assignee: Frederik Bode
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> It seems that we don't have an integration test for Avro IO on Python 3:
> fastavro_it_test [1] depends on both avro and fastavro, however
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=261368=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261368 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 17/Jun/19 14:07 Start Date: 17/Jun/19 14:07 Worklog Time Spent: 10m Work Description: fredo838 commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-502697299 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 261368) Time Spent: 1h 10m (was: 1h) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=257953=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-257953 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 11/Jun/19 17:34 Start Date: 11/Jun/19 17:34 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-500946577 also R: @chamikaramj who may have input on what this test should look like and will likely be helping with the merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 257953) Time Spent: 1h (was: 50m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[
https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=257948=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-257948
]
ASF GitHub Bot logged work on BEAM-7455:
Author: ASF GitHub Bot
Created on: 11/Jun/19 17:33
Start Date: 11/Jun/19 17:33
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8818: [BEAM-7455]
Improve Avro IO integration test coverage on Python 3.
URL: https://github.com/apache/beam/pull/8818#discussion_r292576662
##
File path: sdks/python/apache_beam/io/avroio_it_test.py
##
@@ -132,73 +128,54 @@ def batch_indices(start):
return range(start * batch_size, (start + 1) * batch_size)
# A `PCollection` with `num_records` avro records
-records_pcoll = \
-self.test_pipeline \
-| 'create-batches' >> Create(batches) \
-| 'expand-batches' >> FlatMap(batch_indices) \
-| 'create-records' >> Map(record)
-
-fastavro_output = '/'.join([self.output, 'fastavro'])
-avro_output = '/'.join([self.output, 'avro'])
-
-# pylint: disable=expression-not-assigned
-records_pcoll \
-| 'write_fastavro' >> WriteToAvro(
-fastavro_output,
-parse_schema(json.loads(self.SCHEMA_STRING)),
-use_fastavro=True
-)
-
-# pylint: disable=expression-not-assigned
-records_pcoll \
-| 'write_avro' >> WriteToAvro(
-avro_output,
-Parse(self.SCHEMA_STRING),
-use_fastavro=False
-)
-
-result = self.test_pipeline.run()
-result.wait_until_finish()
-assert result.state == PipelineState.DONE
-
-fastavro_read_pipeline = TestPipeline(is_integration_test=True)
-
-fastavro_records = \
-fastavro_read_pipeline \
-| 'create-fastavro' >> Create(['%s*' % fastavro_output]) \
-| 'read-fastavro' >> ReadAllFromAvro(use_fastavro=True) \
-| Map(lambda rec: (rec['number'], rec))
-
-avro_records = \
-fastavro_read_pipeline \
-| 'create-avro' >> Create(['%s*' % avro_output]) \
-| 'read-avro' >> ReadAllFromAvro(use_fastavro=False) \
-| Map(lambda rec: (rec['number'], rec))
-
-def check(elem):
- v = elem[1]
-
- def assertEqual(l, r):
-if l != r:
- raise BeamAssertException('Assertion failed: %s == %s' % (l, r))
-
- assertEqual(v.keys(), ['avro', 'fastavro'])
- avro_values = v['avro']
- fastavro_values = v['fastavro']
- assertEqual(avro_values, fastavro_values)
- assertEqual(len(avro_values), 1)
-
-# pylint: disable=expression-not-assigned
-{
-'avro': avro_records,
-'fastavro': fastavro_records
-} \
-| CoGroupByKey() \
-| Map(check)
-
-self.addCleanup(delete_files, [self.output])
-fastavro_read_pipeline.run().wait_until_finish()
-assert result.state == PipelineState.DONE
+_ = (write_pipeline
+ | 'CreateBatches' >> Create(batches)
+ | 'ExpandBatches' >> FlatMap(batch_indices)
+ | 'CreateRecords' >> Map(record)
+ | 'WriteAvro' >> WriteToAvro(files_output,
+ self.SCHEMA,
+ file_name_suffix=".avro",
+ use_fastavro=self.use_fastavro)
+)
+
+write_result = write_pipeline.run()
+write_result.wait_until_finish()
+assert write_result.state == PipelineState.DONE
+
+def check(x, mean):
+ if not x == mean:
+raise BeamAssertException('Assertion failed: %s == %s' % (x, mean))
+
+_ = (read_pipeline
+ | "ReadAvro" >> ReadFromAvro(files_output + '*')
+ | 'ExtractNumber' >> Map(lambda x: x['number'])
+ | 'CalculateMean' >> Mean.Globally()
+ | 'MeanIsCorrect' >> Map(
Review comment:
This test evaluates correctness only of a value derived from 'number' field.
Can we instead compare that all attributes of the record are written and read
correctly? Also, comparing float numbers with `==` is not reliable due to a
lack of precision in the float numbers representation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 257948)
Time Spent: 50m (was: 40m)
> Improve Avro IO integration test coverage on Python 3.
> --
>
> Key: BEAM-7455
> URL: https://issues.apache.org/jira/browse/BEAM-7455
> Project: Beam
> Issue Type: Sub-task
> Components: io-python-avro
>Reporter: Valentyn Tymofieiev
>
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=257856=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-257856 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 11/Jun/19 15:37 Start Date: 11/Jun/19 15:37 Worklog Time Spent: 10m Work Description: fredo838 commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-500898217 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 257856) Time Spent: 40m (was: 0.5h) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=257849=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-257849 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 11/Jun/19 15:29 Start Date: 11/Jun/19 15:29 Worklog Time Spent: 10m Work Description: fredo838 commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-500894317 CC: @Juta This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 257849) Time Spent: 0.5h (was: 20m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=257848=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-257848 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 11/Jun/19 15:28 Start Date: 11/Jun/19 15:28 Worklog Time Spent: 10m Work Description: fredo838 commented on issue #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818#issuecomment-500894008 R: @tvalentyn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 257848) Time Spent: 20m (was: 10m) > Improve Avro IO integration test coverage on Python 3. > -- > > Key: BEAM-7455 > URL: https://issues.apache.org/jira/browse/BEAM-7455 > Project: Beam > Issue Type: Sub-task > Components: io-python-avro >Reporter: Valentyn Tymofieiev >Assignee: Frederik Bode >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > It seems that we don't have an integration test for Avro IO on Python 3: > fastavro_it_test [1] depends on both avro and fastavro, however avro package > currently does not work with Beam on Python 3, so we don't have an > integration test that exercises Avro IO on Python 3. > We should add an integration test for Avro IO that does not need both > libraries at the same time, and instead can run using either library. > [~frederik] is this something you could help with? > cc: [~chamikara] [~Juta] > [1] > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/fastavro_it_test.py -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7455) Improve Avro IO integration test coverage on Python 3.
[ https://issues.apache.org/jira/browse/BEAM-7455?focusedWorklogId=257847=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-257847 ] ASF GitHub Bot logged work on BEAM-7455: Author: ASF GitHub Bot Created on: 11/Jun/19 15:28 Start Date: 11/Jun/19 15:28 Worklog Time Spent: 10m Work Description: fredo838 commented on pull request #8818: [BEAM-7455] Improve Avro IO integration test coverage on Python 3. URL: https://github.com/apache/beam/pull/8818 This is is part of a series of PRs with goal to make Apache Beam PY3 compatible. The proposal with the outlined approach has been documented here: https://s.apache.org/beam-python-3. This PR adds a new integration test for AvroIO. The test tests both python 2 and 3, and both the avro library and the fastavro library. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [