[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=271241&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271241 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 02/Jul/19 22:01 Start Date: 02/Jul/19 22:01 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8878: [BEAM-5315] improve test coverage bigquery special chars URL: https://github.com/apache/beam/pull/8878 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271241) Time Spent: 22h 10m (was: 22h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Juta Staes >Priority: Major > Fix For: Not applicable > > Time Spent: 22h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=271010&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271010 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 02/Jul/19 16:22 Start Date: 02/Jul/19 16:22 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #8878: [BEAM-5315] improve test coverage bigquery special chars URL: https://github.com/apache/beam/pull/8878#issuecomment-507748765 @chamikaramj could you please help merge? I checked that modified tests have passed in postcommit suites, but we can re-run if you don't feel comfortable. Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271010) Time Spent: 22h (was: 21h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Juta Staes >Priority: Major > Fix For: Not applicable > > Time Spent: 22h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=266696&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266696 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 25/Jun/19 14:17 Start Date: 25/Jun/19 14:17 Worklog Time Spent: 10m Work Description: Juta commented on issue #8878: [BEAM-5315] improve test coverage bigquery special chars URL: https://github.com/apache/beam/pull/8878#issuecomment-505465413 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266696) Time Spent: 21h 50m (was: 21h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Juta Staes >Priority: Major > Fix For: Not applicable > > Time Spent: 21h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=266693&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266693 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 25/Jun/19 14:16 Start Date: 25/Jun/19 14:16 Worklog Time Spent: 10m Work Description: Juta commented on issue #8878: [BEAM-5315] improve test coverage bigquery special chars URL: https://github.com/apache/beam/pull/8878#issuecomment-505464947 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266693) Time Spent: 21h 40m (was: 21.5h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Juta Staes >Priority: Major > Fix For: Not applicable > > Time Spent: 21h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=266616&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266616 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 25/Jun/19 12:52 Start Date: 25/Jun/19 12:52 Worklog Time Spent: 10m Work Description: Juta commented on issue #8878: [BEAM-5315] improve test coverage bigquery special chars URL: https://github.com/apache/beam/pull/8878#issuecomment-505430512 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266616) Time Spent: 21.5h (was: 21h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Juta Staes >Priority: Major > Fix For: Not applicable > > Time Spent: 21.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=266565&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266565 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 25/Jun/19 11:31 Start Date: 25/Jun/19 11:31 Worklog Time Spent: 10m Work Description: Juta commented on issue #8878: [BEAM-5315] improve test coverage bigquery special chars URL: https://github.com/apache/beam/pull/8878#issuecomment-505403936 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266565) Time Spent: 21h 20m (was: 21h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Juta Staes >Priority: Major > Fix For: Not applicable > > Time Spent: 21h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=266563&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266563 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 25/Jun/19 11:30 Start Date: 25/Jun/19 11:30 Worklog Time Spent: 10m Work Description: Juta commented on issue #8878: [BEAM-5315] improve test coverage bigquery special chars URL: https://github.com/apache/beam/pull/8878#issuecomment-505402086 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266563) Time Spent: 21h 10m (was: 21h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Juta Staes >Priority: Major > Fix For: Not applicable > > Time Spent: 21h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=266559&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266559 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 25/Jun/19 11:25 Start Date: 25/Jun/19 11:25 Worklog Time Spent: 10m Work Description: Juta commented on issue #8878: [BEAM-5315] improve test coverage bigquery special chars URL: https://github.com/apache/beam/pull/8878#issuecomment-505402086 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266559) Time Spent: 21h (was: 20h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Juta Staes >Priority: Major > Fix For: Not applicable > > Time Spent: 21h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=266467&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266467 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 25/Jun/19 09:05 Start Date: 25/Jun/19 09:05 Worklog Time Spent: 10m Work Description: Juta commented on issue #8878: [BEAM-5315] improve test coverage bigquery special chars URL: https://github.com/apache/beam/pull/8878#issuecomment-505356694 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 266467) Time Spent: 20h 50m (was: 20h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Juta Staes >Priority: Major > Fix For: Not applicable > > Time Spent: 20h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=261975&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261975
]
ASF GitHub Bot logged work on BEAM-5315:
Author: ASF GitHub Bot
Created on: 18/Jun/19 02:23
Start Date: 18/Jun/19 02:23
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8878: [BEAM-5315]
improve test coverage bigquery special chars
URL: https://github.com/apache/beam/pull/8878#discussion_r294587655
##
File path: sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
##
@@ -96,6 +96,29 @@ def create_table(self, tablename):
self.bigquery_client.insert_rows(
self.project, self.dataset_id, tablename, table_data)
+ def create_table_non_ascii(self, tablename):
+table_schema = bigquery.TableSchema()
+table_field = bigquery.TableFieldSchema()
+table_field.name = 'str'
+table_field.type = 'STRING'
+table_schema.fields.append(table_field)
+table = bigquery.Table(
+tableReference=bigquery.TableReference(
+projectId=self.project,
+datasetId=self.dataset_id,
+tableId=tablename),
+schema=table_schema)
+request = bigquery.BigqueryTablesInsertRequest(
+projectId=self.project, datasetId=self.dataset_id, table=table)
+self.bigquery_client.client.tables.Insert(request)
+table_data = [
+{'str': '你好'},
Review comment:
I'd suggest we combine this scenario with standard read scenario to avoid
starting extra pipeline and reduce flakiness. WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 261975)
Time Spent: 20h 40m (was: 20.5h)
> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Juta Staes
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 20h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=261522&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261522 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 17/Jun/19 16:35 Start Date: 17/Jun/19 16:35 Worklog Time Spent: 10m Work Description: Juta commented on issue #8878: [BEAM-5315] improve test coverage bigquery special chars URL: https://github.com/apache/beam/pull/8878#issuecomment-502758696 R: @tvalentyn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 261522) Time Spent: 20.5h (was: 20h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Juta Staes >Priority: Major > Fix For: Not applicable > > Time Spent: 20.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=261423&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261423 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 17/Jun/19 15:05 Start Date: 17/Jun/19 15:05 Worklog Time Spent: 10m Work Description: Juta commented on issue #8878: [BEAM-5315] improve test coverage bigquery special chars URL: https://github.com/apache/beam/pull/8878#issuecomment-502722068 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 261423) Time Spent: 20h 20m (was: 20h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Juta Staes >Priority: Major > Fix For: Not applicable > > Time Spent: 20h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=261420&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261420 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 17/Jun/19 15:03 Start Date: 17/Jun/19 15:03 Worklog Time Spent: 10m Work Description: Juta commented on pull request #8878: [BEAM-5315] improve test coverage bigquery special chars URL: https://github.com/apache/beam/pull/8878 Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) Pre-Commit Tests Status (on master branch) --- |Java | Python | Go | Website --- | --- | --- | --- | --- Non-porta
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=259872&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-259872 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 13/Jun/19 20:04 Start Date: 13/Jun/19 20:04 Worklog Time Spent: 10m Work Description: markflyhigh commented on pull request #8852: [BEAM-5315] Fix Py3 incompatibility in tfrecordio_test. URL: https://github.com/apache/beam/pull/8852 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 259872) Time Spent: 20h (was: 19h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Juta Staes >Priority: Major > Fix For: Not applicable > > Time Spent: 20h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=259838&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-259838 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 13/Jun/19 19:01 Start Date: 13/Jun/19 19:01 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #8852: [BEAM-5315] Fix Py3 incompatibility in tfrecordio_test. URL: https://github.com/apache/beam/pull/8852#issuecomment-501837763 R: @markflyhigh This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 259838) Time Spent: 19h 50m (was: 19h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Juta Staes >Priority: Major > Fix For: Not applicable > > Time Spent: 19h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=259834&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-259834 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 13/Jun/19 18:58 Start Date: 13/Jun/19 18:58 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #8852: [BEAM-5315] Fix Py3 incompatibility in tfrecordio_test. URL: https://github.com/apache/beam/pull/8852 Beam PreCommit tests didn't catch this since we don't run a suite with Tensorflow installed. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [ Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=198269&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198269 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 13/Feb/19 18:42 Start Date: 13/Feb/19 18:42 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #7804: [BEAM-5315] python 3 datastore fail gracefully URL: https://github.com/apache/beam/pull/7804 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198269) Time Spent: 19.5h (was: 19h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Labels: triaged > Time Spent: 19.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=198255&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198255 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 13/Feb/19 18:20 Start Date: 13/Feb/19 18:20 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #7804: [BEAM-5315] python 3 datastore fail gracefully URL: https://github.com/apache/beam/pull/7804#discussion_r256525248 ## File path: sdks/python/tox.ini ## @@ -93,7 +93,7 @@ setenv = RUN_SKIPPED_PY3_TESTS=0 extras = test,gcp modules = - apache_beam.typehints,apache_beam.coders,apache_beam.options,apache_beam.tools,apache_beam.utils,apache_beam.internal,apache_beam.metrics,apache_beam.portability,apache_beam.pipeline_test,apache_beam.pvalue_test,apache_beam.runners,apache_beam.io.hadoopfilesystem_test,apache_beam.io.gcp.tests.utils_test,apache_beam.io.gcp.big_query_query_to_table_it_test,apache_beam.io.gcp.bigquery_io_read_it_test,apache_beam.io.gcp.bigquery_test,apache_beam.io.gcp.pubsub_integration_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.internal,apache_beam.io.filesystem_test,apache_beam.io.filesystems_test,apache_beam.io.sources_test,apache_beam.transforms,apache_beam.testing,apache_beam.io.filesystemio_test,apache_beam.io.localfilesystem_test,apache_beam.io.range_trackers_test,apache_beam.io.restriction_trackers_test,apache_beam.io.source_test_utils_test,apache_beam.io.concat_source_test,apache_beam.io.filebasedsink_test,apache_beam.io.filebasedsource_test,apache_beam.io.textio_test,apache_beam.io.tfrecordio_test,apache_beam.examples.wordcount_debugging_test,apache_beam.examples.wordcount_minimal_test,apache_beam.examples.wordcount_test,apache_beam.io.parquetio_test,apache_beam.io.gcp.gcsfilesystem_test,apache_beam.io.gcp.gcsio_integration_test,apache_beam.io.gcp.bigquery_test,apache_beam.io.gcp.big_query_query_to_table_it_test,apache_beam.io.gcp.bigquery_io_read_it_test,apache_beam.io.gcp.bigquery_tools_test,apache_beam.io.gcp.pubsub_integration_test,apache_beam.io.gcp.pubsub_test + apache_beam.typehints,apache_beam.coders,apache_beam.options,apache_beam.tools,apache_beam.utils,apache_beam.internal,apache_beam.metrics,apache_beam.portability,apache_beam.pipeline_test,apache_beam.pvalue_test,apache_beam.runners,apache_beam.io.hadoopfilesystem_test,apache_beam.io.gcp.tests.utils_test,apache_beam.io.gcp.big_query_query_to_table_it_test,apache_beam.io.gcp.bigquery_io_read_it_test,apache_beam.io.gcp.bigquery_test,apache_beam.io.gcp.pubsub_integration_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.internal,apache_beam.io.filesystem_test,apache_beam.io.filesystems_test,apache_beam.io.sources_test,apache_beam.transforms,apache_beam.testing,apache_beam.io.filesystemio_test,apache_beam.io.localfilesystem_test,apache_beam.io.range_trackers_test,apache_beam.io.restriction_trackers_test,apache_beam.io.source_test_utils_test,apache_beam.io.concat_source_test,apache_beam.io.filebasedsink_test,apache_beam.io.filebasedsource_test,apache_beam.io.textio_test,apache_beam.io.tfrecordio_test,apache_beam.examples.wordcount_debugging_test,apache_beam.examples.wordcount_minimal_test,apache_beam.examples.wordcount_test,apache_beam.io.parquetio_test,apache_beam.io.gcp.gcsfilesystem_test,apache_beam.io.gcp.gcsio_integration_test,apache_beam.io.gcp.bigquery_test,apache_beam.io.gcp.big_query_query_to_table_it_test,apache_beam.io.gcp.bigquery_io_read_it_test,apache_beam.io.gcp.bigquery_tools_test,apache_beam.io.gcp.pubsub_integration_test,apache_beam.io.gcp.pubsub_test,apache_beam.io.gcp.datastore,apache_beam.io.gcp.datastore_write_it_test Review comment: Adding Datastore tests doesn't change anything since we skip them anyway, but it's a harmless change, so we can merge it. Let's try to simplify this to run all tests in a future PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198255) Time Spent: 19h 20m (was: 19h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Labels: triaged > Time Spent: 19h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=197684&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-197684 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 12/Feb/19 16:44 Start Date: 12/Feb/19 16:44 Worklog Time Spent: 10m Work Description: Juta commented on issue #7804: [BEAM-5315] python 3 datastore fail gracefully URL: https://github.com/apache/beam/pull/7804#issuecomment-462836304 R: @tvalentyn @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 197684) Time Spent: 19h 10m (was: 19h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Labels: triaged > Time Spent: 19h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=197642&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-197642 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 12/Feb/19 15:28 Start Date: 12/Feb/19 15:28 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7804: [BEAM-5315] python 3 datastore fail gracefully URL: https://github.com/apache/beam/pull/7804#issuecomment-462804038 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 197642) Time Spent: 19h (was: 18h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Labels: triaged > Time Spent: 19h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=196936&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-196936 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 11/Feb/19 14:25 Start Date: 11/Feb/19 14:25 Worklog Time Spent: 10m Work Description: Juta commented on pull request #7804: [BEAM-5315] python 3 datastore fail gracefully URL: https://github.com/apache/beam/pull/7804 This is is part of a series of PRs with goal to make Apache Beam PY3 compatible. The proposal with the outlined approach has been documented here: https://s.apache.org/beam-python-3. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 196936) Time Spent: 18h 50m (was: 18h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Labels: triaged > Time Spent: 18h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=194250&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194250 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 04/Feb/19 21:36 Start Date: 04/Feb/19 21:36 Worklog Time Spent: 10m Work Description: charlesccychen commented on pull request #7644: [BEAM-5315] Partially port Avroio module URL: https://github.com/apache/beam/pull/7644 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194250) Time Spent: 18h 40m (was: 18.5h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 18h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=194249&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194249 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 04/Feb/19 21:35 Start Date: 04/Feb/19 21:35 Worklog Time Spent: 10m Work Description: charlesccychen commented on issue #7644: [BEAM-5315] Partially port Avroio module URL: https://github.com/apache/beam/pull/7644#issuecomment-460422323 Thanks, this LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194249) Time Spent: 18.5h (was: 18h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 18.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=194050&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194050 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 04/Feb/19 15:26 Start Date: 04/Feb/19 15:26 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #7644: [BEAM-5315] Partially port Avroio module URL: https://github.com/apache/beam/pull/7644#issuecomment-460289903 LGTM. Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 194050) Time Spent: 18h 20m (was: 18h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 18h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=194048&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194048
]
ASF GitHub Bot logged work on BEAM-5315:
Author: ASF GitHub Bot
Created on: 04/Feb/19 15:25
Start Date: 04/Feb/19 15:25
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #7644: [BEAM-5315]
Partially port Avroio module
URL: https://github.com/apache/beam/pull/7644#discussion_r253513642
##
File path: sdks/python/tox.ini
##
@@ -57,17 +57,13 @@ commands =
setenv =
BEAM_EXPERIMENTAL_PY3=1
RUN_SKIPPED_PY3_TESTS=0
-modules =
-
apache_beam.typehints,apache_beam.coders,apache_beam.options,apache_beam.tools,apache_beam.utils,apache_beam.internal,apache_beam.metrics,apache_beam.portability,apache_beam.pipeline_test,apache_beam.pvalue_test,apache_beam.runners,apache_beam.io.hadoopfilesystem_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.tests.utils_test,apache_beam.io.gcp.big_query_query_to_table_it_test,apache_beam.io.gcp.bigquery_io_read_it_test,apache_beam.io.gcp.bigquery_test,apache_beam.io.gcp.gcsfilesystem_test,apache_beam.io.gcp.gcsio_test,apache_beam.io.gcp.pubsub_integration_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.internal,apache_beam.io.filesystem_test,apache_beam.io.filesystems_test,apache_beam.io.range_trackers_test,apache_beam.io.sources_test,apache_beam.transforms,apache_beam.testing,apache_beam.io.filesystemio_test,apache_beam.io.localfilesystem_test,apache_beam.io.range_trackers_test,apache_beam.io.restriction_trackers_test,apache_beam.io.source_test_utils_test,apache_beam.io.concat_source_test,apache_beam.io.filebasedsink_test,apache_beam.io.filebasedsource_test,apache_beam.io.textio_test,apache_beam.io.tfrecordio_test,apache_beam.examples.wordcount_debugging_test,apache_beam.examples.wordcount_minimal_test,apache_beam.examples.wordcount_test,apache_beam.io.parquetio_test,apache_beam.examples.cookbook.bigtableio_it_test
commands =
python --version
pip --version
{toxinidir}/scripts/run_tox_cleanup.sh
- python setup.py nosetests --tests {[testenv:py3]modules}
+ python apache_beam/examples/complete/autocomplete_test.py
Review comment:
As per
https://github.com/apache/beam/commit/245facdbc166298cdb96521165a7514653fa8e57
we try to execute one test outside of the test suite to make sure nothing
fails if we do so.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 194048)
Time Spent: 18h 10m (was: 18h)
> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Time Spent: 18h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=193996&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-193996 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 04/Feb/19 12:34 Start Date: 04/Feb/19 12:34 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7644: [BEAM-5315] Partially port Avroio module URL: https://github.com/apache/beam/pull/7644#issuecomment-460233478 PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 193996) Time Spent: 18h (was: 17h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 18h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=193948&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-193948
]
ASF GitHub Bot logged work on BEAM-5315:
Author: ASF GitHub Bot
Created on: 04/Feb/19 09:57
Start Date: 04/Feb/19 09:57
Worklog Time Spent: 10m
Work Description: RobbeSneyders commented on pull request #7644:
[BEAM-5315] Partially port Avroio module
URL: https://github.com/apache/beam/pull/7644#discussion_r253403062
##
File path: sdks/python/tox.ini
##
@@ -57,17 +57,13 @@ commands =
setenv =
BEAM_EXPERIMENTAL_PY3=1
RUN_SKIPPED_PY3_TESTS=0
-modules =
-
apache_beam.typehints,apache_beam.coders,apache_beam.options,apache_beam.tools,apache_beam.utils,apache_beam.internal,apache_beam.metrics,apache_beam.portability,apache_beam.pipeline_test,apache_beam.pvalue_test,apache_beam.runners,apache_beam.io.hadoopfilesystem_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.tests.utils_test,apache_beam.io.gcp.big_query_query_to_table_it_test,apache_beam.io.gcp.bigquery_io_read_it_test,apache_beam.io.gcp.bigquery_test,apache_beam.io.gcp.gcsfilesystem_test,apache_beam.io.gcp.gcsio_test,apache_beam.io.gcp.pubsub_integration_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.internal,apache_beam.io.filesystem_test,apache_beam.io.filesystems_test,apache_beam.io.range_trackers_test,apache_beam.io.sources_test,apache_beam.transforms,apache_beam.testing,apache_beam.io.filesystemio_test,apache_beam.io.localfilesystem_test,apache_beam.io.range_trackers_test,apache_beam.io.restriction_trackers_test,apache_beam.io.source_test_utils_test,apache_beam.io.concat_source_test,apache_beam.io.filebasedsink_test,apache_beam.io.filebasedsource_test,apache_beam.io.textio_test,apache_beam.io.tfrecordio_test,apache_beam.examples.wordcount_debugging_test,apache_beam.examples.wordcount_minimal_test,apache_beam.examples.wordcount_test,apache_beam.io.parquetio_test,apache_beam.examples.cookbook.bigtableio_it_test
commands =
python --version
pip --version
{toxinidir}/scripts/run_tox_cleanup.sh
- python setup.py nosetests --tests {[testenv:py3]modules}
+ python apache_beam/examples/complete/autocomplete_test.py
+ python setup.py nosetests --ignore-files vcfio_test\.py
Review comment:
Ok, sounds good.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 193948)
Time Spent: 17h 50m (was: 17h 40m)
> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Time Spent: 17h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=193943&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-193943
]
ASF GitHub Bot logged work on BEAM-5315:
Author: ASF GitHub Bot
Created on: 04/Feb/19 09:41
Start Date: 04/Feb/19 09:41
Worklog Time Spent: 10m
Work Description: RobbeSneyders commented on pull request #7644:
[BEAM-5315] Partially port Avroio module
URL: https://github.com/apache/beam/pull/7644#discussion_r253397482
##
File path: sdks/python/tox.ini
##
@@ -57,17 +57,13 @@ commands =
setenv =
BEAM_EXPERIMENTAL_PY3=1
RUN_SKIPPED_PY3_TESTS=0
-modules =
-
apache_beam.typehints,apache_beam.coders,apache_beam.options,apache_beam.tools,apache_beam.utils,apache_beam.internal,apache_beam.metrics,apache_beam.portability,apache_beam.pipeline_test,apache_beam.pvalue_test,apache_beam.runners,apache_beam.io.hadoopfilesystem_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.tests.utils_test,apache_beam.io.gcp.big_query_query_to_table_it_test,apache_beam.io.gcp.bigquery_io_read_it_test,apache_beam.io.gcp.bigquery_test,apache_beam.io.gcp.gcsfilesystem_test,apache_beam.io.gcp.gcsio_test,apache_beam.io.gcp.pubsub_integration_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.internal,apache_beam.io.filesystem_test,apache_beam.io.filesystems_test,apache_beam.io.range_trackers_test,apache_beam.io.sources_test,apache_beam.transforms,apache_beam.testing,apache_beam.io.filesystemio_test,apache_beam.io.localfilesystem_test,apache_beam.io.range_trackers_test,apache_beam.io.restriction_trackers_test,apache_beam.io.source_test_utils_test,apache_beam.io.concat_source_test,apache_beam.io.filebasedsink_test,apache_beam.io.filebasedsource_test,apache_beam.io.textio_test,apache_beam.io.tfrecordio_test,apache_beam.examples.wordcount_debugging_test,apache_beam.examples.wordcount_minimal_test,apache_beam.examples.wordcount_test,apache_beam.io.parquetio_test,apache_beam.examples.cookbook.bigtableio_it_test
commands =
python --version
pip --version
{toxinidir}/scripts/run_tox_cleanup.sh
- python setup.py nosetests --tests {[testenv:py3]modules}
+ python apache_beam/examples/complete/autocomplete_test.py
Review comment:
This test is executed in all other tox environments. I don't know why it is
added separately, so I also don't know why we shouldn't do it in Python 3.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 193943)
Time Spent: 17h 40m (was: 17.5h)
> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Time Spent: 17h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=193480&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-193480
]
ASF GitHub Bot logged work on BEAM-5315:
Author: ASF GitHub Bot
Created on: 01/Feb/19 18:27
Start Date: 01/Feb/19 18:27
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #7644: [BEAM-5315]
Partially port Avroio module
URL: https://github.com/apache/beam/pull/7644#discussion_r253151749
##
File path: sdks/python/tox.ini
##
@@ -57,17 +57,13 @@ commands =
setenv =
BEAM_EXPERIMENTAL_PY3=1
RUN_SKIPPED_PY3_TESTS=0
-modules =
-
apache_beam.typehints,apache_beam.coders,apache_beam.options,apache_beam.tools,apache_beam.utils,apache_beam.internal,apache_beam.metrics,apache_beam.portability,apache_beam.pipeline_test,apache_beam.pvalue_test,apache_beam.runners,apache_beam.io.hadoopfilesystem_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.tests.utils_test,apache_beam.io.gcp.big_query_query_to_table_it_test,apache_beam.io.gcp.bigquery_io_read_it_test,apache_beam.io.gcp.bigquery_test,apache_beam.io.gcp.gcsfilesystem_test,apache_beam.io.gcp.gcsio_test,apache_beam.io.gcp.pubsub_integration_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.internal,apache_beam.io.filesystem_test,apache_beam.io.filesystems_test,apache_beam.io.range_trackers_test,apache_beam.io.sources_test,apache_beam.transforms,apache_beam.testing,apache_beam.io.filesystemio_test,apache_beam.io.localfilesystem_test,apache_beam.io.range_trackers_test,apache_beam.io.restriction_trackers_test,apache_beam.io.source_test_utils_test,apache_beam.io.concat_source_test,apache_beam.io.filebasedsink_test,apache_beam.io.filebasedsource_test,apache_beam.io.textio_test,apache_beam.io.tfrecordio_test,apache_beam.examples.wordcount_debugging_test,apache_beam.examples.wordcount_minimal_test,apache_beam.examples.wordcount_test,apache_beam.io.parquetio_test,apache_beam.examples.cookbook.bigtableio_it_test
commands =
python --version
pip --version
{toxinidir}/scripts/run_tox_cleanup.sh
- python setup.py nosetests --tests {[testenv:py3]modules}
+ python apache_beam/examples/complete/autocomplete_test.py
Review comment:
Looks like this line shouldn't be here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 193480)
Time Spent: 17.5h (was: 17h 20m)
> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Time Spent: 17.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=193477&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-193477 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 01/Feb/19 18:17 Start Date: 01/Feb/19 18:17 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 193477) Time Spent: 17h (was: 16h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 17h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=193479&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-193479
]
ASF GitHub Bot logged work on BEAM-5315:
Author: ASF GitHub Bot
Created on: 01/Feb/19 18:26
Start Date: 01/Feb/19 18:26
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #7644: [BEAM-5315]
Partially port Avroio module
URL: https://github.com/apache/beam/pull/7644#discussion_r253151295
##
File path: sdks/python/tox.ini
##
@@ -57,17 +57,13 @@ commands =
setenv =
BEAM_EXPERIMENTAL_PY3=1
RUN_SKIPPED_PY3_TESTS=0
-modules =
-
apache_beam.typehints,apache_beam.coders,apache_beam.options,apache_beam.tools,apache_beam.utils,apache_beam.internal,apache_beam.metrics,apache_beam.portability,apache_beam.pipeline_test,apache_beam.pvalue_test,apache_beam.runners,apache_beam.io.hadoopfilesystem_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.tests.utils_test,apache_beam.io.gcp.big_query_query_to_table_it_test,apache_beam.io.gcp.bigquery_io_read_it_test,apache_beam.io.gcp.bigquery_test,apache_beam.io.gcp.gcsfilesystem_test,apache_beam.io.gcp.gcsio_test,apache_beam.io.gcp.pubsub_integration_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.internal,apache_beam.io.filesystem_test,apache_beam.io.filesystems_test,apache_beam.io.range_trackers_test,apache_beam.io.sources_test,apache_beam.transforms,apache_beam.testing,apache_beam.io.filesystemio_test,apache_beam.io.localfilesystem_test,apache_beam.io.range_trackers_test,apache_beam.io.restriction_trackers_test,apache_beam.io.source_test_utils_test,apache_beam.io.concat_source_test,apache_beam.io.filebasedsink_test,apache_beam.io.filebasedsource_test,apache_beam.io.textio_test,apache_beam.io.tfrecordio_test,apache_beam.examples.wordcount_debugging_test,apache_beam.examples.wordcount_minimal_test,apache_beam.examples.wordcount_test,apache_beam.io.parquetio_test,apache_beam.examples.cookbook.bigtableio_it_test
commands =
python --version
pip --version
{toxinidir}/scripts/run_tox_cleanup.sh
- python setup.py nosetests --tests {[testenv:py3]modules}
+ python apache_beam/examples/complete/autocomplete_test.py
+ python setup.py nosetests --ignore-files vcfio_test\.py
Review comment:
I mean let's add `@skip` statement to the test class in vcfio_test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 193479)
Time Spent: 17h 20m (was: 17h 10m)
> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Time Spent: 17h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=193478&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-193478
]
ASF GitHub Bot logged work on BEAM-5315:
Author: ASF GitHub Bot
Created on: 01/Feb/19 18:25
Start Date: 01/Feb/19 18:25
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #7644: [BEAM-5315]
Partially port Avroio module
URL: https://github.com/apache/beam/pull/7644#discussion_r253151087
##
File path: sdks/python/tox.ini
##
@@ -57,17 +57,13 @@ commands =
setenv =
BEAM_EXPERIMENTAL_PY3=1
RUN_SKIPPED_PY3_TESTS=0
-modules =
-
apache_beam.typehints,apache_beam.coders,apache_beam.options,apache_beam.tools,apache_beam.utils,apache_beam.internal,apache_beam.metrics,apache_beam.portability,apache_beam.pipeline_test,apache_beam.pvalue_test,apache_beam.runners,apache_beam.io.hadoopfilesystem_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.tests.utils_test,apache_beam.io.gcp.big_query_query_to_table_it_test,apache_beam.io.gcp.bigquery_io_read_it_test,apache_beam.io.gcp.bigquery_test,apache_beam.io.gcp.gcsfilesystem_test,apache_beam.io.gcp.gcsio_test,apache_beam.io.gcp.pubsub_integration_test,apache_beam.io.hdfs_integration_test,apache_beam.io.gcp.internal,apache_beam.io.filesystem_test,apache_beam.io.filesystems_test,apache_beam.io.range_trackers_test,apache_beam.io.sources_test,apache_beam.transforms,apache_beam.testing,apache_beam.io.filesystemio_test,apache_beam.io.localfilesystem_test,apache_beam.io.range_trackers_test,apache_beam.io.restriction_trackers_test,apache_beam.io.source_test_utils_test,apache_beam.io.concat_source_test,apache_beam.io.filebasedsink_test,apache_beam.io.filebasedsource_test,apache_beam.io.textio_test,apache_beam.io.tfrecordio_test,apache_beam.examples.wordcount_debugging_test,apache_beam.examples.wordcount_minimal_test,apache_beam.examples.wordcount_test,apache_beam.io.parquetio_test,apache_beam.examples.cookbook.bigtableio_it_test
commands =
python --version
pip --version
{toxinidir}/scripts/run_tox_cleanup.sh
- python setup.py nosetests --tests {[testenv:py3]modules}
+ python apache_beam/examples/complete/autocomplete_test.py
+ python setup.py nosetests --ignore-files vcfio_test\.py
Review comment:
How about we skip VCF IO tests in Python 3, to keep tox.ini cleaner?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 193478)
Time Spent: 17h 10m (was: 17h)
> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Time Spent: 17h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=193474&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-193474 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 01/Feb/19 18:11 Start Date: 01/Feb/19 18:11 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685#issuecomment-459815176 Thanks, @Juta ! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 193474) Time Spent: 16h 50m (was: 16h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 16h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=193315&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-193315 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 01/Feb/19 10:38 Start Date: 01/Feb/19 10:38 Worklog Time Spent: 10m Work Description: Juta commented on issue #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685#issuecomment-459644338 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 193315) Time Spent: 16h 40m (was: 16.5h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 16h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=193264&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-193264 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 01/Feb/19 08:26 Start Date: 01/Feb/19 08:26 Worklog Time Spent: 10m Work Description: Juta commented on issue #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685#issuecomment-459644338 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 193264) Time Spent: 16h 20m (was: 16h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 16h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=193265&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-193265 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 01/Feb/19 08:26 Start Date: 01/Feb/19 08:26 Worklog Time Spent: 10m Work Description: Juta commented on issue #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685#issuecomment-459387284 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 193265) Time Spent: 16.5h (was: 16h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 16.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=192909&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192909 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 31/Jan/19 16:07 Start Date: 31/Jan/19 16:07 Worklog Time Spent: 10m Work Description: Juta commented on issue #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685#issuecomment-459401337 build failed due to flaky gscio tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 192909) Time Spent: 16h 10m (was: 16h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 16h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=192897&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192897 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 31/Jan/19 15:31 Start Date: 31/Jan/19 15:31 Worklog Time Spent: 10m Work Description: Juta commented on issue #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685#issuecomment-459387284 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 192897) Time Spent: 16h (was: 15h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 16h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=192848&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192848 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 31/Jan/19 14:08 Start Date: 31/Jan/19 14:08 Worklog Time Spent: 10m Work Description: Juta commented on issue #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685#issuecomment-459357028 R: @tvalentyn @markflyhigh @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 192848) Time Spent: 15h 50m (was: 15h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 15h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=192805&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192805 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 31/Jan/19 13:18 Start Date: 31/Jan/19 13:18 Worklog Time Spent: 10m Work Description: Juta commented on issue #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685#issuecomment-459334820 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 192805) Time Spent: 15.5h (was: 15h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 15.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=192807&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192807 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 31/Jan/19 13:18 Start Date: 31/Jan/19 13:18 Worklog Time Spent: 10m Work Description: Juta commented on issue #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685#issuecomment-459340680 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 192807) Time Spent: 15h 40m (was: 15.5h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 15h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=192795&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192795 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 31/Jan/19 12:58 Start Date: 31/Jan/19 12:58 Worklog Time Spent: 10m Work Description: Juta commented on issue #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685#issuecomment-459334820 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 192795) Time Spent: 15h 20m (was: 15h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 15h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=192794&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192794 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 31/Jan/19 12:58 Start Date: 31/Jan/19 12:58 Worklog Time Spent: 10m Work Description: Juta commented on issue #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685#issuecomment-459333681 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 192794) Time Spent: 15h 10m (was: 15h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 15h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=192790&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192790 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 31/Jan/19 12:54 Start Date: 31/Jan/19 12:54 Worklog Time Spent: 10m Work Description: Juta commented on issue #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685#issuecomment-459333681 Run Python Precommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 192790) Time Spent: 14h 50m (was: 14h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 14h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=192791&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192791 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 31/Jan/19 12:55 Start Date: 31/Jan/19 12:55 Worklog Time Spent: 10m Work Description: Juta commented on issue #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685#issuecomment-459333681 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 192791) Time Spent: 15h (was: 14h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 15h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=192756&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192756 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 31/Jan/19 11:15 Start Date: 31/Jan/19 11:15 Worklog Time Spent: 10m Work Description: Juta commented on pull request #7685: [BEAM-5315] Python 3 port bigquery and pubsub related io modules URL: https://github.com/apache/beam/pull/7685 This is is part of a series of PRs with goal to make Apache Beam PY3 compatible. The proposal with the outlined approach has been documented here: https://s.apache.org/beam-python-3. This PR adds a py3-gcp test suite to tox and ports the gcs related io modules. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 192756) Time Spent: 14h 40m (was: 14.5h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 14h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassia
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=192631&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192631 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 31/Jan/19 02:47 Start Date: 31/Jan/19 02:47 Worklog Time Spent: 10m Work Description: aaltay commented on issue #7644: [BEAM-5315] Partially port Avroio module URL: https://github.com/apache/beam/pull/7644#issuecomment-459194353 hey @RobbeSneyders could you resolve the conflict please? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 192631) Time Spent: 14.5h (was: 14h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 14.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=192038&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192038 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 30/Jan/19 05:31 Start Date: 30/Jan/19 05:31 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #7644: [BEAM-5315] Partially port Avroio module URL: https://github.com/apache/beam/pull/7644#issuecomment-458817326 Thanks @RobbeSneyders . @aaltay or @charlesccychen - please help with the merge. Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 192038) Time Spent: 14h 20m (was: 14h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 14h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=192037&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-192037 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 30/Jan/19 05:30 Start Date: 30/Jan/19 05:30 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #7644: [BEAM-5315] Partially port Avroio module URL: https://github.com/apache/beam/pull/7644#issuecomment-458817326 @aaltay or @charlesccychen for the merge. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 192037) Time Spent: 14h 10m (was: 14h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 14h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=191988&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191988 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 30/Jan/19 02:14 Start Date: 30/Jan/19 02:14 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #7644: [BEAM-5315] Partially port Avroio module URL: https://github.com/apache/beam/pull/7644#issuecomment-458784688 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 191988) Time Spent: 14h (was: 13h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 14h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=191967&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191967 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 30/Jan/19 01:26 Start Date: 30/Jan/19 01:26 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #7644: [BEAM-5315] Partially port Avroio module URL: https://github.com/apache/beam/pull/7644#issuecomment-458774895 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 191967) Time Spent: 13h 50m (was: 13h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 13h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=191900&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-191900 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 29/Jan/19 21:20 Start Date: 29/Jan/19 21:20 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7644: [BEAM-5315] Partially port Avroio module URL: https://github.com/apache/beam/pull/7644#issuecomment-458711449 Tests currently fail due to `tox.ini` changes. When #7583 gets merged, this should not be an issue anymore. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 191900) Time Spent: 13h 40m (was: 13.5h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 13h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=190966&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-190966 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 28/Jan/19 13:28 Start Date: 28/Jan/19 13:28 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on pull request #7644: [BEAM-5315] Partially port Avroio module URL: https://github.com/apache/beam/pull/7644 This is is part of a series of PRs with goal to make Apache Beam PY3 compatible. The proposal with the outlined approach has been documented here: https://s.apache.org/beam-python-3. This PR partially ports the avroio module. 4 tests (2x the same test, for Avro and FastAvro) are still failing. I created [BEAM-6522](https://issues.apache.org/jira/browse/BEAM-6522) to track this. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | --- | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 190966) Time Spent: 13.5h (was: 13h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 13.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=189125&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-189125 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 23/Jan/19 18:38 Start Date: 23/Jan/19 18:38 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #7582: [BEAM-5315] Python 3 port io.parquetio module URL: https://github.com/apache/beam/pull/7582 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 189125) Time Spent: 13h 20m (was: 13h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 13h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=187840&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187840 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 21/Jan/19 17:03 Start Date: 21/Jan/19 17:03 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7582: [BEAM-5315] Python 3 port io.parquetio module URL: https://github.com/apache/beam/pull/7582#issuecomment-456141469 R: @tvalentyn @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187840) Time Spent: 13h 10m (was: 13h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 13h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=187763&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187763 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 21/Jan/19 12:44 Start Date: 21/Jan/19 12:44 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on pull request #7582: [BEAM-5315] Python 3 port io.parquetio module URL: https://github.com/apache/beam/pull/7582 This is is part of a series of PRs with goal to make Apache Beam PY3 compatible. The proposal with the outlined approach has been documented here: https://s.apache.org/beam-python-3. This PR ports the `io.parquetio` module. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187763) Time Spent: 13h (was: 12h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 13h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=185496&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185496 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 15/Jan/19 23:36 Start Date: 15/Jan/19 23:36 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #7503: [BEAM-5315] Python 3 port io.tfrecordio module URL: https://github.com/apache/beam/pull/7503 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185496) Time Spent: 12h 50m (was: 12h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 12h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=185345&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185345 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 15/Jan/19 17:02 Start Date: 15/Jan/19 17:02 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7503: [BEAM-5315] Python 3 port io.tfrecordio module URL: https://github.com/apache/beam/pull/7503#issuecomment-454469249 R: @tvalentyn @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185345) Time Spent: 12h 40m (was: 12.5h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 12h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=185000&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185000 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 14/Jan/19 22:50 Start Date: 14/Jan/19 22:50 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on pull request #7503: [BEAM-5315] Python 3 port io.tfrecordio module URL: https://github.com/apache/beam/pull/7503 This is is part of a series of PRs with goal to make Apache Beam PY3 compatible. The proposal with the outlined approach has been documented here: https://s.apache.org/beam-python-3. This PR ports the `io.tfrecordio` module. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185000) Time Spent: 12.5h (was: 12h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 12.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=182789&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-182789 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 09/Jan/19 01:19 Start Date: 09/Jan/19 01:19 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #7394: [BEAM-5315] Python 3 port io.textio module URL: https://github.com/apache/beam/pull/7394 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 182789) Time Spent: 12h 20m (was: 12h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 12h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=182017&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-182017 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 07/Jan/19 20:02 Start Date: 07/Jan/19 20:02 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on pull request #7394: [BEAM-5315] Python 3 port io.textio module URL: https://github.com/apache/beam/pull/7394#discussion_r245780402 ## File path: sdks/python/apache_beam/io/filebasedsource_test.py ## @@ -87,6 +87,22 @@ class EOL(object): def write_data( num_lines, no_data=False, directory=None, prefix=tempfile.template, eol=EOL.LF): + """Writes test data to a temporary file. + + Args: +num_lines (int): The number of lines to write. +no_data (bool): If :data:`True`, empty lines will be written, otherwise + each line will contain a concatenation of b'line' and the line number. +directory (str): The name of the directory to create the temporary file in. +prefix (str): The prefix to use for the temporary file. +eol (int): The line ending to use when writing. + :class:`~apache_beam.io.textio_test.EOL` exposes attributes that can be + used here to define the eol. + + Returns: +Tuple[str, List[str]): A tuple of the filename and a list of the written Review comment: You're right. Fixed it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 182017) Time Spent: 12h 10m (was: 12h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 12h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=181973&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-181973 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 07/Jan/19 19:18 Start Date: 07/Jan/19 19:18 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #7394: [BEAM-5315] Python 3 port io.textio module URL: https://github.com/apache/beam/pull/7394#discussion_r245767467 ## File path: sdks/python/apache_beam/io/filebasedsource_test.py ## @@ -87,6 +87,22 @@ class EOL(object): def write_data( num_lines, no_data=False, directory=None, prefix=tempfile.template, eol=EOL.LF): + """Writes test data to a temporary file. + + Args: +num_lines (int): The number of lines to write. +no_data (bool): If :data:`True`, empty lines will be written, otherwise + each line will contain a concatenation of b'line' and the line number. +directory (str): The name of the directory to create the temporary file in. +prefix (str): The prefix to use for the temporary file. +eol (int): The line ending to use when writing. + :class:`~apache_beam.io.textio_test.EOL` exposes attributes that can be + used here to define the eol. + + Returns: +Tuple[str, List[str]): A tuple of the filename and a list of the written Review comment: Thanks. I wish we had more docstrings like this one throughout SDK internals. Tuple[str, List[bytes]] in Line 103? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 181973) Time Spent: 12h (was: 11h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 12h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=181691&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-181691 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 07/Jan/19 11:05 Start Date: 07/Jan/19 11:05 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on pull request #7394: [BEAM-5315] Python 3 port io.textio module URL: https://github.com/apache/beam/pull/7394#discussion_r245619804 ## File path: sdks/python/apache_beam/io/textio_test.py ## @@ -53,6 +53,16 @@ from apache_beam.transforms.core import Create +def write_data(*args, **kwargs): Review comment: Good point. I copied the implementation and added documentation in both files. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 181691) Time Spent: 11h 50m (was: 11h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 11h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180851&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180851 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 03/Jan/19 19:56 Start Date: 03/Jan/19 19:56 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #7394: [BEAM-5315] Python 3 port io.textio module URL: https://github.com/apache/beam/pull/7394#discussion_r245114813 ## File path: sdks/python/apache_beam/io/textio_test.py ## @@ -53,6 +53,16 @@ from apache_beam.transforms.core import Create +def write_data(*args, **kwargs): Review comment: It is hard to reason about this helper in the context of the test scenarios. Let's copy the existing implementation of write_data into this test suite, modify it to return strings as second tuple argument, and add a docstring explaining the arguments and the type of result. This would make it easier for these helpers to evolve separately. For example, for the purposes of textio_test, I think it makes sense to allow customizing payload instead of default 'line', and adding more interesting testcases of unicode strings in the future. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180851) Time Spent: 11h 40m (was: 11.5h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 11h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180850&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180850 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 03/Jan/19 19:54 Start Date: 03/Jan/19 19:54 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #7394: [BEAM-5315] Python 3 port io.textio module URL: https://github.com/apache/beam/pull/7394#discussion_r245114813 ## File path: sdks/python/apache_beam/io/textio_test.py ## @@ -53,6 +53,16 @@ from apache_beam.transforms.core import Create +def write_data(*args, **kwargs): Review comment: It is hard to reason about this helper in the context of the test scenarios. Let's copy the existing implementation of write_data into this test suite, modify it to return strings as second tuple argument, and add a docstring explaining the arguments and the type of result. This would make it easier for these helpers evolve separately. For example, for the purposes of textio_test, I think it makes sense to allow customizing payload instead of default 'line', and adding more interesting testcases of unicode strings in the future. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180850) Time Spent: 11.5h (was: 11h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 11.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180635&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180635 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 03/Jan/19 11:53 Start Date: 03/Jan/19 11:53 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7394: [BEAM-5315] Python 3 port io.textio module URL: https://github.com/apache/beam/pull/7394#issuecomment-451122870 PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180635) Time Spent: 11h 20m (was: 11h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 11h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180473&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180473 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 03/Jan/19 00:52 Start Date: 03/Jan/19 00:52 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7394: [BEAM-5315] Python 3 port io.textio module URL: https://github.com/apache/beam/pull/7394#issuecomment-451030577 R: @tvalentyn @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180473) Time Spent: 11h 10m (was: 11h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 11h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180472&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180472 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 03/Jan/19 00:51 Start Date: 03/Jan/19 00:51 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on pull request #7394: [BEAM-5315] Python 3 port io.textio module URL: https://github.com/apache/beam/pull/7394 This is is part of a series of PRs with goal to make Apache Beam PY3 compatible. The proposal with the outlined approach has been documented here: https://s.apache.org/beam-python-3. Since the io package is quite big, with a large amount of open issues, I've decided to split up the porting of this package into multiple PRs. This PR ports the `io.textio.py` module. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180472) Time Spent: 11h (was: 10h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 11h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180433&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180433 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 02/Jan/19 22:19 Start Date: 02/Jan/19 22:19 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #7386: [BEAM-5315] Python 3 port io.filebased_* modules URL: https://github.com/apache/beam/pull/7386 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180433) Time Spent: 10h 50m (was: 10h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 10h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180384&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180384 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 02/Jan/19 19:45 Start Date: 02/Jan/19 19:45 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #7383: [BEAM-5315] Python 3 port io.source* and io.concat_source* modules URL: https://github.com/apache/beam/pull/7383 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180384) Time Spent: 10h 40m (was: 10.5h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 10h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180355&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180355 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 02/Jan/19 18:33 Start Date: 02/Jan/19 18:33 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #7358: [BEAM-5315] Python 3 port io.*_trackers modules URL: https://github.com/apache/beam/pull/7358 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180355) Time Spent: 10.5h (was: 10h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 10.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180223&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180223 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 02/Jan/19 16:20 Start Date: 02/Jan/19 16:20 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7386: [BEAM-5315] Python 3 port io.filebased_* modules URL: https://github.com/apache/beam/pull/7386#issuecomment-450908319 R: @tvalentyn @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180223) Time Spent: 10h 20m (was: 10h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 10h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180197&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180197 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 02/Jan/19 15:34 Start Date: 02/Jan/19 15:34 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7386: [BEAM-5315] Python 3 port io.filebased_* modules URL: https://github.com/apache/beam/pull/7386#issuecomment-450894688 This PR is rebased on top of #7358 since it is dependent on those changes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180197) Time Spent: 10h 10m (was: 10h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 10h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180196&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180196 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 02/Jan/19 15:34 Start Date: 02/Jan/19 15:34 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on pull request #7386: [BEAM-5315] Python 3 port io.filebased_* modules URL: https://github.com/apache/beam/pull/7386 This is is part of a series of PRs with goal to make Apache Beam PY3 compatible. The proposal with the outlined approach has been documented here: https://s.apache.org/beam-python-3. Since the io package is quite big, with a large amount of open issues, I've decided to split up the porting of this package into multiple PRs. This PR ports the `io.filebasedsource.py` module and adds the already passing `io.filebasedsink_test.py `module to the Python 3 test suite. Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180196) Time Spent: 10h (was: 9h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 10h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180186&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180186 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 02/Jan/19 15:01 Start Date: 02/Jan/19 15:01 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7383: [BEAM-5315] Python 3 port io.source* and io.concat_source* modules URL: https://github.com/apache/beam/pull/7383#issuecomment-450885418 R: @tvalentyn @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180186) Time Spent: 9h 50m (was: 9h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 9h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180105&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180105 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 02/Jan/19 10:40 Start Date: 02/Jan/19 10:40 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on pull request #7383: [BEAM-5315] Python 3 port io.source* and io.concat_source* modules URL: https://github.com/apache/beam/pull/7383 This is is part of a series of PRs with goal to make Apache Beam PY3 compatible. The proposal with the outlined approach has been documented here: https://s.apache.org/beam-python-3. Since the io package is quite big, with a large amount of open issues, I've decided to split up the porting of this package into multiple PRs. This PR ports the `io.source_test_utils.py` and `io.sources_test` modules and adds the already passing `io.concat_source_test.py `module to the Python 3 test suite. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180105) Time Spent: 9.5h (was: 9h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 9.5h > Remaining Estimate: 0h > -- This messag
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180106&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180106 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 02/Jan/19 10:41 Start Date: 02/Jan/19 10:41 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7383: [BEAM-5315] Python 3 port io.source* and io.concat_source* modules URL: https://github.com/apache/beam/pull/7383#issuecomment-450832405 This PR is rebased on top of #7358 since it is dependent on those changes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180106) Time Spent: 9h 40m (was: 9.5h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 9h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=180103&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180103 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 02/Jan/19 10:40 Start Date: 02/Jan/19 10:40 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7358: [BEAM-5315] Python 3 port io.*_trackers modules URL: https://github.com/apache/beam/pull/7358#issuecomment-450832094 I've updated the docstring. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180103) Time Spent: 9h 20m (was: 9h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 9h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=179269&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179269 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 27/Dec/18 22:44 Start Date: 27/Dec/18 22:44 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #7358: [BEAM-5315] Python 3 port io.*_trackers modules URL: https://github.com/apache/beam/pull/7358#discussion_r244236498 ## File path: sdks/python/apache_beam/io/range_trackers.py ## @@ -390,13 +390,13 @@ def _string_to_int(s, prec): if not s: return 0 elif len(s) < prec: - s += '\0' * (prec - len(s)) + s += b'\0' * (prec - len(s)) else: s = s[:prec] return int(codecs.encode(s, 'hex'), 16) @staticmethod - def _string_from_int(i, prec): + def _bytestring_from_int(i, prec): Review comment: nit: s/string/bytestring/ on line 401. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 179269) Time Spent: 9h 10m (was: 9h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 9h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=179217&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179217 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 27/Dec/18 20:15 Start Date: 27/Dec/18 20:15 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7358: [BEAM-5315] Python 3 port io.*_trackers modules URL: https://github.com/apache/beam/pull/7358#issuecomment-450223237 R: @tvalentyn @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 179217) Time Spent: 9h (was: 8h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 9h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=178796&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178796 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 26/Dec/18 22:12 Start Date: 26/Dec/18 22:12 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #7326: [BEAM-5315] [BEAM-5627] Python 3 port io.filesystemio module URL: https://github.com/apache/beam/pull/7326 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 178796) Time Spent: 8h 50m (was: 8h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 8h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=178769&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-178769 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 26/Dec/18 20:37 Start Date: 26/Dec/18 20:37 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #7326: [BEAM-5315] [BEAM-5627] Python 3 port io.filesystemio module URL: https://github.com/apache/beam/pull/7326#issuecomment-450022693 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 178769) Time Spent: 8h 40m (was: 8.5h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 8h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=177387&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177387 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 20/Dec/18 09:14 Start Date: 20/Dec/18 09:14 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7326: [BEAM-5315] [BEAM-5627] Python 3 port io.filesystemio module URL: https://github.com/apache/beam/pull/7326#issuecomment-448927200 Run RAT PreCommit Run Python PreCommit Run Portable_Python PreCommit Manually starting tests since the automatic trigger does not seem to work anymore after my last push. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177387) Time Spent: 8.5h (was: 8h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=177375&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177375 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 20/Dec/18 08:48 Start Date: 20/Dec/18 08:48 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7326: [BEAM-5315] [BEAM-5627] Python 3 port io.filesystemio module URL: https://github.com/apache/beam/pull/7326#issuecomment-448919839 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177375) Time Spent: 8h 20m (was: 8h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 8h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=177213&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177213 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 19/Dec/18 22:03 Start Date: 19/Dec/18 22:03 Worklog Time Spent: 10m Work Description: aaltay commented on a change in pull request #7318: [BEAM-5315] [BEAM-5627] [BEAM-5623] Python 3 port io.filesystem module URL: https://github.com/apache/beam/pull/7318#discussion_r243091352 ## File path: sdks/python/apache_beam/io/filesystem.py ## @@ -209,7 +209,7 @@ def _fetch_to_internal_buffer(self, num_bytes): # objects for the unused compressed data. if (self._compression_type == CompressionTypes.BZIP2 or self._compression_type == CompressionTypes.GZIP): - if self._decompressor.unused_data != '': + if self._decompressor.unused_data != b'': Review comment: Thanks for catching! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177213) Time Spent: 7h 40m (was: 7.5h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 7h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=177221&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177221 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 19/Dec/18 22:12 Start Date: 19/Dec/18 22:12 Worklog Time Spent: 10m Work Description: RobbeSneyders opened a new pull request #7326: [BEAM-5315] [BEAM-5627] Python 3 port io.filesystemio module URL: https://github.com/apache/beam/pull/7326 This is is part of a series of PRs with goal to make Apache Beam PY3 compatible. The proposal with the outlined approach has been documented here: https://s.apache.org/beam-python-3. Since the io package is quite big, with a large amount of open issues, I've decided to split up the porting of this package into multiple PRs. This PR ports the `io.filesystemio` module and adds the already passing `io.localfilesystem_test` module to the Python 3 test suite. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177221) Time Spent: 8h (was: 7h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 8h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=177222&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177222 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 19/Dec/18 22:12 Start Date: 19/Dec/18 22:12 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7326: [BEAM-5315] [BEAM-5627] Python 3 port io.filesystemio module URL: https://github.com/apache/beam/pull/7326#issuecomment-448762551 R: @tvalentyn @aaltay @markflyhigh This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177222) Time Spent: 8h 10m (was: 8h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 8h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=177214&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177214
]
ASF GitHub Bot logged work on BEAM-5315:
Author: ASF GitHub Bot
Created on: 19/Dec/18 22:03
Start Date: 19/Dec/18 22:03
Worklog Time Spent: 10m
Work Description: aaltay closed pull request #7318: [BEAM-5315]
[BEAM-5627] [BEAM-5623] Python 3 port io.filesystem module
URL: https://github.com/apache/beam/pull/7318
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/io/filesystem.py
b/sdks/python/apache_beam/io/filesystem.py
index 083d365ee700..7dd3a5b3419c 100644
--- a/sdks/python/apache_beam/io/filesystem.py
+++ b/sdks/python/apache_beam/io/filesystem.py
@@ -209,7 +209,7 @@ def _fetch_to_internal_buffer(self, num_bytes):
# objects for the unused compressed data.
if (self._compression_type == CompressionTypes.BZIP2 or
self._compression_type == CompressionTypes.GZIP):
- if self._decompressor.unused_data != '':
+ if self._decompressor.unused_data != b'':
buf = self._decompressor.unused_data
self._decompressor = (
bz2.BZ2Decompressor()
@@ -259,7 +259,7 @@ def readline(self):
line = self._read_from_internal_buffer(
lambda: self._read_buffer.readline())
bytes_io.write(line)
- if line.endswith('\n') or not line:
+ if line.endswith(b'\n') or not line:
break # Newline or EOF reached.
return bytes_io.getvalue()
diff --git a/sdks/python/apache_beam/io/filesystem_test.py
b/sdks/python/apache_beam/io/filesystem_test.py
index c39da9a837e7..abbadde97233 100644
--- a/sdks/python/apache_beam/io/filesystem_test.py
+++ b/sdks/python/apache_beam/io/filesystem_test.py
@@ -265,7 +265,7 @@ class TestCompressedFile(unittest.TestCase):
which will be deleted at the end of the tests (when tearDown() is called).
"""
- content = """- the BEAM -
+ content = b"""- the BEAM -
How things really are we would like to know.
Does
Time
@@ -291,10 +291,6 @@ def _create_temp_file(self):
self._tempfiles.append(path)
return path
- @unittest.skipIf(sys.version_info[0] == 3 and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3'
- 'TODO: BEAM-5627')
def _create_compressed_file(self, compression_type, content):
file_name = self._create_temp_file()
@@ -395,7 +391,7 @@ def test_read_from_end_returns_no_data(self):
seek_position = 0
compressed_fd.seek(seek_position, os.SEEK_END)
-expected_data = ''
+expected_data = b''
uncompressed_data = compressed_fd.read(10)
self.assertEqual(uncompressed_data, expected_data)
@@ -435,14 +431,10 @@ def test_read_and_seek_back_to_beginning(self):
self.assertEqual(first_pass, second_pass)
- @unittest.skipIf(sys.version_info[0] == 3 and
- os.environ.get('RUN_SKIPPED_PY3_TESTS') != '1',
- 'This test still needs to be fixed on Python 3'
- 'TODO: BEAM-5627')
def test_tell(self):
-lines = ['line%d\n' % i for i in range(10)]
+lines = [b'line%d\n' % i for i in range(10)]
tmpfile = self._create_temp_file()
-with open(tmpfile, 'w') as f:
+with open(tmpfile, 'wb') as f:
writeable = CompressedFile(f)
current_offset = 0
for line in lines:
@@ -450,7 +442,7 @@ def test_tell(self):
current_offset += len(line)
self.assertEqual(current_offset, writeable.tell())
-with open(tmpfile) as f:
+with open(tmpfile, 'rb') as f:
readable = CompressedFile(f)
current_offset = 0
while True:
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 177214)
Time Spent: 7h 50m (was: 7h 40m)
> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Time Spent: 7h 50m
> Remaining Estimate: 0h
>
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=177048&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177048 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 19/Dec/18 15:48 Start Date: 19/Dec/18 15:48 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on a change in pull request #7318: [BEAM-5315] [BEAM-5627] [BEAM-5623] Python 3 port io.filesystem module URL: https://github.com/apache/beam/pull/7318#discussion_r242966689 ## File path: sdks/python/apache_beam/io/filesystem.py ## @@ -209,7 +209,7 @@ def _fetch_to_internal_buffer(self, num_bytes): # objects for the unused compressed data. if (self._compression_type == CompressionTypes.BZIP2 or self._compression_type == CompressionTypes.GZIP): - if self._decompressor.unused_data != '': + if self._decompressor.unused_data != b'': Review comment: This change fixes the hanging tests. This code block resulted in an infinite loop when `self._decompressor.unused_data = b''`. On Python 2, this was not an issue since `b'' == ''` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177048) Time Spent: 7h 20m (was: 7h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 7h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=177049&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177049 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 19/Dec/18 15:48 Start Date: 19/Dec/18 15:48 Worklog Time Spent: 10m Work Description: RobbeSneyders commented on issue #7318: [BEAM-5315] [BEAM-5627] [BEAM-5623] Python 3 port io.filesystem module URL: https://github.com/apache/beam/pull/7318#issuecomment-448642727 R: @tvalentyn @aaltay @markflyhigh This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177049) Time Spent: 7.5h (was: 7h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=177045&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-177045 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 19/Dec/18 15:45 Start Date: 19/Dec/18 15:45 Worklog Time Spent: 10m Work Description: RobbeSneyders opened a new pull request #7318: [BEAM-5315] [BEAM-5627] [BEAM-5623] Python 3 port io.filesystem module URL: https://github.com/apache/beam/pull/7318 This is is part of a series of PRs with goal to make Apache Beam PY3 compatible. The proposal with the outlined approach has been documented here: https://s.apache.org/beam-python-3. Since the io package is quite big, with a large amount of open issues, I've decided to split up the porting of this package into multiple PRs. This PR ports the io.filesystem module and fixes the hanging tests related to gzip (BEAM-5623), which were blocking further porting of the io package. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 177045) Time Spent: 7h 10m (was: 7h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Robbe >Priority: Major > Time Spent: 7h 10m > Remaining Estimate: 0h > -- This message was sent b
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=168067&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-168067
]
ASF GitHub Bot logged work on BEAM-5315:
Author: ASF GitHub Bot
Created on: 21/Nov/18 00:32
Start Date: 21/Nov/18 00:32
Worklog Time Spent: 10m
Work Description: aaltay closed pull request #6925: [BEAM-5315] Partially
port IO: avro schema parsing and codecs
URL: https://github.com/apache/beam/pull/6925
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/io/avroio.py
b/sdks/python/apache_beam/io/avroio.py
index 79c2505df4a..b648ad205ba 100644
--- a/sdks/python/apache_beam/io/avroio.py
+++ b/sdks/python/apache_beam/io/avroio.py
@@ -330,14 +330,14 @@ def offset(self):
@staticmethod
def _decompress_bytes(data, codec):
-if codec == 'null':
+if codec == b'null':
return data
-elif codec == 'deflate':
+elif codec == b'deflate':
# zlib.MAX_WBITS is the window size. '-' sign indicates that this is
# raw data (without headers). See zlib and Avro documentations for more
# details.
return zlib.decompress(data, -zlib.MAX_WBITS)
-elif codec == 'snappy':
+elif codec == b'snappy':
# Snappy is an optional avro codec.
# See Snappy and Avro documentation for more details.
try:
@@ -360,8 +360,10 @@ def num_records(self):
def records(self):
decoder = avroio.BinaryDecoder(
io.BytesIO(self._decompressed_block_bytes))
-reader = avroio.DatumReader(
-writers_schema=self._schema, readers_schema=self._schema)
+
+writer_schema = self._schema
+reader_schema = self._schema
+reader = avroio.DatumReader(writer_schema, reader_schema)
current_record = 0
while current_record < self._num_records:
diff --git a/sdks/python/apache_beam/io/filesystemio_test.py
b/sdks/python/apache_beam/io/filesystemio_test.py
index f11e1c45623..0a038812b02 100644
--- a/sdks/python/apache_beam/io/filesystemio_test.py
+++ b/sdks/python/apache_beam/io/filesystemio_test.py
@@ -177,7 +177,7 @@ def _read_and_verify(self, stream, expected, buffer_size):
data_list.append(data)
bytes_read += len(data)
self.assertEqual(stream.tell(), bytes_read)
-self.assertEqual(''.join(data_list), expected)
+self.assertEqual(b''.join(data_list), expected)
def test_pipe_stream(self):
block_sizes = list(4**i for i in range(0, 12))
diff --git a/sdks/python/apache_beam/io/tfrecordio_test.py
b/sdks/python/apache_beam/io/tfrecordio_test.py
index 6421fa5c591..07480b22519 100644
--- a/sdks/python/apache_beam/io/tfrecordio_test.py
+++ b/sdks/python/apache_beam/io/tfrecordio_test.py
@@ -405,6 +405,9 @@ def test_process_auto(self):
assert_that(result, equal_to(['foo', 'bar']))
+@unittest.skipIf(sys.version_info[0] == 3,
+ 'This test still needs to be fixed on Python 3'
+ 'TODO: BEAM-5623 - several IO tests hang indefinitely')
class TestEnd2EndWriteAndRead(unittest.TestCase):
def create_inputs(self):
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 168067)
Time Spent: 7h (was: 6h 50m)
> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Simon
>Priority: Major
> Time Spent: 7h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=167930&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-167930 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 20/Nov/18 19:22 Start Date: 20/Nov/18 19:22 Worklog Time Spent: 10m Work Description: tvalentyn commented on a change in pull request #6925: [BEAM-5315] Partially port IO: avro schema parsing and codecs URL: https://github.com/apache/beam/pull/6925#discussion_r235137307 ## File path: sdks/python/apache_beam/io/tfrecordio_test.py ## @@ -405,6 +405,9 @@ def test_process_auto(self): assert_that(result, equal_to(['foo', 'bar'])) +@unittest.skipIf(sys.version_info[0] == 3, Review comment: The more tests we run on Jenkins in Python 3, the more Py3 regressions we can capture. If 1 out of 10 tests in the package is passing, it may be easier to enlist one test in tox.ini, than add all tests in the package and skip 9 that are not passing. If 9 out of 10 tests are passing, we can file a JIRA for a remaining tests, if it's failing due to a self-contained failure and skip it, and add the package. If we think that the whole package will be fixed in ~a week, it may not be worth the churn to add skipping instructions and add tests one-by-one. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 167930) Time Spent: 6h 50m (was: 6h 40m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Simon >Priority: Major > Time Spent: 6h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=167773&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-167773 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 20/Nov/18 15:08 Start Date: 20/Nov/18 15:08 Worklog Time Spent: 10m Work Description: splovyt commented on a change in pull request #6925: [BEAM-5315] Partially port IO: avro schema parsing and codecs URL: https://github.com/apache/beam/pull/6925#discussion_r235039855 ## File path: sdks/python/apache_beam/io/tfrecordio_test.py ## @@ -405,6 +405,9 @@ def test_process_auto(self): assert_that(result, equal_to(['foo', 'bar'])) +@unittest.skipIf(sys.version_info[0] == 3, Review comment: This test hangs when executed on the master branch both in my local environment as well as in an environment I set up on a VM. @RobbeSneyders verified this and this is also the case in his environments. The reason all IO changes pass the jenkins checks is because in tox.ini none of the IO tests are being added and are therefore are not checked. They are of course presumably being tested locally, so it rarely results in an issue, but if the local check is done before a rebase this could conflict with another change imo, although in an indirect manner because otherwise there would be a merge conflict. Do you think we should add the altered io tests to the tox file from now on and go back to skipping remaining errors in order to pass the jenkins environment? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 167773) Time Spent: 6.5h (was: 6h 20m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Simon >Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=167774&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-167774 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 20/Nov/18 15:09 Start Date: 20/Nov/18 15:09 Worklog Time Spent: 10m Work Description: splovyt commented on a change in pull request #6925: [BEAM-5315] Partially port IO: avro schema parsing and codecs URL: https://github.com/apache/beam/pull/6925#discussion_r235039855 ## File path: sdks/python/apache_beam/io/tfrecordio_test.py ## @@ -405,6 +405,9 @@ def test_process_auto(self): assert_that(result, equal_to(['foo', 'bar'])) +@unittest.skipIf(sys.version_info[0] == 3, Review comment: This test hangs when executed on the master branch both in my local environment as well as in an environment I set up on a VM. @RobbeSneyders verified this and this is also the case in his environments. The reason all IO changes pass the jenkins checks is because in tox.ini none of the IO tests are being added and are therefore not checked. They are of course presumably being tested locally, so it rarely results in an issue, but if the local check is done before a rebase this could conflict with another change imo, although in an indirect manner because otherwise there would be a merge conflict. Do you think we should add the altered io tests to the tox file from now on and go back to skipping remaining errors in order to pass the jenkins environment? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 167774) Time Spent: 6h 40m (was: 6.5h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Simon >Priority: Major > Time Spent: 6h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=167768&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-167768 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 20/Nov/18 14:49 Start Date: 20/Nov/18 14:49 Worklog Time Spent: 10m Work Description: splovyt commented on a change in pull request #6925: [BEAM-5315] Partially port IO: avro schema parsing and codecs URL: https://github.com/apache/beam/pull/6925#discussion_r235031351 ## File path: sdks/python/apache_beam/io/avroio.py ## @@ -360,8 +360,10 @@ def num_records(self): def records(self): decoder = avroio.BinaryDecoder( io.BytesIO(self._decompressed_block_bytes)) -reader = avroio.DatumReader( -writers_schema=self._schema, readers_schema=self._schema) +# Kwargs writers_schema and readers_schema were renamed to +# writer_schema and reader_schema in py3. +# Therefore, the kwargs were left out to be both py2 and py3 compatible. Review comment: Ah yes, I was indeed looking for something like this! Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 167768) Time Spent: 6h 20m (was: 6h 10m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Simon >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=167548&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-167548 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 19/Nov/18 23:40 Start Date: 19/Nov/18 23:40 Worklog Time Spent: 10m Work Description: tvalentyn commented on a change in pull request #6925: [BEAM-5315] Partially port IO: avro schema parsing and codecs URL: https://github.com/apache/beam/pull/6925#discussion_r234822869 ## File path: sdks/python/apache_beam/io/tfrecordio_test.py ## @@ -405,6 +405,9 @@ def test_process_auto(self): assert_that(result, equal_to(['foo', 'bar'])) +@unittest.skipIf(sys.version_info[0] == 3, Review comment: There was a change recently in TFRecordIO: https://github.com/apache/beam/commit/5bb33aa131ebd6976fdf5a31b3d6e7689f53#diff-6f2f08b1aad2a8511b307fb8736da843 - is this test still failing? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 167548) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Simon >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=167547&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-167547 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 19/Nov/18 23:40 Start Date: 19/Nov/18 23:40 Worklog Time Spent: 10m Work Description: tvalentyn commented on a change in pull request #6925: [BEAM-5315] Partially port IO: avro schema parsing and codecs URL: https://github.com/apache/beam/pull/6925#discussion_r234822883 ## File path: sdks/python/apache_beam/io/avroio.py ## @@ -360,8 +360,10 @@ def num_records(self): def records(self): decoder = avroio.BinaryDecoder( io.BytesIO(self._decompressed_block_bytes)) -reader = avroio.DatumReader( -writers_schema=self._schema, readers_schema=self._schema) +# Kwargs writers_schema and readers_schema were renamed to +# writer_schema and reader_schema in py3. +# Therefore, the kwargs were left out to be both py2 and py3 compatible. Review comment: This comment is relevant for PR review, but I think it's not necessary for readers of the code. I suggest we remove it. If we want to keep the names of positional arguments for readability, we can do: ``` writer_schema=self._schema reader_schema=self._schema reader = avroio.DatumReader( writer_schema, reader_schema) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 167547) Time Spent: 6h 10m (was: 6h) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Simon >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[ https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=167399&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-167399 ] ASF GitHub Bot logged work on BEAM-5315: Author: ASF GitHub Bot Created on: 19/Nov/18 14:14 Start Date: 19/Nov/18 14:14 Worklog Time Spent: 10m Work Description: splovyt commented on issue #6925: [BEAM-5315] Partially port IO: avro schema parsing and codecs URL: https://github.com/apache/beam/pull/6925#issuecomment-439906064 Following a series of edits based on the status of #7022, most of the commits included in this PR were reverted. The remaining edits solve about 10 errors in the IO package. Thanks @RobbeSneyders for providing a solution! @markflyhigh @aaltay @tvalentyn @robertwb PTAL! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 167399) Time Spent: 6h (was: 5h 50m) > Finish Python 3 porting for io module > - > > Key: BEAM-5315 > URL: https://issues.apache.org/jira/browse/BEAM-5315 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Robbe >Assignee: Simon >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5315) Finish Python 3 porting for io module
[
https://issues.apache.org/jira/browse/BEAM-5315?focusedWorklogId=163570&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-163570
]
ASF GitHub Bot logged work on BEAM-5315:
Author: ASF GitHub Bot
Created on: 07/Nov/18 18:30
Start Date: 07/Nov/18 18:30
Worklog Time Spent: 10m
Work Description: aaltay closed pull request #6975: [BEAM-5315] Always
write newline in bytes for textsink
URL: https://github.com/apache/beam/pull/6975
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/sdks/python/apache_beam/io/textio.py
b/sdks/python/apache_beam/io/textio.py
index f96a02ec8a2..73ab4a9deaa 100644
--- a/sdks/python/apache_beam/io/textio.py
+++ b/sdks/python/apache_beam/io/textio.py
@@ -383,7 +383,7 @@ def open(self, temp_path):
if self._header is not None:
file_handle.write(self._header)
if self._append_trailing_newlines:
-file_handle.write('\n')
+file_handle.write(b'\n')
return file_handle
def display_data(self):
@@ -397,7 +397,7 @@ def write_encoded_record(self, file_handle, encoded_value):
"""Writes a single encoded record."""
file_handle.write(encoded_value)
if self._append_trailing_newlines:
- file_handle.write('\n')
+ file_handle.write(b'\n')
def _create_text_source(
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 163570)
Time Spent: 5h 50m (was: 5h 40m)
> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Robbe
>Assignee: Simon
>Priority: Major
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
