[
https://issues.apache.org/jira/browse/BEAM-8949?focusedWorklogId=432202&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-432202
]
ASF GitHub Bot logged work on BEAM-8949:
----------------------------------------
Author: ASF GitHub Bot
Created on: 08/May/20 17:48
Start Date: 08/May/20 17:48
Worklog Time Spent: 10m
Work Description: mszb commented on a change in pull request #11210:
URL: https://github.com/apache/beam/pull/11210#discussion_r422280614
##########
File path: sdks/python/apache_beam/io/gcp/experimental/spannerio_test.py
##########
@@ -499,6 +499,7 @@ def test_batch_byte_size(
# and each bach should contains 25 mutations.
res = (
p | beam.Create(mutation_group)
+ | 'combine to list' >> beam.combiners.ToList()
Review comment:
The user does not have to add ToList transform in the production
pipeline. I only added this to test the batch process.
The previous implementation of batching (without ToList transform) was as
per the java implementation but without the sorting of the transactions by
table and primary key (this is also documented as a feature to be added later).
##########
File path: sdks/python/apache_beam/io/gcp/experimental/spannerio.py
##########
@@ -1008,31 +1007,30 @@ def _reset_count(self):
self._cells = 0
def process(self, element):
- mg_info = element.info
+ for elem in element:
Review comment:
Make sense, in that case, we don't need to alter the connector code
anymore, it was working as expected. Thanks, @chamikaramj for the feedback as
it is always helpful.
I'll remove the changes from the spanner io connector and update the IT test
code for the assertion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 432202)
Time Spent: 10.5h (was: 10h 20m)
> Add Spanner IO Integration Test for Python
> ------------------------------------------
>
> Key: BEAM-8949
> URL: https://issues.apache.org/jira/browse/BEAM-8949
> Project: Beam
> Issue Type: Test
> Components: io-py-gcp
> Reporter: Shoaib Zafar
> Assignee: Shoaib Zafar
> Priority: Major
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> Spanner IO (Python SDK) contains PTransform which uses the BatchAPI to read
> from the spanner. Currently, it only contains direct runner unit tests. In
> order to make this functionality available for the users, integration tests
> also need to be added.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
