[
https://issues.apache.org/jira/browse/BEAM-3342?focusedWorklogId=266364&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-266364
]
ASF GitHub Bot logged work on BEAM-3342:
----------------------------------------
Author: ASF GitHub Bot
Created on: 25/Jun/19 06:49
Start Date: 25/Jun/19 06:49
Worklog Time Spent: 10m
Work Description: mf2199 commented on pull request #8457: [BEAM-3342]
Create a Cloud Bigtable IO connector for Python
URL: https://github.com/apache/beam/pull/8457#discussion_r297029891
##########
File path: sdks/python/apache_beam/io/gcp/bigtableio.py
##########
@@ -141,3 +148,129 @@ def expand(self, pvalue):
| beam.ParDo(_BigTableWriteFn(beam_options['project_id'],
beam_options['instance_id'],
beam_options['table_id'])))
+
+
+class BigtableSource(iobase.BoundedSource):
+ def __init__(self, project_id, instance_id, table_id, filter_=None):
+ """ Constructor of the Read connector of Bigtable
+
+ Args:
+ project_id: [string] GCP Project of to write the Rows
+ instance_id: [string] GCP Instance to write the Rows
+ table_id: [string] GCP Table to write the `DirectRows`
+ filter_: [RowFilter] Filter to apply to columns in a row.
+ """
+ super(self.__class__, self).__init__()
+ self._init({'project_id': project_id,
+ 'instance_id': instance_id,
+ 'table_id': table_id,
+ 'filter_': filter_})
+
+ def __getstate__(self):
+ return self.beam_options
+
+ def __setstate__(self, options):
+ self._init(options)
+
+ def _init(self, options):
+ self.beam_options = options
+ self.table = None
+ self.sample_row_keys = None
+ self.row_count = Metrics.counter(self.__class__.__name__, 'Row count')
+
+ def _get_table(self):
+ if self.table is None:
+ self.table = Client(project=self.beam_options['project_id'])\
+ .instance(self.beam_options['instance_id'])\
+ .table(self.beam_options['table_id'])
+ return self.table
+
+ def get_sample_row_keys(self):
+ """ Get a sample of row keys in the table.
+
+ The returned row keys will delimit contiguous sections of the table of
+ approximately equal size, which can be used to break up the data for
+ distributed tasks like mapreduces.
+ :returns: A cancel-able iterator. Can be consumed by calling ``next()``
+ or by casting to a :class:`list` and can be cancelled
by
+ calling ``cancel()``.
+
+ ***** NOTE: For unclear reasons, the function returns generator even
Review comment:
As the BoundedSource [BigtableSource] has been replaced by a
PTransform/DoFn, this one no longer applies.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 266364)
Time Spent: 23.5h (was: 23h 20m)
> Create a Cloud Bigtable IO connector for Python
> -----------------------------------------------
>
> Key: BEAM-3342
> URL: https://issues.apache.org/jira/browse/BEAM-3342
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Solomon Duskis
> Assignee: Solomon Duskis
> Priority: Major
> Time Spent: 23.5h
> Remaining Estimate: 0h
>
> I would like to create a Cloud Bigtable python connector.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
