[
https://issues.apache.org/jira/browse/BEAM-11587?focusedWorklogId=769387&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-769387
]
ASF GitHub Bot logged work on BEAM-11587:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 12/May/22 02:03
Start Date: 12/May/22 02:03
Worklog Time Spent: 10m
Work Description: svetakvsundhar commented on code in PR #17159:
URL: https://github.com/apache/beam/pull/17159#discussion_r870886625
##########
sdks/python/apache_beam/io/gcp/bigquery.py:
##########
@@ -2525,6 +2526,12 @@ def _get_pipeline_details(unused_elm):
**self._kwargs))
| _PassThroughThenCleanupTempDatasets(project_to_cleanup_pcoll))
+ def get_pcoll_from_schema(table_schema):
+ pcoll_val = apache_beam.io.gcp.bigquery_schema_tools.\
+ produce_pcoll_with_schema(table_schema)
+ return beam.Map(lambda values: pcoll_val(**values)).with_output_types(
Review Comment:
@TheNeuralBit just to follow up here, it looks like ```rowcoder``` was
indeed the coder being used. I verified that the asserts passed here
https://github.com/svetakvsundhar/beam/blob/bqio/sdks/python/apache_beam/coders/row_coder_test.py#L263,
and they both proved to be using rowcoder.
Are there any other intermediate methods I can run on our ```usertype``` to
verify during the rowcoder ptransform? One thing I noted was that
```usertype``` doesn't have a ```.schema``` property, and that I wasn't able to
verify if the argument of this ptransform is a schema
(```apache_beam.portability.api.schema_pb2.Schema```). Is there an easy way to
verify that the input is indeed of that type?
Issue Time Tracking
-------------------
Worklog Id: (was: 769387)
Time Spent: 8h 20m (was: 8h 10m)
> Support pd.read_gbq and DataFrame.to_gbq
> ----------------------------------------
>
> Key: BEAM-11587
> URL: https://issues.apache.org/jira/browse/BEAM-11587
> Project: Beam
> Issue Type: New Feature
> Components: dsl-dataframe, io-py-gcp, sdk-py-core
> Reporter: Brian Hulette
> Assignee: Svetak Vihaan Sundhar
> Priority: P3
> Labels: dataframe-api
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> We should support
> [read_gbq|https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_gbq.html]
> andÂ
> [to_gbq|https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_gbq.html]
> in the DataFrame API when gcp extras are installed.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
