[
https://issues.apache.org/jira/browse/BEAM-6611?focusedWorklogId=278667&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-278667
]
ASF GitHub Bot logged work on BEAM-6611:
----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Jul/19 02:36
Start Date: 18/Jul/19 02:36
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #8871: [BEAM-6611]
BigQuery file loads in Streaming for Python SDK
URL: https://github.com/apache/beam/pull/8871#discussion_r304713800
##########
File path: sdks/python/apache_beam/io/gcp/bigquery_file_loads.py
##########
@@ -622,8 +653,9 @@ def expand(self, pcoll):
test_client=self.test_client,
temporary_tables=self.temp_tables,
additional_bq_parameters=self.additional_bq_parameters),
- load_job_name_pcv, *self.schema_side_inputs).with_outputs(
- TriggerLoadJobs.TEMP_TABLES, main='main')
+ load_job_name_pcv, self.is_streaming_pipeline,
Review comment:
I wonder if we should pass `is_streaming_pipeline` at pipeline construction
(i.e. in the constructor) rather than a side input. It would allow us to show
it as .p.ex. display data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 278667)
Time Spent: 4.5h (was: 4h 20m)
> A Python Sink for BigQuery with File Loads in Streaming
> -------------------------------------------------------
>
> Key: BEAM-6611
> URL: https://issues.apache.org/jira/browse/BEAM-6611
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Reporter: Pablo Estrada
> Assignee: Tanay Tummalapalli
> Priority: Major
> Labels: gsoc, gsoc2019, mentor
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> The Java SDK supports a bunch of methods for writing data into BigQuery,
> while the Python SDK supports the following:
> - Streaming inserts for streaming pipelines [As seen in [bigquery.py and
> BigQueryWriteFn|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py#L649-L813]]
> - File loads for batch pipelines [As implemented in [PR
> 7655|https://github.com/apache/beam/pull/7655]]
> Qucik and dirty early design doc: https://s.apache.org/beam-bqfl-py-streaming
> The Java SDK also supports File Loads for Streaming pipelines [see BatchLoads
> application|https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1709-L1776].
> File loads have the advantage of being much cheaper than streaming inserts
> (although they also are slower for the records to show up in the table).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
