[ https://issues.apache.org/jira/browse/BEAM-6611?focusedWorklogId=280124&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-280124 ]
ASF GitHub Bot logged work on BEAM-6611: ---------------------------------------- Author: ASF GitHub Bot Created on: 20/Jul/19 19:13 Start Date: 20/Jul/19 19:13 Worklog Time Spent: 10m Work Description: ttanay commented on issue #8871: [BEAM-6611] BigQuery file loads in Streaming for Python SDK URL: https://github.com/apache/beam/pull/8871#issuecomment-513492189 Run Python 3.6 PostCommit ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 280124) Time Spent: 6h 50m (was: 6h 40m) > A Python Sink for BigQuery with File Loads in Streaming > ------------------------------------------------------- > > Key: BEAM-6611 > URL: https://issues.apache.org/jira/browse/BEAM-6611 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core > Reporter: Pablo Estrada > Assignee: Tanay Tummalapalli > Priority: Major > Labels: gsoc, gsoc2019, mentor > Time Spent: 6h 50m > Remaining Estimate: 0h > > The Java SDK supports a bunch of methods for writing data into BigQuery, > while the Python SDK supports the following: > - Streaming inserts for streaming pipelines [As seen in [bigquery.py and > BigQueryWriteFn|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/bigquery.py#L649-L813]] > - File loads for batch pipelines [As implemented in [PR > 7655|https://github.com/apache/beam/pull/7655]] > Qucik and dirty early design doc: https://s.apache.org/beam-bqfl-py-streaming > The Java SDK also supports File Loads for Streaming pipelines [see BatchLoads > application|https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1709-L1776]. > File loads have the advantage of being much cheaper than streaming inserts > (although they also are slower for the records to show up in the table). -- This message was sent by Atlassian JIRA (v7.6.14#76016)