[
https://issues.apache.org/jira/browse/BEAM-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15746824#comment-15746824
]
ASF GitHub Bot commented on BEAM-1151:
--------------------------------------
GitHub user reuvenlax opened a pull request:
https://github.com/apache/incubator-beam/pull/1609
BEAM-1151 Add failure handling to BigQueryIO.Write
Allow a pipeline to handle streaming insert failures when using
BigQueryIO.Write in streaming mode. TableRow objects that fail to insert are
returned to the user as a PCollection.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/reuvenlax/incubator-beam bq_dead_letter
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/1609.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1609
----
commit 379af6129f91ffb1b1b3aa917f3e0f13cb412132
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-11-17T18:57:41Z
Add dead-letter support to BigQueryIO
Add dead-letter support to BigQueryIO
Strip out non-testing changes.
commit 8b2f3010058674a5ed4359148397296b7aad31de
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-11-19T23:09:10Z
Remove dead code.
commit 4cf5edf61d29466811cb3487561d02f51bbdfd5a
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-11-21T20:30:29Z
Add testing coverage for BigQuery streaming writes.
commit fa33d7621907f9c7e37a769b530514eb44a4c10a
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-11-21T21:52:01Z
Fix checkstyle errors and remove remaining tests out of
BigQueryTableInserterTest.
commit a91ac51f3f127921e02c64c4f7c8e2c1c4828eaf
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-11-21T22:32:03Z
Fix more checkstyle errors.
commit e31b593bd2f8d007efeeb14fd30145bc56fff01a
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-11-26T12:55:28Z
Make sure to clear static cache of created tables between tests.
commit 2011433e201c983575208d1353f4221e31b44b0e
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-12-09T20:01:10Z
Refactor tryCreateTable method. Removes unused create and write disposition
code on this path.
commit d5b549c24870bcd6593dc6616360d6c9565381d2
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-12-10T03:39:02Z
Fix test failure. Remove no-longer applicable test.
commit 98febf780981ec978fef9a5a9392a2e605e74326
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-12-13T00:35:38Z
Apply review comments.
commit 9e8cadbff8ac7fe5f5fc3d4a6c07221316144ef4
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-12-13T05:28:42Z
Fix failures
commit 1f840efe0a71469fde3db37fba910c296ce0d76c
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-12-13T08:27:12Z
Fix broken test
commit 2a446a508822da167162a022448d3d5f347287f9
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-12-13T21:52:44Z
Template code broke windowed BigQuery output - fix.
Fix invalid GCS access in BigQueryIOTest
commit 14da9adec51a43e6f02672677d640a88fd42894b
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-11-17T18:57:41Z
Add dead-letter support to BigQueryIO
commit dfced01604d7dff5c46c99389f1f945ac5ab8995
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-11-19T22:28:10Z
Add dead-letter support to BigQueryIO
commit 9e66dffabaa71c7e01f3cc46736df40b163e9e82
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-11-26T12:53:55Z
Add some tests of dead letter policy, and default retry policies.
commit 3e3cfcf311c6db0ac97ae360c1b403d1a00cae2c
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-12-01T08:29:58Z
Add testing.
commit 2af0de1c84ac48bfcf4ec17af8e9d80012ff90c2
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-12-14T00:57:48Z
Fixes to test
----
> BigQueryIO.Write has no way of handling failures
> ------------------------------------------------
>
> Key: BEAM-1151
> URL: https://issues.apache.org/jira/browse/BEAM-1151
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Affects Versions: 0.4.0-incubating
> Reporter: Reuven Lax
> Assignee: Reuven Lax
> Fix For: 0.3.0-incubating
>
>
> BigQueryIO.Write in streaming mode uses BigQuery's streaming insert API.
> Inserts can fail for a variety of reasons, and the current connector provides
> no way of managing those failures. This PR allows better failure handling.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
