[
https://issues.apache.org/jira/browse/BEAM-3200?focusedWorklogId=314504&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-314504
]
ASF GitHub Bot logged work on BEAM-3200:
----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Sep/19 17:27
Start Date: 18/Sep/19 17:27
Worklog Time Spent: 10m
Work Description: reuvenlax commented on issue #9556: [BEAM-3200,
BEAM-3772] Fix: Honor create and write disposition with dynamic destinations
URL: https://github.com/apache/beam/pull/9556#issuecomment-532784823
Also, can you verify that your DynamicDestinations class is deterministic?
The code as written strongly assumes that a single DestinationT always
deterministically maps to a single BQ table. If that does not hold, many
things could break, and it could also cause the error you are seeing.
On Wed, Sep 18, 2019 at 10:22 AM Reuven Lax <re...@google.com> wrote:
> The triggers here are per key, and the key is the TableDestination. It
> should be impossible for there to be a trigger with no data for a given
> table, unless there is a very deep bug in Beam itself. This might require
> debugging a live job to figure out what happened. This PR will actually
> break the current functionality.
>
> These failures showed up for other users when they triggered the temp-file
> path, and we incorrectly applied the create-disposition change to the
> temp-file path (because the WriteTables transform was reused). However that
> bug should be fixed now in the codebase.
>
> On Tue, Sep 17, 2019 at 8:39 AM Lukasz Cwik <notificati...@github.com>
> wrote:
>
>> Another user seemed to complain of this as well:
>>
https://stackoverflow.com/questions/57969706/bigqueryio-only-first-day-table-can-be-created-despite-having-createdispositi
>>
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub
>>
<https://github.com/apache/beam/pull/9556?email_source=notifications&email_token=AFAYJVL4ONWYGT3324IY4V3QKD243A5CNFSM4IWBC3BKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD646U2Y#issuecomment-532277867>,
>> or mute the thread
>>
<https://github.com/notifications/unsubscribe-auth/AFAYJVJTH3P6ZXBOPZB3RHDQKD243ANCNFSM4IWBC3BA>
>> .
>>
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 314504)
Time Spent: 3h 40m (was: 3.5h)
> Streaming Pipeline throws RuntimeException when using DynamicDestinations and
> Method.FILE_LOADS
> -----------------------------------------------------------------------------------------------
>
> Key: BEAM-3200
> URL: https://issues.apache.org/jira/browse/BEAM-3200
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.2.0
> Reporter: AJ
> Priority: Critical
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> I am trying to use Method.FILE_LOADS for loading data into BQ in my streaming
> pipeline using RC3 release of 2.2.0. I am writing to around 500 tables using
> DynamicDestinations and I am also using
> withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED). Everything works
> fine when the first time bigquery load jobs get triggered. But on subsequent
> triggers pipeline throws a RuntimeException about table not found even though
> I created the pipeline with CreateDisposition.CREATE_IF_NEEDED. The exact
> exception is:
> {code}
> java.lang.RuntimeException: Failed to create load job with id prefix
> 717aed9ed1ef4aa7a616e1132f8b7f6d_a0928cae3d670b32f01ab2d9fe5cc0ee_00001_00001,
> reached max retries: 3, last failed load job: {
> "configuration" : {
> "load" : {
> "createDisposition" : "CREATE_NEVER",
> "destinationTable" : {
> "datasetId" : ...,
> "projectId" : ...,
> "tableId" : ....
> },
> "errors" : [ }
> "message" : "Not found: Table ....,
> "reason" : "notFound"
> } ],
> {code}
> My theory is all the subsequent load jobs get trigged using CREATE_NEVER
> disposition and
> this might be due to
> https://github.com/apache/beam/blob/release-2.2.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L140
> When using DynamicDestinations all the destination tables might not be known
> during the first trigger and hence the pipeline's create disposition should
> be respected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
