aprochko commented on issue #30177:
URL: https://github.com/apache/beam/issues/30177#issuecomment-3140763668
Hello @liferoad
I've faced different issue which is I think even more interesting and also
makes the jobs stucked. I was following your PR and hoped it would fix the
warnings, but now I faced with new case.
Background: we have two totally isolated streams in one job both use
BigQueryIO
Append to Context: key=ShardedKey{key=2, shardId=[1]}
streamName=projects/<PROJECT>/datasets/events_02/tables/**landmarks_02**/streams/**Cic2Y2Y4YjUwOS0wMDAwLTI2YzUtOGU3ZC1mNGY1ZTgwOWVjMmM6czI**
offset=5952366 numRows=0 tryIteration: 1 failed with
com.google.cloud.bigquery.storage.v1.Exceptions$OffsetOutOfRange: OUT_OF_RANGE:
The offset is beyond stream, expected offset 52858, received 5952366 Entity:
projects/<PROJECT>/datasets/events_02/tables/**trinity_02**/streams/**CiQ2ZDM0MWFiNy0wMDAwLTJlNDItYmY1Yi04ODNkMjRmOGVhYTQ**
Will retry with a new stream
landmarks_02 is incorrect there.
It's one message which contains both (different streams)! And amazing thing
there that then it will start trying to retry with the wrong table
(landmarks_02):
Append to Context: key=ShardedKey{key=2, shardId=[1]}
streamName=projects/bpde-prd-tracking-data/datasets/events_02/tables/landmarks_02/streams/Cic2YjI1MmQ0Ny0wMDAwLTJiYjUtYWRkOS0zYzI4NmQzNjJkY2U6czI
offset=0 numRows=0 tryIteration: **240** failed with
com.google.cloud.bigquery.storage.v1.Exceptions$SchemaMismatchedException:
INVALID_ARGUMENT: Input schema has more fields than BigQuery schema, extra
fields: 'field1trinity,field2trinity,field3trinity' Entity:
projects/<PROJECT>/datasets/events_02/tables/landmarks_02/streams/Cic2YjI1MmQ0Ny0wMDAwLTJiYjUtYWRkOS0zYzI4NmQzNjJkY2U6czI
Will retry with a new stream
So it somehow shared the client and streams and then tries 240 times and
just make the "landmark" stream stucked.
Do you think your fix could solve the issue? It seems like you only fixed
retrying, right, so it should fix the issue but not the root cause?
Should I open the new issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
