[ https://issues.apache.org/jira/browse/BEAM-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15948014#comment-15948014 ]
Daniel Halperin commented on BEAM-1834: --------------------------------------- It sounds like this a feature request for BigQueryIO supporting Data-dependent Schemas in addition to Data-dependent tables? The latter is explicitly not supported at this time. > Bigquery Write validation doesn't work well with ValueInSingleWindow > -------------------------------------------------------------------- > > Key: BEAM-1834 > URL: https://issues.apache.org/jira/browse/BEAM-1834 > Project: Beam > Issue Type: Bug > Components: sdk-java-gcp > Reporter: Kevin Peterson > Assignee: Daniel Halperin > > I am using the new {{Write<T> to(SerializableFunction<ValueInSingleWindow<T>, > String> tableSpecFunction)}} function to write data to different Bigquery > tables depending on the values. I'm my case, the values can have a different > schema (it starts as an {{Any}} encoded protobuf, which I parse and expand to > a {{TableRow}} object). > Since the tables have different schemas, the existing implementation of > {{withSchema}} doesn't work. > Some options: > # Allow {{CreateDisposition.CREATE_NEVER}} in this situation. Failed inserts > from a missing table just fail (and eventually pass through via BEAM-190). > # Add a new {{withSchema(SerializableFunction<ValueInSingleWindow<T>, > TableSchema>}} function. > I think eventually both of the above should be allowable configurations, but > just one will unblock my current error. Happy to implement, given some > guidance on design preferences. -- This message was sent by Atlassian JIRA (v6.3.15#6346)