[ https://issues.apache.org/jira/browse/BEAM-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491095#comment-16491095 ]
Peter Brumblay commented on BEAM-4370: -------------------------------------- Ok. I have figured it out through a lot of pain and suffering. This is NOT a bug in BigQueryIO. The problem: If you include a *malformed* table decorator, you get this error message from the BIGQUERY REST API: "Invalid table ID \"merging_partitions$2018-03-03\". Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used." Note 2 things 1. The table decorator I'm trying to use is NOT in the correct format. it should NOT be $YYYY-MM-DD. It SHOULD be $YYYYMMDD. 2. I get an error message which is COMPLETELY misleading. Table decorators CAN and SHOULD be used when working with partitions. This is a working example: [https://gist.github.com/pbrumblay/b49bf08e0b652319120dfb1c861f0936] Closing. > Cannot write to bigquery partitions using DynamicDestination or > SerializableFunction<> > -------------------------------------------------------------------------------------- > > Key: BEAM-4370 > URL: https://issues.apache.org/jira/browse/BEAM-4370 > Project: Beam > Issue Type: Bug > Components: io-java-gcp > Affects Versions: 2.4.0 > Reporter: Peter Brumblay > Assignee: Chamikara Jayalath > Priority: Major > Fix For: Not applicable > > > When trying to use BigQueryIO.writeTableRows() with > DynamicDestinations<TableRow, String> or > SerializableFunction<ValueInSingleWindow<TableRow>, TableDestination>, the > operation fails with an error message claiming that decorators are not > allowed: > "Invalid table ID <table>$<partition>. Table IDs must be alphanumeric (plus > underscores) and must be at most 1024 characters long. Also, Table decorators > cannot be used." > This should work according to ... > # > [https://stackoverflow.com/questions/47351578/create-dynamic-side-outputs-in-apache-beam-dataflow#comment81668927_47351578] > and [https://stackoverflow.com/a/43505535] > # > [https://shinesolutions.com/2017/12/05/fun-with-serializable-functions-and-dynamic-destinations-in-cloud-dataflow/] > See the following for a complete example demonstrating the error with both > scenarios: > [https://github.com/pbrumblay/bigqueryioproblems/blob/master/src/main/java/com/fearlesstg/CannotWriteToPartitionedTable.java] > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)