Andre created BEAM-2370: --------------------------- Summary: BigQuery Insert with Partition Decorator throwing error Key: BEAM-2370 URL: https://issues.apache.org/jira/browse/BEAM-2370 Project: Beam Issue Type: Bug Components: sdk-java-gcp Affects Versions: 2.0.0 Environment: DirectRunner Reporter: Andre Assignee: Daniel Halperin
Running a DataFlow job with the DirectRunner which is inserting data into a partitioned table using decorators throws the following error multiple times BUT still inserts records into the right partition. {code:java|title=Error} org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl executeWithRetries INFO: Ignore the error and retry the request. com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request { "code" : 400, "errors" : [ { "domain" : "global", "message" : "Invalid table ID \"mytable_orders$20170516\". Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used.", "reason" : "invalid" } ], "message" : "Invalid table ID \"mytable_orders$20170516\". Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used." } at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146) {code} {code:java|title=Code} // Write TableRows to BQ rows.apply("TransformationStep", ParDo.of(new Outputter())) .apply("WindowDaily", Window. <TableRow> into(CalendarWindows.days(1))) .apply("WriteToBQ", BigQueryIO.writeTableRows() .to(new SerializableFunction <ValueInSingleWindow<TableRow> , TableDestination> () { private static final long serialVersionUID = 8196602721734820219 L; @Override public TableDestination apply(ValueInSingleWindow <TableRow> value) { String dayString = DateTimeFormat.forPattern("yyyyMMdd").withZone(DateTimeZone.UTC) .print(((IntervalWindow) value.getWindow()).start()); TableDestination td = new TableDestination("my-project:dataset.mytable_orders$" + dayString, ""); return td; } }).withSchema(mySchema) .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER) .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE)); {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)