[ https://issues.apache.org/jira/browse/BEAM-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338269#comment-16338269 ]
Eugene Kirpichov commented on BEAM-3501: ---------------------------------------- Please include the complete error stacktrace rather than just the error message, and ideally include also a Dataflow job ID so an oncall engineer can look at what's going on at bigquery side. > BigQuery Partitioned table creation/write fails when destination has > partition decorator > ---------------------------------------------------------------------------------------- > > Key: BEAM-3501 > URL: https://issues.apache.org/jira/browse/BEAM-3501 > Project: Beam > Issue Type: Bug > Components: sdk-java-gcp > Reporter: Darshan Mehta > Assignee: Chamikara Jayalath > Priority: Major > > Following is the code that writes to BigQuery: > {code:java} > BigQueryIO.writeTableRows() > .to(destination) > .withCreateDisposition(CREATE_IF_NEEDED) > .withWriteDisposition(WRITE_APPEND) > .withSchema(tableSchema) > .expand(tableRows);{code} > > Here's the destination's implementation: > {code:java} > public TableDestination apply(ValueInSingleWindow<TableRow> input) { > String partition = timestampExtractor.apply(input.getValue()) > .toString(DateTimeFormat.forPattern("yyyyMMdd").withZoneUTC()); > TableReference tableReference = new TableReference(); > tableReference.setDatasetId(dataset); > tableReference.setProjectId(projectId); > tableReference.setTableId(String.format("%s_%s", table, partition)); > log.debug("Will write to BigQuery table: %s", tableReference); > return new TableDestination(tableReference, null); > }{code} > > When the dataflow tries to write to this table, I see the following message: > {code:java} > "errors" : [ { > "domain" : "global", > "message" : "Cannot read partition information from a table that is not > partitioned: <project_id>:<dataset>.<table>$19730522", > "reason" : "invalid" > } ]{code} > So, it looks like it's not creating tables with partition in the first place? > Apache beam version : 2.2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)