[
https://issues.apache.org/jira/browse/BEAM-5191?focusedWorklogId=270428&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-270428
]
ASF GitHub Bot logged work on BEAM-5191:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Jul/19 19:03
Start Date: 01/Jul/19 19:03
Worklog Time Spent: 10m
Work Description: chamikaramj commented on pull request #8945:
[BEAM-5191] Support for BigQuery clustering
URL: https://github.com/apache/beam/pull/8945#discussion_r299177287
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
##########
@@ -1793,6 +1814,32 @@ static String getExtractDestinationUri(String
extractDestinationDir) {
return toBuilder().setJsonTimePartitioning(partitioning).build();
}
+ /**
+ * Specifies the clustering fields to use when writing to a single output
table. Can only be
+ * used when {@link#withTimePartitioning(TimePartitioning)} is set. If
{@link
+ * #to(SerializableFunction)} or {@link #to(DynamicDestinations)} is used
to write to dynamic
+ * tables, this setting is ignored; instead, see {@link
#enableClustering()}.
+ */
+ public Write<T> withClustering(Clustering clustering) {
+ checkArgument(clustering != null, "clustering can not be null");
+ return
toBuilder().setEnableClustering(true).setClustering(clustering).build();
+ }
+
+ /**
+ * Allows writing to clustered tables when {@link
#to(SerializableFunction)} or {@link
+ * #to(DynamicDestinations)} is used. The returned {@link
TableDestination} objects should
+ * specify the clustering fields per table. If writing to a single table,
use {@link
+ * #withClustering(Clustering)} instead to specify the clustering fields.
+ *
+ * <p>Setting this option enables use of {@link TableDestinationCoderV3}
which encodes
+ * clustering information. Pipelines using an older coder must be drained
before setting this
+ * option, since {@link TableDestinationCoderV3} will not be able to read
state written with a
+ * previous version.
+ */
+ public Write<T> enableClustering() {
Review comment:
Having these two methods seems to make the API pretty brittle.
How about just having one function withClustering() that optionally takes a
clustering object ? In dynamic destinations case optional Clustering object can
be skipped/null and the method will behave similar to enableClustering() ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 270428)
Time Spent: 8h 50m (was: 8h 40m)
> Add support for writing to BigQuery clustered tables
> ----------------------------------------------------
>
> Key: BEAM-5191
> URL: https://issues.apache.org/jira/browse/BEAM-5191
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Affects Versions: 2.6.0
> Reporter: Robert Sahlin
> Assignee: Wout Scheepers
> Priority: Minor
> Labels: features, newbie
> Time Spent: 8h 50m
> Remaining Estimate: 0h
>
> Google recently added support for clustered tables in BigQuery. It would be
> useful to set clustering columns the same way as for partitioning. It should
> support multiple fields (4) for clustering.
> For example:
> [BigQueryIO.Write|https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html]<[T|https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html]>
> .withClustering(new Clustering().setField("productId").setType("STRING"))
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
