reuvenlax commented on a change in pull request #16795:
URL: https://github.com/apache/beam/pull/16795#discussion_r808304752
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiLoads.java
##########
@@ -100,48 +102,23 @@ public WriteResult
expandTriggered(PCollection<KV<DestinationT, ElementT>> input
// Handle triggered, low-latency loads into BigQuery.
PCollection<KV<DestinationT, ElementT>> inputInGlobalWindow =
input.apply("rewindowIntoGlobal", Window.into(new GlobalWindows()));
-
- // First shard all the records.
- // TODO(reuvenlax): Add autosharding support so that users don't have to
pick a shard count.
- PCollection<KV<ShardedKey<DestinationT>, byte[]>> shardedRecords =
+ PCollection<KV<ShardedKey<DestinationT>, Iterable<byte[]>>> groupedRecords
=
inputInGlobalWindow
.apply("Convert", new
StorageApiConvertMessages<>(dynamicDestinations, bqServices))
.apply(
- "AddShard",
- ParDo.of(
- new DoFn<KV<DestinationT, byte[]>,
KV<ShardedKey<DestinationT>, byte[]>>() {
- int shardNumber;
-
- @Setup
- public void setup() {
- shardNumber =
ThreadLocalRandom.current().nextInt(numShards);
- }
-
- @ProcessElement
- public void processElement(
- @Element KV<DestinationT, byte[]> element,
- OutputReceiver<KV<ShardedKey<DestinationT>, byte[]>>
o) {
- DestinationT destination = element.getKey();
- ByteBuffer buffer = ByteBuffer.allocate(Integer.BYTES);
- buffer.putInt(++shardNumber % numShards);
- o.output(
- KV.of(ShardedKey.of(destination, buffer.array()),
element.getValue()));
- }
- }))
- .setCoder(KvCoder.of(ShardedKey.Coder.of(destinationCoder),
ByteArrayCoder.of()));
-
- PCollection<KV<ShardedKey<DestinationT>, Iterable<byte[]>>> groupedRecords
=
- shardedRecords.apply(
- "GroupIntoBatches",
- GroupIntoBatches.<ShardedKey<DestinationT>, byte[]>ofByteSize(
- MAX_BATCH_SIZE_BYTES, (byte[] e) -> (long) e.length)
- .withMaxBufferingDuration(triggeringFrequency));
+ "GroupIntoBatches",
+ GroupIntoBatches.<DestinationT, byte[]>ofByteSize(
+ MAX_BATCH_SIZE_BYTES, (byte[] e) -> (long) e.length)
+ .withMaxBufferingDuration(triggeringFrequency)
+ .withShardedKey());
Review comment:
Shouldn't we only be doing this if the user specified auto sharding?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
