[
https://issues.apache.org/jira/browse/BEAM-5422?focusedWorklogId=418175&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418175
]
ASF GitHub Bot logged work on BEAM-5422:
----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Apr/20 03:04
Start Date: 08/Apr/20 03:04
Worklog Time Spent: 10m
Work Description: udim commented on pull request #11241: [BEAM-5422]
Document DynamicDestinations.getTable uniqueness requirement
URL: https://github.com/apache/beam/pull/11241#discussion_r405231575
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinations.java
##########
@@ -142,7 +142,11 @@ void setSideInputAccessorFromProcessContext(DoFn<?,
?>.ProcessContext context) {
return null;
}
- /** Returns a {@link TableDestination} object for the destination. May not
return null. */
+ /**
+ * Returns a {@link TableDestination} object for the destination. May not
return null. Return
+ * value needs to be unique to each destination: may not return the same
{@link TableDestination}
+ * for different destinations.
Review comment:
TLDR: Pablo is right.
In Python SDK, a user function translates an element to a TableReference.
In Java SDK, a user DynamicDestinations instance translates an element to a
DestinationT, and then to a TableDestination.
Java does a reshuffle on (DestinationT, element) pairs, while Python does it
on (TableReference, element) pairs.
(Not sure why Java uses an intermediate DestinationT. Convenience? Better
GBK performance? Lower resource use?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 418175)
Time Spent: 1h 20m (was: 1h 10m)
> Update BigQueryIO DynamicDestinations documentation to clarify usage of
> getDestination() and getTable()
> -------------------------------------------------------------------------------------------------------
>
> Key: BEAM-5422
> URL: https://issues.apache.org/jira/browse/BEAM-5422
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Reporter: Chamikara Madhusanka Jayalath
> Assignee: Chamikara Madhusanka Jayalath
> Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Currently, there are some details related to these methods that should be
> further clarified. For example, getTable() is expected to return a unique
> value for each destination.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
