[
https://issues.apache.org/jira/browse/BEAM-2817?focusedWorklogId=80621&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-80621
]
ASF GitHub Bot logged work on BEAM-2817:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Mar/18 22:47
Start Date: 14/Mar/18 22:47
Worklog Time Spent: 10m
Work Description: jkff commented on a change in pull request #4606:
[BEAM-2817] BigQuery queries are allowed to run in either BATCH or IN…
URL: https://github.com/apache/beam/pull/4606#discussion_r174633409
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySource.java
##########
@@ -74,12 +77,19 @@ private BigQueryQuerySource(
Boolean useLegacySql,
BigQueryServices bqServices,
Coder<T> coder,
- SerializableFunction<SchemaAndRecord, T> parseFn) {
+ SerializableFunction<SchemaAndRecord, T> parseFn,
+ Priority priority) {
super(stepUuid, bqServices, coder, parseFn);
this.query = checkNotNull(query, "query");
this.flattenResults = checkNotNull(flattenResults, "flattenResults");
this.useLegacySql = checkNotNull(useLegacySql, "useLegacySql");
this.dryRunJobStats = new AtomicReference<>();
+ if (priority != BigQueryIO.TypedRead.Priority.BATCH
Review comment:
I'm fixing this in my PR. Probably not too big of a deal, just this feature
won't really be usable in the release, but will be in the next one. No
regression.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 80621)
Time Spent: 2.5h (was: 2h 20m)
> Bigquery queries should allow options to run in batch mode or not
> -----------------------------------------------------------------
>
> Key: BEAM-2817
> URL: https://issues.apache.org/jira/browse/BEAM-2817
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.0.0
> Reporter: Lara Schmidt
> Assignee: Justin Tumale
> Priority: Major
> Labels: newbie, starter
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> When bigquery read does a query it sets the mode to batch. A batch query can
> be very slow to schedule as it batches it with other queries. However it
> doesn't use batch quota which is better for some cases. However, in some
> cases a fast query is better (especially in timed tests). It would be a good
> idea to have a configuration to the bigquery source to set this per-read.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
