[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[ https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=357545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-357545 ] ASF GitHub Bot logged work on BEAM-8275: Author: ASF GitHub Bot Created on: 11/Dec/19 00:58 Start Date: 11/Dec/19 00:58 Worklog Time Spent: 10m Work Description: amaliujia commented on pull request #10260: [BEAM-8275, BEAM-8428] Update SQL BigQuery doc URL: https://github.com/apache/beam/pull/10260 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 357545) Time Spent: 2.5h (was: 2h 20m) > Beam SQL should support BigQuery in DIRECT_READ mode > > > Key: BEAM-8275 > URL: https://issues.apache.org/jira/browse/BEAM-8275 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Kirill Kozlov >Priority: Major > Fix For: 2.16.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode. > We also need to support DIRECT_READ mode. The method should be configurable > by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage > of the DIRECT_READ features for filter and project push down. > References: > [https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api] > [https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html] > [https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[ https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=352326=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352326 ] ASF GitHub Bot logged work on BEAM-8275: Author: ASF GitHub Bot Created on: 03/Dec/19 00:16 Start Date: 03/Dec/19 00:16 Worklog Time Spent: 10m Work Description: 11moon11 commented on pull request #10260: [BEAM-8275] Update SQL BigQuery doc URL: https://github.com/apache/beam/pull/10260 BigQueryTable now: * Can accept `TBLPROPERTIES` * Perform project and predicate push-down when method is set to `DIRECT_READ` Documentation should reflect that. Added missing commas and fixed typos. R: @apilloud CC: @TheNeuralBit CC: @amaliujia Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[ https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=320637=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-320637 ] ASF GitHub Bot logged work on BEAM-8275: Author: ASF GitHub Bot Created on: 30/Sep/19 16:34 Start Date: 30/Sep/19 16:34 Worklog Time Spent: 10m Work Description: apilloud commented on pull request #9625: [BEAM-8275] Beam SQL should support BigQuery in DIRECT_READ mode URL: https://github.com/apache/beam/pull/9625 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 320637) Time Spent: 2h 10m (was: 2h) > Beam SQL should support BigQuery in DIRECT_READ mode > > > Key: BEAM-8275 > URL: https://issues.apache.org/jira/browse/BEAM-8275 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode. > We also need to support DIRECT_READ mode. The method should be configurable > by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage > of the DIRECT_READ features for filter and project push down. > References: > [https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api] > [https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html] > [https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[ https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=320636=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-320636 ] ASF GitHub Bot logged work on BEAM-8275: Author: ASF GitHub Bot Created on: 30/Sep/19 16:33 Start Date: 30/Sep/19 16:33 Worklog Time Spent: 10m Work Description: apilloud commented on issue #9625: [BEAM-8275] Beam SQL should support BigQuery in DIRECT_READ mode URL: https://github.com/apache/beam/pull/9625#issuecomment-536643567 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 320636) Time Spent: 2h (was: 1h 50m) > Beam SQL should support BigQuery in DIRECT_READ mode > > > Key: BEAM-8275 > URL: https://issues.apache.org/jira/browse/BEAM-8275 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode. > We also need to support DIRECT_READ mode. The method should be configurable > by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage > of the DIRECT_READ features for filter and project push down. > References: > [https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api] > [https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html] > [https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[
https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=319722=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319722
]
ASF GitHub Bot logged work on BEAM-8275:
Author: ASF GitHub Bot
Created on: 27/Sep/19 18:21
Start Date: 27/Sep/19 18:21
Worklog Time Spent: 10m
Work Description: 11moon11 commented on pull request #9625: [BEAM-8275]
Beam SQL should support BigQuery in DIRECT_READ mode
URL: https://github.com/apache/beam/pull/9625#discussion_r329195834
##
File path:
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
##
@@ -154,6 +156,78 @@ public void testSQLRead() {
assertEquals(state, State.DONE);
}
+ @Test
+ public void testSQLRead_withDirectRead() {
Review comment:
Added an integration test for BigQuery read in EXPORT mode, fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 319722)
Time Spent: 1h 50m (was: 1h 40m)
> Beam SQL should support BigQuery in DIRECT_READ mode
>
>
> Key: BEAM-8275
> URL: https://issues.apache.org/jira/browse/BEAM-8275
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Kirill Kozlov
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode.
> We also need to support DIRECT_READ mode. The method should be configurable
> by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage
> of the DIRECT_READ features for filter and project push down.
> References:
> [https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api]
> [https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html]
> [https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[
https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=319721=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319721
]
ASF GitHub Bot logged work on BEAM-8275:
Author: ASF GitHub Bot
Created on: 27/Sep/19 18:20
Start Date: 27/Sep/19 18:20
Worklog Time Spent: 10m
Work Description: 11moon11 commented on pull request #9625: [BEAM-8275]
Beam SQL should support BigQuery in DIRECT_READ mode
URL: https://github.com/apache/beam/pull/9625#discussion_r329195599
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java
##
@@ -45,15 +49,45 @@
*/
@Experimental
class BigQueryTable extends BaseBeamTable implements Serializable {
+ @VisibleForTesting static final String METHOD_PROPERTY = "method";
@VisibleForTesting final String bqLocation;
private final ConversionOptions conversionOptions;
private BeamTableStatistics rowCountStatistics = null;
private static final Logger LOGGER =
LoggerFactory.getLogger(BigQueryTable.class);
+ @VisibleForTesting Method method;
Review comment:
Set Method to be final, fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 319721)
Time Spent: 1h 40m (was: 1.5h)
> Beam SQL should support BigQuery in DIRECT_READ mode
>
>
> Key: BEAM-8275
> URL: https://issues.apache.org/jira/browse/BEAM-8275
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Kirill Kozlov
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode.
> We also need to support DIRECT_READ mode. The method should be configurable
> by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage
> of the DIRECT_READ features for filter and project push down.
> References:
> [https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api]
> [https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html]
> [https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[
https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=319720=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319720
]
ASF GitHub Bot logged work on BEAM-8275:
Author: ASF GitHub Bot
Created on: 27/Sep/19 18:20
Start Date: 27/Sep/19 18:20
Worklog Time Spent: 10m
Work Description: 11moon11 commented on pull request #9625: [BEAM-8275]
Beam SQL should support BigQuery in DIRECT_READ mode
URL: https://github.com/apache/beam/pull/9625#discussion_r329195441
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java
##
@@ -45,15 +49,45 @@
*/
@Experimental
class BigQueryTable extends BaseBeamTable implements Serializable {
+ @VisibleForTesting static final String METHOD_PROPERTY = "method";
@VisibleForTesting final String bqLocation;
private final ConversionOptions conversionOptions;
private BeamTableStatistics rowCountStatistics = null;
private static final Logger LOGGER =
LoggerFactory.getLogger(BigQueryTable.class);
+ @VisibleForTesting Method method;
- BigQueryTable(Table table, BigQueryUtils.ConversionOptions options) {
+ BigQueryTable(Table table, BigQueryUtils.ConversionOptions options)
+ throws InvalidPropertyException {
Review comment:
Removed `throws RuntimeException`, fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 319720)
Time Spent: 1.5h (was: 1h 20m)
> Beam SQL should support BigQuery in DIRECT_READ mode
>
>
> Key: BEAM-8275
> URL: https://issues.apache.org/jira/browse/BEAM-8275
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Kirill Kozlov
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode.
> We also need to support DIRECT_READ mode. The method should be configurable
> by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage
> of the DIRECT_READ features for filter and project push down.
> References:
> [https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api]
> [https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html]
> [https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[
https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=319254=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319254
]
ASF GitHub Bot logged work on BEAM-8275:
Author: ASF GitHub Bot
Created on: 26/Sep/19 22:13
Start Date: 26/Sep/19 22:13
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #9625: [BEAM-8275]
Beam SQL should support BigQuery in DIRECT_READ mode
URL: https://github.com/apache/beam/pull/9625#discussion_r328848385
##
File path:
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryReadWriteIT.java
##
@@ -154,6 +156,78 @@ public void testSQLRead() {
assertEquals(state, State.DONE);
}
+ @Test
+ public void testSQLRead_withDirectRead() {
Review comment:
We should probably also add a test that EXPORT works, so when that is no
longer the default we will still have a test for it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 319254)
Time Spent: 1h 10m (was: 1h)
> Beam SQL should support BigQuery in DIRECT_READ mode
>
>
> Key: BEAM-8275
> URL: https://issues.apache.org/jira/browse/BEAM-8275
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Kirill Kozlov
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode.
> We also need to support DIRECT_READ mode. The method should be configurable
> by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage
> of the DIRECT_READ features for filter and project push down.
> References:
> [https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api]
> [https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html]
> [https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[
https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=319255=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319255
]
ASF GitHub Bot logged work on BEAM-8275:
Author: ASF GitHub Bot
Created on: 26/Sep/19 22:13
Start Date: 26/Sep/19 22:13
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #9625: [BEAM-8275]
Beam SQL should support BigQuery in DIRECT_READ mode
URL: https://github.com/apache/beam/pull/9625#discussion_r326830964
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java
##
@@ -45,15 +49,45 @@
*/
@Experimental
class BigQueryTable extends BaseBeamTable implements Serializable {
+ @VisibleForTesting static final String METHOD_PROPERTY = "method";
@VisibleForTesting final String bqLocation;
private final ConversionOptions conversionOptions;
private BeamTableStatistics rowCountStatistics = null;
private static final Logger LOGGER =
LoggerFactory.getLogger(BigQueryTable.class);
+ @VisibleForTesting Method method;
Review comment:
This is set exactly once (per code path) in the constructor, so you should
be able to make it `final`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 319255)
Time Spent: 1h 10m (was: 1h)
> Beam SQL should support BigQuery in DIRECT_READ mode
>
>
> Key: BEAM-8275
> URL: https://issues.apache.org/jira/browse/BEAM-8275
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Kirill Kozlov
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode.
> We also need to support DIRECT_READ mode. The method should be configurable
> by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage
> of the DIRECT_READ features for filter and project push down.
> References:
> [https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api]
> [https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html]
> [https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[
https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=319256=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319256
]
ASF GitHub Bot logged work on BEAM-8275:
Author: ASF GitHub Bot
Created on: 26/Sep/19 22:13
Start Date: 26/Sep/19 22:13
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #9625: [BEAM-8275]
Beam SQL should support BigQuery in DIRECT_READ mode
URL: https://github.com/apache/beam/pull/9625#discussion_r326830808
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java
##
@@ -45,15 +49,45 @@
*/
@Experimental
class BigQueryTable extends BaseBeamTable implements Serializable {
+ @VisibleForTesting static final String METHOD_PROPERTY = "method";
@VisibleForTesting final String bqLocation;
private final ConversionOptions conversionOptions;
private BeamTableStatistics rowCountStatistics = null;
private static final Logger LOGGER =
LoggerFactory.getLogger(BigQueryTable.class);
+ @VisibleForTesting Method method;
- BigQueryTable(Table table, BigQueryUtils.ConversionOptions options) {
+ BigQueryTable(Table table, BigQueryUtils.ConversionOptions options)
+ throws InvalidPropertyException {
Review comment:
This statement is redundant, please remove. `InvalidPropertyException`
indirectly extends `RuntimeException`. Every java method implicity `throws
RuntimeException`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 319256)
Time Spent: 1h 20m (was: 1h 10m)
> Beam SQL should support BigQuery in DIRECT_READ mode
>
>
> Key: BEAM-8275
> URL: https://issues.apache.org/jira/browse/BEAM-8275
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Kirill Kozlov
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode.
> We also need to support DIRECT_READ mode. The method should be configurable
> by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage
> of the DIRECT_READ features for filter and project push down.
> References:
> [https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api]
> [https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html]
> [https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[ https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=318496=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-318496 ] ASF GitHub Bot logged work on BEAM-8275: Author: ASF GitHub Bot Created on: 25/Sep/19 18:08 Start Date: 25/Sep/19 18:08 Worklog Time Spent: 10m Work Description: apilloud commented on issue #9625: [BEAM-8275] Beam SQL should support BigQuery in DIRECT_READ mode URL: https://github.com/apache/beam/pull/9625#issuecomment-535143660 Clicked the wrong button. Sorry Anton! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 318496) Time Spent: 1h (was: 50m) > Beam SQL should support BigQuery in DIRECT_READ mode > > > Key: BEAM-8275 > URL: https://issues.apache.org/jira/browse/BEAM-8275 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode. > We also need to support DIRECT_READ mode. The method should be configurable > by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage > of the DIRECT_READ features for filter and project push down. > References: > [https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api] > [https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html] > [https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[ https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=316014=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316014 ] ASF GitHub Bot logged work on BEAM-8275: Author: ASF GitHub Bot Created on: 21/Sep/19 00:54 Start Date: 21/Sep/19 00:54 Worklog Time Spent: 10m Work Description: amaliujia commented on issue #9625: [BEAM-8275] Beam SQL should support BigQuery in DIRECT_READ mode URL: https://github.com/apache/beam/pull/9625#issuecomment-533754059 Run SQL Postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 316014) Time Spent: 50m (was: 40m) > Beam SQL should support BigQuery in DIRECT_READ mode > > > Key: BEAM-8275 > URL: https://issues.apache.org/jira/browse/BEAM-8275 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode. > We also need to support DIRECT_READ mode. The method should be configurable > by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage > of the DIRECT_READ features for filter and project push down. > References: > [https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api] > [https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html] > [https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[ https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=315782=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-315782 ] ASF GitHub Bot logged work on BEAM-8275: Author: ASF GitHub Bot Created on: 20/Sep/19 16:33 Start Date: 20/Sep/19 16:33 Worklog Time Spent: 10m Work Description: 11moon11 commented on pull request #9625: [BEAM-8275] Beam SQL should support BigQuery in DIRECT_READ mode URL: https://github.com/apache/beam/pull/9625 Add a table property 'method' to BigQuery. When no property specified, use Method.DEFAULT. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[ https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=315784=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-315784 ] ASF GitHub Bot logged work on BEAM-8275: Author: ASF GitHub Bot Created on: 20/Sep/19 16:33 Start Date: 20/Sep/19 16:33 Worklog Time Spent: 10m Work Description: 11moon11 commented on issue #9625: [BEAM-8275] Beam SQL should support BigQuery in DIRECT_READ mode URL: https://github.com/apache/beam/pull/9625#issuecomment-533624666 R: @apilloud This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 315784) Time Spent: 40m (was: 0.5h) > Beam SQL should support BigQuery in DIRECT_READ mode > > > Key: BEAM-8275 > URL: https://issues.apache.org/jira/browse/BEAM-8275 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode. > We also need to support DIRECT_READ mode. The method should be configurable > by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage > of the DIRECT_READ features for filter and project push down. > References: > [https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api] > [https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html] > [https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[ https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=315361=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-315361 ] ASF GitHub Bot logged work on BEAM-8275: Author: ASF GitHub Bot Created on: 19/Sep/19 23:27 Start Date: 19/Sep/19 23:27 Worklog Time Spent: 10m Work Description: 11moon11 commented on pull request #9622: [BEAM-8275] Beam SQL should support BigQuery in DIRECT_READ mode URL: https://github.com/apache/beam/pull/9622 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 315361) Time Spent: 20m (was: 10m) > Beam SQL should support BigQuery in DIRECT_READ mode > > > Key: BEAM-8275 > URL: https://issues.apache.org/jira/browse/BEAM-8275 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Andrew Pilloud >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > SQL currently only supports reading from BigQuery in DEFAULT (EXPORT) mode. > We also need to support DIRECT_READ mode. The method should be configurable > by TBLPROPERTIES through the SQL CLI. This will enable us to take advantage > of the DIRECT_READ features for filter and project push down. > References: > [https://beam.apache.org/documentation/io/built-in/google-bigquery/#storage-api] > [https://beam.apache.org/blog/2019/06/04/adding-data-sources-to-sql.html] > [https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTable.java] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8275) Beam SQL should support BigQuery in DIRECT_READ mode
[ https://issues.apache.org/jira/browse/BEAM-8275?focusedWorklogId=315360=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-315360 ] ASF GitHub Bot logged work on BEAM-8275: Author: ASF GitHub Bot Created on: 19/Sep/19 23:26 Start Date: 19/Sep/19 23:26 Worklog Time Spent: 10m Work Description: 11moon11 commented on pull request #9622: [BEAM-8275] Beam SQL should support BigQuery in DIRECT_READ mode URL: https://github.com/apache/beam/pull/9622 Add support for switching between different modes for BigQuery Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[