[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders
[ https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16315412#comment-16315412 ] ASF GitHub Bot commented on BEAM-3008: -- dmytroivanov4206 opened a new pull request #4357: [BEAM-3008] Adds parameters templatization for Bigtable URL: https://github.com/apache/beam/pull/4357 * Adds ValueProvider for instanceId, projectId and tableId * Removes BigtableOptions deprecated methods * Updates existing unit tests Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [x] Each commit in the pull request should have a meaningful subject line and body. - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [x] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > BigtableIO should use ValueProviders > - > > Key: BEAM-3008 > URL: https://issues.apache.org/jira/browse/BEAM-3008 > Project: Beam > Issue Type: New Feature > Components: sdk-java-gcp >Reporter: Solomon Duskis >Assignee: Solomon Duskis > > [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO > templatization. This Issue is a request to get a fully featured template for > BigtableIO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders
[
https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297690#comment-16297690
]
ASF GitHub Bot commented on BEAM-3008:
--
chamikaramj closed pull request #4277: [BEAM-3008] Introduces BigtableConfig to
contain similar logic for Read and Write
URL: https://github.com/apache/beam/pull/4277
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java
new file mode 100644
index 000..ba633d0cdbb
--- /dev/null
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.io.gcp.bigtable;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import com.google.auto.value.AutoValue;
+import com.google.cloud.bigtable.config.BigtableOptions;
+import com.google.cloud.bigtable.config.CredentialOptions;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.MoreObjects;
+import java.io.Serializable;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+
+/**
+ * Configuration for a Cloud Bigtable client.
+ */
+@AutoValue
+abstract class BigtableConfig implements Serializable {
+
+ /**
+ * Returns the project id being written to.
+ */
+ @Nullable
+ abstract String getProjectId();
+
+ /**
+ * Returns the instance id being written to.
+ */
+ @Nullable
+ abstract String getInstanceId();
+
+ /**
+ * Returns the table being read from.
+ */
+ @Nullable
+ abstract String getTableId();
+
+ /**
+ * Returns the Google Cloud Bigtable instance being written to, and other
parameters.
+ *
+ * @deprecated will be replaced by bigtable options configurator.
+ */
+ @Deprecated
+ @Nullable
+ abstract BigtableOptions getBigtableOptions();
+
+ /**
+ * Configurator of the effective Bigtable Options.
+ */
+ @Nullable
+ abstract SerializableFunction getBigtableOptionsConfigurator();
+
+ /**
+ * Weather validate that table exists before writing.
+ */
+ abstract boolean getValidate();
+
+ /**
+ * {@link BigtableService} used only for testing.
+ */
+ @Nullable
+ abstract BigtableService getBigtableService();
+
+ abstract Builder toBuilder();
+
+ static BigtableConfig.Builder builder() {
+return new AutoValue_BigtableConfig.Builder();
+ }
+
+ @AutoValue.Builder
+ abstract static class Builder {
+
+abstract Builder setProjectId(String projectId);
+
+abstract Builder setInstanceId(String instanceId);
+
+abstract Builder setTableId(String tableId);
+
+/**
+ * @deprecated will be replaced by bigtable options configurator.
+ */
+@Deprecated
+abstract Builder setBigtableOptions(BigtableOptions options);
+
+abstract Builder setValidate(boolean validate);
+
+abstract Builder setBigtableOptionsConfigurator(
+ SerializableFunction
optionsConfigurator);
+
+abstract Builder setBigtableService(BigtableService bigtableService);
+
+abstract BigtableConfig build();
+ }
+
+ BigtableConfig withProjectId(String projectId) {
+checkNotNull(projectId, "Project Id of BigTable can not be null");
+return toBuilder().setProjectId(projectId).build();
+ }
+
+ BigtableConfig withInstanceId(String instanceId) {
+checkNotNull(instanceId, "Instance Id of BigTable can not be null");
+return toBuilder().setInstanceId(instanceId).build();
+
[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders
[ https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16294239#comment-16294239 ] ASF GitHub Bot commented on BEAM-3008: -- dmytroivanov4206 opened a new pull request #4277: [BEAM-3008] Introduces BigtableConfig to contain similar logic for Read and Write URL: https://github.com/apache/beam/pull/4277 Moves out common configuration parts from BigtableIO.Read and BigtableIO.Write to a BigtableConfig class. Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [x] Each commit in the pull request should have a meaningful subject line and body. - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [x] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > BigtableIO should use ValueProviders > - > > Key: BEAM-3008 > URL: https://issues.apache.org/jira/browse/BEAM-3008 > Project: Beam > Issue Type: New Feature > Components: sdk-java-gcp >Reporter: Solomon Duskis >Assignee: Solomon Duskis > > [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO > templatization. This Issue is a request to get a fully featured template for > BigtableIO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders
[
https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16286771#comment-16286771
]
ASF GitHub Bot commented on BEAM-3008:
--
chamikaramj closed pull request #4205: [BEAM-3008] Adds BigtableOptions
configurator to the BigtableIO
URL: https://github.com/apache/beam/pull/4205
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index 8b4609da224..febdc1f53b0 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -122,43 +122,19 @@
* idempotent transformation to that row.
*
* To configure a Cloud Bigtable sink, you must supply a table id, a
project id, an instance id
- * and optionally and optionally a {@link BigtableOptions} to provide more
specific connection
- * configuration, for example:
+ * and optionally a configuration function for {@link BigtableOptions} to
provide more specific
+ * connection configuration, for example:
*
* {@code
* PCollection>> data = ...;
*
* data.apply("write",
* BigtableIO.write()
- * .setProjectId("project")
- * .setInstanceId("instance")
+ * .withProjectId("project")
+ * .withInstanceId("instance")
* .withTableId("table"));
* }
*
- * Using local emulator
- *
- * In order to use local emulator for Bigtable you should use:
- *
- * {@code
- * BigtableOptions.Builder optionsBuilder =
- * new BigtableOptions.Builder()
- * .setUsePlaintextNegotiation(true)
- * .setCredentialOptions(CredentialOptions.nullCredential())
- * .setDataHost("127.0.0.1") // network interface where Bigtable
emulator is bound
- * .setInstanceAdminHost("127.0.0.1")
- * .setTableAdminHost("127.0.0.1")
- * .setPort(LOCAL_EMULATOR_PORT))
- *
- * PCollection>> data = ...;
- *
- * data.apply("write",
- * BigtableIO.write()
- * .withBigtableOptions(optionsBuilder)
- * .setProjectId("project")
- * .setInstanceId("instance")
- * .withTableId("table");
- * }
- *
* Experimental
*
* This connector for Cloud Bigtable is considered experimental and may
break or receive
@@ -239,12 +215,23 @@ public static Write write() {
@Nullable
abstract BigtableService getBigtableService();
-/** Returns the Google Cloud Bigtable instance being read from, and other
parameters. */
+/**
+ * Returns the Google Cloud Bigtable instance being read from, and other
parameters.
+ * @deprecated will be replaced by bigtable options configurator.
+ */
+@Deprecated
@Nullable
public abstract BigtableOptions getBigtableOptions();
public abstract boolean getValidate();
+/**
+ * Configurator of the effective Bigtable Options.
+ */
+@Nullable
+abstract SerializableFunction getBigtableOptionsConfigurator();
+
abstract Builder toBuilder();
@AutoValue.Builder
@@ -260,12 +247,17 @@ public static Write write() {
abstract Builder setTableId(String tableId);
+ /** @deprecated will be replaced by bigtable options configurator. */
+ @Deprecated
abstract Builder setBigtableOptions(BigtableOptions options);
abstract Builder setBigtableService(BigtableService bigtableService);
abstract Builder setValidate(boolean validate);
+ abstract Builder setBigtableOptionsConfigurator(
+SerializableFunction
optionsConfigurator);
+
abstract Read build();
}
@@ -302,7 +294,10 @@ public Read withInstanceId(String instanceId) {
* indicated by {@link #withProjectId(String)}, and using any other
specified customizations.
*
* Does not modify this object.
+ *
+ * @deprecated will be replaced by bigtable options configurator.
*/
+@Deprecated
public Read withBigtableOptions(BigtableOptions options) {
checkArgument(options != null, "options can not be null");
return withBigtableOptions(options.toBuilder());
@@ -320,17 +315,29 @@ public Read withBigtableOptions(BigtableOptions options) {
* will have no effect on the returned {@link BigtableIO.Read}.
*
* Does not modify this object.
+ *
+ * @deprecated will be replaced by bigtable options configurator.
*/
+@Deprecated
public Read withBigtableOptions(BigtableOptions.Builder optionsBuilder) {
[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders
[ https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275918#comment-16275918 ] ASF GitHub Bot commented on BEAM-3008: -- dmytroivanov4206 opened a new pull request #4205: [BEAM-3008] Adds BigtableOptions configurator to the BigtableIO URL: https://github.com/apache/beam/pull/4205 - Adds ability to provide configuration function for BigtableOptions. - Deprecates previous BigtableOptions providing methods. Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [x] Each commit in the pull request should have a meaningful subject line and body. - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [x] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > BigtableIO should use ValueProviders > - > > Key: BEAM-3008 > URL: https://issues.apache.org/jira/browse/BEAM-3008 > Project: Beam > Issue Type: New Feature > Components: sdk-java-gcp >Reporter: Solomon Duskis >Assignee: Solomon Duskis > > [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO > templatization. This Issue is a request to get a fully featured template for > BigtableIO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders
[
https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270091#comment-16270091
]
ASF GitHub Bot commented on BEAM-3008:
--
chamikaramj closed pull request #4171: [BEAM-3008] Extends API for BigtableIO
Read and Write by adding withInstanceId and withProjectId
URL: https://github.com/apache/beam/pull/4171
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index 29dc269f7fd..8f04c9dfdfd 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -78,38 +78,37 @@
* The Bigtable source returns a set of rows from a single table, returning
a
* {@code PCollection}.
*
- * To configure a Cloud Bigtable source, you must supply a table id and a
{@link BigtableOptions}
- * or builder configured with the project and other information necessary to
identify the
- * Bigtable instance. By default, {@link BigtableIO.Read} will read all rows
in the table. The row
- * range to be read can optionally be restricted using {@link
BigtableIO.Read#withKeyRange}, and
- * a {@link RowFilter} can be specified using {@link
BigtableIO.Read#withRowFilter}. For example:
+ * To configure a Cloud Bigtable source, you must supply a table id, a
project id, an instance
+ * id and optionally a {@link BigtableOptions} to provide more specific
connection configuration.
+ * By default, {@link BigtableIO.Read} will read all rows in the table. The
row range to be read
+ * can optionally be restricted using {@link BigtableIO.Read#withKeyRange},
and a {@link RowFilter}
+ * can be specified using {@link BigtableIO.Read#withRowFilter}. For example:
*
* {@code
- * BigtableOptions.Builder optionsBuilder =
- * new BigtableOptions.Builder()
- * .setProjectId("project")
- * .setInstanceId("instance");
*
* Pipeline p = ...;
*
* // Scan the entire table.
* p.apply("read",
* BigtableIO.read()
- * .withBigtableOptions(optionsBuilder)
+ * .withProjectId(projectId)
+ * .withInstanceId(instanceId)
* .withTableId("table"));
*
* // Scan a prefix of the table.
* ByteKeyRange keyRange = ...;
* p.apply("read",
* BigtableIO.read()
- * .withBigtableOptions(optionsBuilder)
+ * .withProjectId(projectId)
+ * .withInstanceId(instanceId)
* .withTableId("table")
* .withKeyRange(keyRange));
*
* // Scan a subset of rows that match the specified row filter.
* p.apply("filtered read",
* BigtableIO.read()
- * .withBigtableOptions(optionsBuilder)
+ * .withProjectId(projectId)
+ * .withInstanceId(instanceId)
* .withTableId("table")
* .withRowFilter(filter));
* }
@@ -121,21 +120,17 @@
* {@link ByteString} is the key of the row being mutated, and each {@link
Mutation} represents an
* idempotent transformation to that row.
*
- * To configure a Cloud Bigtable sink, you must supply a table id and a
{@link BigtableOptions}
- * or builder configured with the project and other information necessary to
identify the
- * Bigtable instance, for example:
+ * To configure a Cloud Bigtable sink, you must supply a table id, a
project id, an instance id
+ * and optionally and optionally a {@link BigtableOptions} to provide more
specific connection
+ * configuration, for example:
*
* {@code
- * BigtableOptions.Builder optionsBuilder =
- * new BigtableOptions.Builder()
- * .setProjectId("project")
- * .setInstanceId("instance");
- *
* PCollection>> data = ...;
*
* data.apply("write",
* BigtableIO.write()
- * .withBigtableOptions(optionsBuilder)
+ * .setProjectId("project")
+ * .setInstanceId("instance")
* .withTableId("table"));
* }
*
@@ -146,8 +141,6 @@
* {@code
* BigtableOptions.Builder optionsBuilder =
* new BigtableOptions.Builder()
- * .setProjectId("project")
- * .setInstanceId("instance")
* .setUsePlaintextNegotiation(true)
* .setCredentialOptions(CredentialOptions.nullCredential())
* .setDataHost("127.0.0.1") // network interface where Bigtable
emulator is bound
@@ -160,6 +153,8 @@
* data.apply("write",
* BigtableIO.write()
* .withBigtableOptions(optionsBuilder)
+ * .setProjectId("project")
+ *
[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders
[ https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269811#comment-16269811 ] ASF GitHub Bot commented on BEAM-3008: -- chamikaramj commented on issue #4171: [BEAM-3008] Extends API for BigtableIO Read and Write by adding withInstanceId and withProjectId URL: https://github.com/apache/beam/pull/4171#issuecomment-347711204 Thanks. LGTM. I'll merge. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > BigtableIO should use ValueProviders > - > > Key: BEAM-3008 > URL: https://issues.apache.org/jira/browse/BEAM-3008 > Project: Beam > Issue Type: New Feature > Components: sdk-java-gcp >Reporter: Solomon Duskis >Assignee: Solomon Duskis > > [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO > templatization. This Issue is a request to get a fully featured template for > BigtableIO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders
[ https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268634#comment-16268634 ] ASF GitHub Bot commented on BEAM-3008: -- sduskis commented on issue #4171: [BEAM-3008] Extends API for BigtableIO Read and Write by adding withInstanceId and withProjectId URL: https://github.com/apache/beam/pull/4171#issuecomment-347502089 @chamikaramj This change looks good to me. The Jenkins failure seems to be related to a completely different part of the Beam API. What do we need to do to move this forward? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > BigtableIO should use ValueProviders > - > > Key: BEAM-3008 > URL: https://issues.apache.org/jira/browse/BEAM-3008 > Project: Beam > Issue Type: New Feature > Components: sdk-java-gcp >Reporter: Solomon Duskis >Assignee: Solomon Duskis > > [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO > templatization. This Issue is a request to get a fully featured template for > BigtableIO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders
[
https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16265314#comment-16265314
]
ASF GitHub Bot commented on BEAM-3008:
--
sduskis commented on a change in pull request #4171: [BEAM-3008] Extends API
for BigtableIO Read and Write by adding withInstanceId and withProjectId
URL: https://github.com/apache/beam/pull/4171#discussion_r152978149
##
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
##
@@ -78,38 +78,38 @@
* The Bigtable source returns a set of rows from a single table, returning
a
* {@code PCollection}.
*
- * To configure a Cloud Bigtable source, you must supply a table id and a
{@link BigtableOptions}
- * or builder configured with the project and other information necessary to
identify the
- * Bigtable instance. By default, {@link BigtableIO.Read} will read all rows
in the table. The row
- * range to be read can optionally be restricted using {@link
BigtableIO.Read#withKeyRange}, and
- * a {@link RowFilter} can be specified using {@link
BigtableIO.Read#withRowFilter}. For example:
+ * To configure a Cloud Bigtable source, you must supply a table id, a
project id, an instance
+ * id and optionally a {@link BigtableOptions} to provide more specific
connection configuration.
+ * By default, {@link BigtableIO.Read} will read all rows in the table. The
row range to be read
+ * can optionally be restricted using {@link BigtableIO.Read#withKeyRange},
and a {@link RowFilter}
+ * can be specified using {@link BigtableIO.Read#withRowFilter}. For example:
*
* {@code
- * BigtableOptions.Builder optionsBuilder =
- * new BigtableOptions.Builder()
- * .setProjectId("project")
- * .setInstanceId("instance");
*
* Pipeline p = ...;
*
* // Scan the entire table.
* p.apply("read",
* BigtableIO.read()
* .withBigtableOptions(optionsBuilder)
Review comment:
Can you please remove `.withBigtableOptions(optionsBuilder)` for this
example?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> BigtableIO should use ValueProviders
> -
>
> Key: BEAM-3008
> URL: https://issues.apache.org/jira/browse/BEAM-3008
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-gcp
>Reporter: Solomon Duskis
>Assignee: Solomon Duskis
>
> [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO
> templatization. This Issue is a request to get a fully featured template for
> BigtableIO.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders
[ https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16265065#comment-16265065 ] ASF GitHub Bot commented on BEAM-3008: -- GitHub user dmytroivanov4206 opened a pull request: https://github.com/apache/beam/pull/4171 [BEAM-3008] Extends API for BigtableIO Read and Write by adding withInstanceId and withProjectId Adds withInstanceId and withProjectId to the BigtableIO Read and Write classes, first out of four steps to fix [BEAM-3008] bug. Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [x] Each commit in the pull request should have a meaningful subject line and body. - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [x] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/dmytroivanov4206/beam master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/4171.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4171 commit 28637766921200f03313088bfc0271453c0e3ae7 Author: Dmytro Ivanov Date: 2017-11-24T09:18:14Z Adds withInstanceId and withProjectId to the BigtableIO Read and Write classes, first out of four steps to fix [BEAM-3008] bug. > BigtableIO should use ValueProviders > - > > Key: BEAM-3008 > URL: https://issues.apache.org/jira/browse/BEAM-3008 > Project: Beam > Issue Type: New Feature > Components: sdk-java-gcp >Reporter: Solomon Duskis >Assignee: Solomon Duskis > > [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO > templatization. This Issue is a request to get a fully featured template for > BigtableIO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders
[ https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16265064#comment-16265064 ] ASF GitHub Bot commented on BEAM-3008: -- dmytroivanov4206 opened a new pull request #4171: [BEAM-3008] Extends API for BigtableIO Read and Write by adding withInstanceId and withProjectId URL: https://github.com/apache/beam/pull/4171 Adds withInstanceId and withProjectId to the BigtableIO Read and Write classes, first out of four steps to fix [BEAM-3008] bug. Follow this checklist to help us incorporate your contribution quickly and easily: - [x] Make sure there is a [JIRA issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [x] Each commit in the pull request should have a meaningful subject line and body. - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue. - [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. - [x] Run `mvn clean verify` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > BigtableIO should use ValueProviders > - > > Key: BEAM-3008 > URL: https://issues.apache.org/jira/browse/BEAM-3008 > Project: Beam > Issue Type: New Feature > Components: sdk-java-gcp >Reporter: Solomon Duskis >Assignee: Solomon Duskis > > [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO > templatization. This Issue is a request to get a fully featured template for > BigtableIO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders
[ https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16188971#comment-16188971 ] Solomon Duskis commented on BEAM-3008: -- Cloud Bigtable is a constrained problem. Writes needs a Cloud project id, instance id and table name. Reads also need a scan. HBaseIO currently takes in a Configuration object. If there's a small set of HBase configuration key/value pairs then it's absolutely makes sense to have HBase specific configuration options. HBaseIO and CloudBigtable need different configuration options. I think that we can create an AbstractHBaseIO that defers Connection creation to a child which would have the more specific configuration options. > BigtableIO should use ValueProviders > - > > Key: BEAM-3008 > URL: https://issues.apache.org/jira/browse/BEAM-3008 > Project: Beam > Issue Type: New Feature > Components: sdk-java-gcp >Reporter: Solomon Duskis >Assignee: Solomon Duskis > > [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO > templatization. This Issue is a request to get a fully featured template for > BigtableIO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-3008) BigtableIO should use ValueProviders
[ https://issues.apache.org/jira/browse/BEAM-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16188805#comment-16188805 ] Ismaël Mejía commented on BEAM-3008: [~sduskis] Do you think is worth to do this too for HBaseIO (for eventual uniformity with the in-progress Cloud Bigtable HBase connector). > BigtableIO should use ValueProviders > - > > Key: BEAM-3008 > URL: https://issues.apache.org/jira/browse/BEAM-3008 > Project: Beam > Issue Type: New Feature > Components: sdk-java-gcp >Reporter: Solomon Duskis >Assignee: Solomon Duskis > > [https://github.com/apache/beam/pull/2057] is an effort towards BigtableIO > templatization. This Issue is a request to get a fully featured template for > BigtableIO. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
