[beam] branch master updated: Merge pull request #12597: [BEAM-10685] Added integration test for streaming for SnowflakeIO
This is an automated email from the ASF dual-hosted git repository.
kkucharczyk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2b2b8e7 Merge pull request #12597: [BEAM-10685] Added integration
test for streaming for SnowflakeIO
2b2b8e7 is described below
commit 2b2b8e71b7697b2933ddcef7e2454077684ddee8
Author: Kasia Kucharczyk <2536609+kkuch...@users.noreply.github.com>
AuthorDate: Wed Aug 19 15:48:41 2020 +0200
Merge pull request #12597: [BEAM-10685] Added integration test for
streaming for SnowflakeIO
---
sdks/java/io/snowflake/build.gradle| 1 +
.../io/snowflake/test/StreamingSnowflakeIOIT.java | 196 +
.../beam/sdk/io/snowflake/test/TestUtils.java | 18 ++
3 files changed, 215 insertions(+)
diff --git a/sdks/java/io/snowflake/build.gradle
b/sdks/java/io/snowflake/build.gradle
index ea8204d..f2b48fe 100644
--- a/sdks/java/io/snowflake/build.gradle
+++ b/sdks/java/io/snowflake/build.gradle
@@ -36,6 +36,7 @@ dependencies {
testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
testCompile project(path: ":sdks:java:io:common", configuration:
"testRuntime")
testCompile project(path: ":sdks:java:testing:test-utils", configuration:
"testRuntime")
+ testCompile 'com.google.cloud:google-cloud-storage:1.102.0'
testCompile library.java.avro
testCompile library.java.junit
testCompile library.java.hamcrest_core
diff --git
a/sdks/java/io/snowflake/src/test/java/org/apache/beam/sdk/io/snowflake/test/StreamingSnowflakeIOIT.java
b/sdks/java/io/snowflake/src/test/java/org/apache/beam/sdk/io/snowflake/test/StreamingSnowflakeIOIT.java
new file mode 100644
index 000..4584a99
--- /dev/null
+++
b/sdks/java/io/snowflake/src/test/java/org/apache/beam/sdk/io/snowflake/test/StreamingSnowflakeIOIT.java
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.snowflake.test;
+
+import static
org.apache.beam.sdk.io.snowflake.test.TestUtils.SnowflakeIOITPipelineOptions;
+import static
org.apache.beam.sdk.io.snowflake.test.TestUtils.getTestRowDataMapper;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists.newArrayList;
+import static
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets.newHashSet;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+import java.util.Set;
+import java.util.UUID;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.common.TestRow;
+import org.apache.beam.sdk.io.snowflake.SnowflakeIO;
+import
org.apache.beam.sdk.io.snowflake.credentials.SnowflakeCredentialsFactory;
+import org.apache.beam.sdk.io.snowflake.enums.StreamingLogLevel;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestStream;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+/**
+ * The test for streaming writing to SnowflakeIO. Because of using {@code
TestStream} it has to be
+ * run with DirectRunner. The test requires a Snowflake's snowpipe name with
the copy into
+ * STREAMING_IOIT table statement.
+ *
+ * Example run:
+ *
+ *
+ * ./gradlew --info -p sdks/java/io/snowflake integrationTest
-DintegrationTestPipelineOptions='[
+ * "--serverName=",
+ * "--username=",
+ * "--privateKeyPath=",
+ * "--privateKeyPassphrase=",
+ * "--database=",
+ * "--schema=",
+ * "--stagingBucketName=",
+ * "--storageIntegrationName=",
+ * "--snowPipe=",
+ * "--
[beam] branch master updated (3558c96 -> 86b8326)
This is an automated email from the ASF dual-hosted git repository.
kkucharczyk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 3558c96 Merge pull request #12149: [BEAM-9897] Add cross-language
support to SnowflakeIO.Read
add 86b8326 Merge pull request #12151: [BEAM-9896] Add streaming for
SnowflakeIO.Write to Java SDK
No new revisions were added by this update.
Summary of changes:
CHANGES.md | 1 +
sdks/java/io/snowflake/build.gradle| 1 +
.../apache/beam/sdk/io/snowflake/SnowflakeIO.java | 588 ++---
.../sdk/io/snowflake/SnowflakePipelineOptions.java | 9 +-
.../{package-info.java => StreamingLogLevel.java} | 7 +-
.../sdk/io/snowflake/services/ServiceConfig.java | 4 +
...onfig.java => SnowflakeBatchServiceConfig.java} | 67 ++-
...iceImpl.java => SnowflakeBatchServiceImpl.java} | 64 ++-
.../io/snowflake/services/SnowflakeService.java| 2 -
.../services/SnowflakeStreamingServiceConfig.java | 71 +++
.../services/SnowflakeStreamingServiceImpl.java| 72 +++
...mpl.java => FakeSnowflakeBatchServiceImpl.java} | 13 +-
...e-info.java => FakeSnowflakeIngestManager.java} | 19 +-
.../test/FakeSnowflakeStreamingServiceImpl.java| 73 +++
.../beam/sdk/io/snowflake/test/TestUtils.java | 4 +
.../test/unit/read/SnowflakeIOReadTest.java| 18 +-
.../test/unit/write/CreateDispositionTest.java | 4 +-
.../unit/write/QueryDispositionLocationTest.java | 6 +-
.../test/unit/write/SchemaDispositionTest.java | 4 +-
.../test/unit/write/SnowflakeIOWriteTest.java | 70 ++-
.../test/unit/write/StreamingWriteTest.java| 321 +++
21 files changed, 1283 insertions(+), 135 deletions(-)
copy
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/enums/{package-info.java
=> StreamingLogLevel.java} (94%)
rename
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/services/{SnowflakeServiceConfig.java
=> SnowflakeBatchServiceConfig.java} (60%)
rename
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/services/{SnowflakeServiceImpl.java
=> SnowflakeBatchServiceImpl.java} (79%)
create mode 100644
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/services/SnowflakeStreamingServiceConfig.java
create mode 100644
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/services/SnowflakeStreamingServiceImpl.java
rename
sdks/java/io/snowflake/src/test/java/org/apache/beam/sdk/io/snowflake/test/{FakeSnowflakeServiceImpl.java
=> FakeSnowflakeBatchServiceImpl.java} (89%)
copy
sdks/java/io/snowflake/src/test/java/org/apache/beam/sdk/io/snowflake/test/{package-info.java
=> FakeSnowflakeIngestManager.java} (70%)
create mode 100644
sdks/java/io/snowflake/src/test/java/org/apache/beam/sdk/io/snowflake/test/FakeSnowflakeStreamingServiceImpl.java
create mode 100644
sdks/java/io/snowflake/src/test/java/org/apache/beam/sdk/io/snowflake/test/unit/write/StreamingWriteTest.java
[beam] branch master updated (1e2b516 -> 3558c96)
This is an automated email from the ASF dual-hosted git repository.
kkucharczyk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 1e2b516 Merge pull request #12380 Cleanup WordCount example.
add 3558c96 Merge pull request #12149: [BEAM-9897] Add cross-language
support to SnowflakeIO.Read
No new revisions were added by this update.
Summary of changes:
CHANGES.md | 1 +
sdks/java/io/snowflake/build.gradle| 1 +
.../{ => snowflake}/expansion-service/build.gradle | 11 +-
.../credentials/SnowflakeCredentialsFactory.java | 50 +++--
.../io/snowflake/crosslanguage/Configuration.java | 130 +
.../crosslanguage/SnowflakeReadRegistrar.java | 90 +
.../io/snowflake/crosslanguage}/package-info.java | 9 +-
sdks/python/apache_beam/io/external/snowflake.py | 204 +
settings.gradle| 1 +
.../content/en/roadmap/connectors-multi-sdk.md | 1 +
10 files changed, 469 insertions(+), 29 deletions(-)
copy sdks/java/io/{ => snowflake}/expansion-service/build.gradle (78%)
create mode 100644
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/crosslanguage/Configuration.java
create mode 100644
sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/crosslanguage/SnowflakeReadRegistrar.java
copy sdks/java/{core/src/main/java/org/apache/beam/sdk/expansion =>
io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/crosslanguage}/package-info.java
(86%)
create mode 100644 sdks/python/apache_beam/io/external/snowflake.py
[beam] branch master updated (4cfd4bf -> 296650d)
This is an automated email from the ASF dual-hosted git repository. kkucharczyk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4cfd4bf Merge pull request #12137: [BEAM-10392] Attempt to fix/enable detection of rabbitmq:test stuckness. add 296650d Merge pull request #12146: [BEAM-10383] Update Snowflake JDBC dependency No new revisions were added by this update. Summary of changes: CHANGES.md | 2 +- sdks/java/io/snowflake/build.gradle| 2 +- .../apache/beam/sdk/io/snowflake/SnowflakeIO.java | 26 ++ .../test/unit/DataSourceConfigurationTest.java | 7 +++--- 4 files changed, 23 insertions(+), 14 deletions(-)
[beam] branch master updated: Merge pull request #11794: [BEAM-9894] Add batch SnowflakeIO.Write to Java SDK
This is an automated email from the ASF dual-hosted git repository.
kkucharczyk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 55ff8b3 Merge pull request #11794: [BEAM-9894] Add batch
SnowflakeIO.Write to Java SDK
55ff8b3 is described below
commit 55ff8b3939e2751056a40bd43fd9cf9114491664
Author: purbanow <37292156+purba...@users.noreply.github.com>
AuthorDate: Wed Jun 24 10:34:16 2020 +0200
Merge pull request #11794: [BEAM-9894] Add batch SnowflakeIO.Write to Java
SDK
* [BEAM-9894] Add batch SnowflakeIO.Write to Java SDK
* fix: add missing license headers
* refactor: method names
* fix: make Location.storageIntegrationName as a nullable variable
* fix: remove MapUserDataObjectsArrayFn class
* fix(SnowFlakeIO): removed Parse class + add @Experimental annotation
* refactor(SnowFlakeIO): removed Location class
* fix(SnowFlakeIO): added missing types
* refactor(SnowFlakeIO): removed CSVSink class
* [BEAM-9894] Snowflake write - change Combine step into Reify
Co-authored-by: Kasia Kucharczyk
---
CHANGES.md | 1 +
.../apache/beam/sdk/io/snowflake/SnowflakeIO.java | 493 +++--
.../sdk/io/snowflake/SnowflakePipelineOptions.java | 18 +-
.../sdk/io/snowflake/SnowflakeServiceImpl.java | 90
.../io/snowflake/{ => enums}/CloudProvider.java| 2 +-
.../WriteDisposition.java} | 19 +-
.../package-info.java} | 16 +-
.../ServiceConfig.java}| 16 +-
.../snowflake/{ => services}/SnowflakeService.java | 19 +-
.../snowflake/services/SnowflakeServiceConfig.java | 93
.../snowflake/services/SnowflakeServiceImpl.java | 208 +
.../package-info.java} | 16 +-
.../io/snowflake/test/FakeSnowflakeDatabase.java | 6 +-
.../snowflake/test/FakeSnowflakeServiceImpl.java | 73 ++-
.../beam/sdk/io/snowflake/test/TestUtils.java | 65 +++
...pelineOptions.java => TestPipelineOptions.java} | 2 +-
.../test/unit/read/SnowflakeIOReadTest.java| 62 ++-
.../unit/write/QueryDispositionLocationTest.java | 160 +++
.../test/unit/write/SnowflakeIOWriteTest.java | 172 +++
19 files changed, 1262 insertions(+), 269 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index 4d616b2..a00d8d2 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -91,6 +91,7 @@
* Basic Kafka read/write support for DataflowRunner (Python)
([BEAM-8019](https://issues.apache.org/jira/browse/BEAM-8019)).
* Sources and sinks for Google Healthcare APIs
(Java)([BEAM-9468](https://issues.apache.org/jira/browse/BEAM-9468)).
+* Support for writing to Snowflake added (Java)
([BEAM-9894](https://issues.apache.org/jira/browse/BEAM-9894)).
## New Features / Improvements
diff --git
a/sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
b/sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
index a67ba32..4895e4e 100644
---
a/sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
+++
b/sdks/java/io/snowflake/src/main/java/org/apache/beam/sdk/io/snowflake/SnowflakeIO.java
@@ -29,6 +29,7 @@ import java.security.PrivateKey;
import java.sql.Connection;
import java.sql.SQLException;
import java.text.SimpleDateFormat;
+import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.UUID;
@@ -37,26 +38,45 @@ import java.util.stream.Collectors;
import javax.annotation.Nullable;
import javax.sql.DataSource;
import net.snowflake.client.jdbc.SnowflakeBasicDataSource;
+import org.apache.beam.sdk.annotations.Experimental;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.io.Compression;
import org.apache.beam.sdk.io.FileIO;
import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.io.WriteFilesResult;
import org.apache.beam.sdk.io.fs.MoveOptions;
import org.apache.beam.sdk.io.fs.ResourceId;
import
org.apache.beam.sdk.io.snowflake.credentials.KeyPairSnowflakeCredentials;
import
org.apache.beam.sdk.io.snowflake.credentials.OAuthTokenSnowflakeCredentials;
import org.apache.beam.sdk.io.snowflake.credentials.SnowflakeCredentials;
import
org.apache.beam.sdk.io.snowflake.credentials.UsernamePasswordSnowflakeCredentials;
+import org.apache.beam.sdk.io.snowflake.enums.WriteDisposition;
+import org.apache.beam.sdk.io.snowflake.services.SnowflakeService;
+import org.apache.beam.sdk.io.snowflake.services.SnowflakeServiceConfig;
+import org.apache.beam.sdk.io.snowflake.services.SnowflakeServi
[beam] branch master updated (53e5cee -> 334682d)
This is an automated email from the ASF dual-hosted git repository. kkucharczyk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 53e5cee Merge pull request #11677: [BEAM-9963] Fix coder type in failing ParDo streaming load test add 334682d Merge pull request #11567: [BEAM-8132] Report Python metrics to InfluxDB No new revisions were added by this update. Summary of changes: .../apache_beam/testing/load_tests/load_test.py| 48 --- .../testing/load_tests/load_test_metrics_utils.py | 92 +- 2 files changed, 128 insertions(+), 12 deletions(-)
[beam] branch master updated (31a5a33 -> 1753b43)
This is an automated email from the ASF dual-hosted git repository.
kkucharczyk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 31a5a33 Merge pull request #10683 from
EDjur/BEAM-7810/valueprovider-namespace
add 801a6bb [BEAM-8941] Implement simple DSL for load tests
add 1753b43 Merge pull request #10543: [BEAM-8941] Implement simple DSL
for load tests
No new revisions were added by this update.
Summary of changes:
.test-infra/jenkins/LoadTestConfig.groovy | 622 +
.../job_LoadTests_Combine_Java_Smoke.groovy| 89 +++
groovy => job_LoadTests_GBK_Java_Smoke.groovy} | 6 +-
3 files changed, 714 insertions(+), 3 deletions(-)
create mode 100644 .test-infra/jenkins/LoadTestConfig.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_Combine_Java_Smoke.groovy
rename .test-infra/jenkins/{job_LoadTests_Java_Smoke.groovy =>
job_LoadTests_GBK_Java_Smoke.groovy} (97%)
[beam] branch master updated (ebe8bdc -> 4254673)
This is an automated email from the ASF dual-hosted git repository. kkucharczyk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ebe8bdc Merge pull request #10622: [BEAM=6857] Fix timermap test to not use TestStream add c188c15 Report status code 0 when no stale jobs are found add 4254673 Merge pull request #10639: [BEAM-8939] Report status code 0 when no stale jobs are found No new revisions were added by this update. Summary of changes: .test-infra/tools/stale_dataflow_jobs_cleaner.sh | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-)
[beam] branch master updated (4abc23a -> c22ba7e)
This is an automated email from the ASF dual-hosted git repository. kkucharczyk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4abc23a Merge pull request #10366: [BEAM-8956] Begin unifying contributor instructions into a single location add 3484558 [BEAM-7115] Fix metrics being incorrectly gathered add c22ba7e Merge pull request #10460: [BEAM-7115] Fix metrics being incorrectly gathered No new revisions were added by this update. Summary of changes: .../apache/beam/sdk/io/tfrecord/TFRecordIOIT.java | 103 - .../beam/sdk/testutils/metrics/MetricsReader.java | 18 2 files changed, 77 insertions(+), 44 deletions(-)
