[jira] [Resolved] (BEAM-3157) BeamSql transform should support other PCollection types

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-3157.

   Resolution: Fixed
Fix Version/s: 2.4.0

> BeamSql transform should support other PCollection types
> 
>
> Key: BEAM-3157
> URL: https://issues.apache.org/jira/browse/BEAM-3157
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Ismaël Mejía
>Assignee: Anton Kedin
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> Currently the Beam SQL transform only supports input and output data 
> represented as a BeamRecord. This seems to me like an usability limitation 
> (even if we can do a ParDo to prepare objects before and after the transform).
> I suppose this constraint comes from the fact that we need to map 
> name/type/value from an object field into Calcite so it is convenient to have 
> a specific data type (BeamRecord) for this. However we can accomplish the 
> same by using a PCollection of JavaBean (where we know the same information 
> via the field names/types/values) or by using Avro records where we also have 
> the Schema information. For the output PCollection we can map the object via 
> a Reference (e.g. a JavaBean to be filled with the names of an Avro object).
> Note: I am assuming for the moment simple mappings since the SQL does not 
> support composite types for the moment.
> A simple API idea would be something like this:
> A simple filter:
> PCollection col = BeamSql.query("SELECT * FROM  WHERE 
> ...").from(MyPojo.class);
> A projection:
> PCollection newCol = BeamSql.query("SELECT id, 
> name").from(MyPojo.class).as(MyNewPojo.class);
> A first approach could be to just add the extra ParDos + transform DoFns 
> however I suppose that for memory use reasons maybe mapping directly into 
> Calcite would make sense.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4204: [BEAM-3157] Generate BeamRecord types from Pojos

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6792846069763770bbfd92a5eb61e64968522967
Merge: f2c87bd f9f70e2
Author: Ismaël Mejía 
AuthorDate: Thu Feb 8 08:17:22 2018 +0100

Merge pull request #4204: [BEAM-3157] Generate BeamRecord types from Pojos

[BEAM-3157] Generate BeamRecord types from Pojos

 .../beam/sdk/values/reflect/ByteBuddyUtils.java| 109 
 .../sdk/values/reflect/DefaultRowTypeFactory.java  | 100 +++
 .../beam/sdk/values/reflect/FieldValueGetter.java  |  36 ++
 .../sdk/values/reflect/GeneratedGetterFactory.java | 106 +++
 .../beam/sdk/values/reflect/GetterFactory.java |  32 +
 .../beam/sdk/values/reflect/ReflectionGetter.java  |  59 +
 .../values/reflect/ReflectionGetterFactory.java|  55 
 .../beam/sdk/values/reflect/ReflectionUtils.java   |  73 +++
 .../apache/beam/sdk/values/reflect/RowFactory.java | 124 ++
 .../beam/sdk/values/reflect/RowTypeFactory.java|  43 +++
 .../beam/sdk/values/reflect/RowTypeGetters.java|  54 
 .../beam/sdk/values/reflect/package-info.java  |  22 
 .../values/reflect/DefaultRowTypeFactoryTest.java  | 125 ++
 .../values/reflect/GeneratedGetterFactoryTest.java | 107 
 .../reflect/ReflectionGetterFactoryTest.java   | 107 
 .../sdk/values/reflect/ReflectionGetterTest.java   |  95 ++
 .../beam/sdk/values/reflect/RowFactoryTest.java| 142 +
 .../sdk/values/reflect/RowTypeGettersTest.java |  43 +++
 .../apache/beam/sdk/extensions/sql/RowSqlType.java |   5 +-
 .../beam/sdk/extensions/sql/SqlRowTypeFactory.java |  90 +
 .../sdk/extensions/sql/SqlRowTypeFactoryTest.java  | 133 +++
 21 files changed, 1658 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[beam] branch master updated (f2c87bd -> 6792846)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f2c87bd  Merge pull request #4635 from apache/revert-4139-depsupdate
 add f9f70e2  [Schema Generation] Generate BeamRecordTypes based on pojos.
 new 6792846  Merge pull request #4204: [BEAM-3157] Generate BeamRecord 
types from Pojos

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/values/reflect/ByteBuddyUtils.java| 109 
 .../sdk/values/reflect/DefaultRowTypeFactory.java  | 100 +++
 .../reflect/FieldValueGetter.java} |  19 +--
 .../sdk/values/reflect/GeneratedGetterFactory.java | 106 +++
 .../beam/sdk/values/reflect/GetterFactory.java |  15 ++-
 .../beam/sdk/values/reflect/ReflectionGetter.java  |  59 +
 .../values/reflect/ReflectionGetterFactory.java|  55 
 .../beam/sdk/values/reflect/ReflectionUtils.java   |  73 +++
 .../apache/beam/sdk/values/reflect/RowFactory.java | 124 ++
 .../reflect/RowTypeFactory.java}   |  23 ++--
 .../beam/sdk/values/reflect/RowTypeGetters.java}   |  28 +++-
 .../beam/sdk/values/reflect}/package-info.java |   4 +-
 .../values/reflect/DefaultRowTypeFactoryTest.java  | 125 ++
 .../values/reflect/GeneratedGetterFactoryTest.java | 107 
 .../reflect/ReflectionGetterFactoryTest.java   | 107 
 .../sdk/values/reflect/ReflectionGetterTest.java   |  95 ++
 .../beam/sdk/values/reflect/RowFactoryTest.java| 142 +
 .../sdk/values/reflect/RowTypeGettersTest.java}|  25 ++--
 .../apache/beam/sdk/extensions/sql/RowSqlType.java |   5 +-
 .../beam/sdk/extensions/sql/SqlRowTypeFactory.java |  90 +
 .../sdk/extensions/sql/SqlRowTypeFactoryTest.java  | 133 +++
 21 files changed, 1496 insertions(+), 48 deletions(-)
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/reflect/ByteBuddyUtils.java
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/reflect/DefaultRowTypeFactory.java
 copy 
sdks/java/core/src/main/java/org/apache/beam/sdk/{runners/PTransformMatcher.java
 => values/reflect/FieldValueGetter.java} (70%)
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/reflect/GeneratedGetterFactory.java
 copy 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/logging/LogWriter.java
 => 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/reflect/GetterFactory.java
 (72%)
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/reflect/ReflectionGetter.java
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/reflect/ReflectionGetterFactory.java
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/reflect/ReflectionUtils.java
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/reflect/RowFactory.java
 copy 
sdks/java/core/src/main/java/org/apache/beam/sdk/{transforms/Materialization.java
 => values/reflect/RowTypeFactory.java} (59%)
 copy 
sdks/java/{nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/sql/adapter/ModelFieldsAdapter.java
 => core/src/main/java/org/apache/beam/sdk/values/reflect/RowTypeGetters.java} 
(56%)
 copy 
{runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils
 => 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/reflect}/package-info.java
 (89%)
 create mode 100644 
sdks/java/core/src/test/java/org/apache/beam/sdk/values/reflect/DefaultRowTypeFactoryTest.java
 create mode 100644 
sdks/java/core/src/test/java/org/apache/beam/sdk/values/reflect/GeneratedGetterFactoryTest.java
 create mode 100644 
sdks/java/core/src/test/java/org/apache/beam/sdk/values/reflect/ReflectionGetterFactoryTest.java
 create mode 100644 
sdks/java/core/src/test/java/org/apache/beam/sdk/values/reflect/ReflectionGetterTest.java
 create mode 100644 
sdks/java/core/src/test/java/org/apache/beam/sdk/values/reflect/RowFactoryTest.java
 copy 
sdks/java/{nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/sql/adapter/ModelFieldsAdapter.java
 => 
core/src/test/java/org/apache/beam/sdk/values/reflect/RowTypeGettersTest.java} 
(60%)
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SqlRowTypeFactory.java
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/SqlRowTypeFactoryTest.java

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #118

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3457] Add Go Gradle precommit

[ccy] Update snippets to fix pickling and clarify encoding issues

[github] Revert "Update cloud spanner library to 0.29.0"

--
[...truncated 733.96 KB...]
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.4.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev355-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.1 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #115

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3457] Add Go Gradle precommit

[ccy] Update snippets to fix pickling and clarify encoding issues

[github] Revert "Update cloud spanner library to 0.29.0"

--
[...truncated 15.15 KB...]
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: monotonic>=0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth<2.0.0dev,>=0.4.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth-httplib2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-gax<0.16dev,>=0.15.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: 
proto-google-cloud-pubsub-v1[grpc]<0.16dev,>=0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: cachetools>=2.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: hdfs, apache-beam
  Running setup.py develop for apache-beam
Successfully installed apache-beam hdfs-2.1.0
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6970693285028277502.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.tfrecordioit_pkb_results 
--temp_dir=
 --official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 
--beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java 
--beam_it_module=sdks/java/io/file-based-io-tests 
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT 
'--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/115/]'
 '--beam_extra_mvn_properties=[filesystem=gcs]'
2018-02-08 06:00:44,139 b723fae6 MainThread INFO Verbose logging to: 

2018-02-08 06:00:44,140 b723fae6 MainThread INFO PerfKitBenchmarker 
version: v1.12.0-260-ge324e61
2018-02-08 06:00:44,141 b723fae6 MainThread INFO Flag values:
--beam_extra_mvn_properties=[filesystem=gcs]
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT
--beam_it_timeout=1200
--beam_sdk=java
--temp_dir=
--maven_binary=/home/jenkins/tools/maven/latest/bin/mvn
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/115/]
--beam_prebuilt
--project=apache-beam-testing
--bigquery_table=beam_performance.tfrecordioit_pkb_results
--official
--beam_it_module=sdks/java/io/file-based-io-tests
--dpb_log_level=INFO
--beam_it_profile=io-it
--benchmarks=beam_integration_benchmark
2018-02-08 06:00:44,434 b723fae6 MainThread WARNING  The key "flags" was not in 
the default config, but was in user overrides. This may indicate a typo.
2018-02-08 06:00:44,434 

Build failed in Jenkins: beam_PerformanceTests_Python #888

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3457] Add Go Gradle precommit

[ccy] Update snippets to fix pickling and clarify encoding issues

[github] Revert "Update cloud spanner library to 0.29.0"

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision f2c87bd385da2399938d6ff414176ea953c3c776 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f2c87bd385da2399938d6ff414176ea953c3c776
Commit message: "Merge pull request #4635 from apache/revert-4139-depsupdate"
 > git rev-list 6745a6119d609b4008e961912e84e932915bdb13 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5584927527848293471.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins265952456903428743.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3211178513825076765.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5811333024766613564.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4987289582997839078.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins975890875177527881.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 

svn commit: r24816 - in /dev/beam/2.3.0: apache-beam-2.3.0-python.zip.asc apache-beam-2.3.0-source-release.zip apache-beam-2.3.0-source-release.zip.asc apache-beam-2.3.0-source-release.zip.md5 apache-

[jbonofre]

Author: jbonofre
Date: Thu Feb  8 05:30:52 2018
New Revision: 24816

Log:
Update Apache Beam 2.3.0 release

Modified:
dev/beam/2.3.0/apache-beam-2.3.0-python.zip.asc
dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip
dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.asc
dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.md5
dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.sha1

Modified: dev/beam/2.3.0/apache-beam-2.3.0-python.zip.asc
==
--- dev/beam/2.3.0/apache-beam-2.3.0-python.zip.asc (original)
+++ dev/beam/2.3.0/apache-beam-2.3.0-python.zip.asc Thu Feb  8 05:30:52 2018
@@ -1,17 +1,17 @@
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1
 
-iQIcBAABAgAGBQJacBqOAAoJEL/y7kLIKC52PNoP/Av1an0rxIDdOxWQlul0suco
-KSTEo61MDRwAahAlwVjM9DsWdDJg/Ont6zvyAMfZQOga8mxHbzhCOCAh0q4XUbW/
-FdFJRUd9/NHdbLqsDAVmcip+S7XcQYSMXlCOlqxrCOmrKvrCgqXqSm36+e4beaE2
-R4GTaoCGGjcwP7+l90bd1R8mrLC3F6SavatoHRPZqhRHXtn4cZD0PxgqN0tjG6x/
-j4nNkCq0iNhm4mD+z6l53KICtlv+A+keX5BTA/5xGjmrnLSEV0lOUytYUp+n3TbW
-4ygv9V+yhG6YaTrq40Dtbjb7iB4vbqKL5wov5ykAL8RMvZqT6MEL/Df7dqBY4Jck
-pJJFhx4CL9MiD77VuhXI37na6auE9mRmYqSYdAY0M0qW239099JuyrWJMCzGe2N4
-41vCR1HYd2SWTIGvHiO7AL8LnJjnv+2oXnhPXYVEamVC8p6Ej7KBhMSEUbY5X3Iv
-yYVqhYbrHYCYlzEIhldoZe3ibCgHyn8XApidB3/akYUYgszbgNFeZdZuR6xbwXh3
-BHWodw/PM+WKAsO2pi8N0HaOpbeqx7NZgL1pwohLvnmxraUC6XyoAME6haBFHCL1
-iQNBSUu7cDQBNnOLRzOKb+F9RXwZSYzGFxhGwEHsCzxtlM4wsPDW6jgVhgjg/QjO
-LRKE7ycALitN4GygI94R
-=9325
+iQIcBAABAgAGBQJae+A7AAoJEL/y7kLIKC52X58QAJkn6JwZD7je+NfPI0feP7Jj
+Cg+BIZ0il6STZfJKO2JfFGyMzr2Djq2iXjJcvXaNMTfltMdVCIhe9HnkGhit70fI
+mErQlC1I4qJcTmOTtAgda0KpfXSK6lf2CFfguThcAjWj+mMlhnXb9BMptlxrOGQl
+J0vfffbBeeXcnSmQaetnBOZJogmcOsBDCgmqi5gB8j6GTtBFdU+7g0JhTDvGSGyI
+AvyYxoQiP9hhvV/e2lUee95iY4rJEKymJBfQHwWJBjr5zreqjaXd1hCv2/4cyYGd
+hm2x/JQAO1XsMk4OM6Qi11QTyKWG7x6GSqrTnm8YMeCDvqBMp05Bhm2WWVq2UvgM
+gqonrqZncsBu4/mGzYr2aTMGBByWDuN9SUFwtZs59dvKtwIFX3fld7/ebHyCc8CO
+3RQkaCUM0QKoq6IFArZBvZHFdUP1zBgFXtwGrsxQUY639ljaExnudf6X2fSI/mfD
+PcksRlZu4EQ6v12krLW+L/SFsISIu6nHWLQwTcHlr4OZGyshFvneFT3EBxm46s4A
+o6xivYDEM2X7+wBrxtYKWymbez7NZSgLz6Pc3B/6860B+RwHJjWmVNfHCr8XSBOp
+rnVIMI3clQ0eCz377VHBOVBXhU0rrCCmjg3XfRVJ5L6QMzgK8PUgus7nMMAhoaw0
+rq009gMzCyTxQSihp6Dg
+=il66
 -END PGP SIGNATURE-

Modified: dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip
==
Binary files - no diff available.

Modified: dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.asc
==
--- dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.asc (original)
+++ dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.asc Thu Feb  8 05:30:52 
2018
@@ -1,17 +1,17 @@
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1
 
-iQIcBAABAgAGBQJab4dwAAoJEL/y7kLIKC52xBoQAIOOKbc/fO3tr9DvYAfB5ujj
-1Tg8++7w5YF/8+SrMmtPeceDLNNxsvc1IMZYbU27TRhfMRE4oOIWUfJ09pR1rpuy
-xOwIQKC/A3GncutlL4weSi3lGLp+lhsY/k+xL8a9suH9aL0Dyx+e/gEwNZugJph9
-A4uJ0RqAHPAK2JTd3htE6HNbZWbHoRz3xv0t/Fq9cQQWGer+hmJjd7x0RID8z/00
-kouCYNWDLcBsp7clD2LEwDYC7JRTWxZN4bNoWEkNHmqgb3/iUNy6mMPinp8yLSWq
-thUuE7p/J+WOaZ8yLHVRHTBQp3VsQF+QSxqVyh9HNmdFbrO9q102WulShHMkiHMZ
-DUKyuxImuzlioQZ+jAbEy1GnbBTa7eSaK8y9dypFwOY+mQkLrBmXv0eMzw9mPQa2
-+JOS1Sfah74vnz+sDnIociHWvEmRndzpz3i1sJcTHWgg3tiCo3TWoMKuhmMTEZeT
-b3Vsiquq2Znb4SusjFoXJiP8wJgnP/oWn8CgSkaUT/lEWtj2OFiUWJ8oCmjopYM0
-564ny0G+F0w99Zr7aJ5xFHh4ZTFE1gLWlM1LQRXMMyuvp633d+eV/WrI37EdtJhM
-Zda8SyWwYGVVoAGv1LtfZ5pG3PVjJK6wcysiCUNGzL5X1ij8sXJIoWtvO7kC/N8N
-bMmP6sreht0Mc4LOBKAa
-=0VHp
+iQIcBAABAgAGBQJae1SfAAoJEL/y7kLIKC52+20QAJkebTvBEu+IUA7oo2h4KakS
+rdtofRT0E/lNB2i09jya8qPuYnLIbKcZZAl2AlFxQ6ngzanXdi1QBaMkpYEsYP0L
+AP25F6QnnuilhOBIv5S80bxa47QQsd498UMvBnwiWXIWw/ZaBfgpMLNL71+OUOGc
+rqMaXyCbvwr45hoIAy3r1OHCDwYm58iAcuVUEyNTaAHlxvYXPXTsOS7kAGCTQANg
+W1v04jXQcenxZNrb+D4oFNs6nHCT1yxPBgU1mEk7u/RRRdw/IFf9xG0AADytzupc
+v7F/Ssa4QjpUoRIvPrmYWF4PAF6b+AOAfqah/kJV8fxrqtlsbw8Qum74ekAmCcIE
+/kIhct7ezGiEDrGw6qWFLzfFH/CD4dii7v+Ut0prBSJAXcCmcLqrg0zCujRbCWuz
+LPdC5lc7KLgNN7vw7yH45jQ26zCe0YJvDYUNJmWM1UAAiwieqMIwSPilMhaF+Pd/
+ykROjtFrjgAiwBbJ7MOSgLut2HperpeDqKpLCcAI0pngKIQSSEywEMxbWbSltXIz
+Q81PRffBA7SOkmIuVT1qMQjR4Ie8saJxflJc8JPwJlXUr0Tc7afA+3Zov9+VMERo
+XGsH0QOLYQoQwT1109xI8PzRgdCdRTpOe4FOgRtdxAZ6+zhEKkS046PGNNR6E/Fv
+ymytpT6yvDzQ9mzQ5kuL
+=/d2I
 -END PGP SIGNATURE-

Modified: dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.md5
==
--- dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.md5 (original)
+++ dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.md5 Thu Feb  8 05:30:52 
2018
@@ -1 +1 @@
-80b2c42a5588da7e7f000150611ec85e  apache-beam-2.3.0-source-release.zip
+5288ab03de17ff82ed0d80ffd41fae9b  apache-beam-2.3.0-source-release.zip

Modified: dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.sha1
==
--- 

[jira] [Assigned] (BEAM-1939) Serialize more coders via URN + Class name

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-1939:
-

Assignee: (was: Thomas Groh)

> Serialize more coders via URN + Class name
> --
>
> Key: BEAM-1939
> URL: https://issues.apache.org/jira/browse/BEAM-1939
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Priority: Trivial
>
> If the size of serializing Standard Coders becomes too large, an arbitrary 
> Standard Coder can be encoded, alongside its components, via an URN and 
> looking up the class when it is to be deserialized.
> See 
> https://github.com/tgroh/beam/commit/070854845346d8e4df824e4aa374688bd095c2c6 
> as an example



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3622) DirectRunner memory issue with Python SDK

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-3622:
--
Component/s: (was: runner-direct)

> DirectRunner memory issue with Python SDK
> -
>
> Key: BEAM-3622
> URL: https://issues.apache.org/jira/browse/BEAM-3622
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: yuri krnr
>Assignee: Thomas Groh
>Priority: Major
>
> After running pipeline for a while in a streaming mode (reading from Pub/Sub 
> and writing to BigQuery, Datastore and another Pub/Sub) I noticed drastic 
> memory usage of a process. Using guppy as a profiler I got the following 
> results:
> start
> {noformat}
>  INFO *** MemoryReport Heap:
>  Partition of a set of 240208 objects. Total size = 34988840 bytes.
>  Index  Count   % Size   % Cumulative  % Kind (class / dict of class)
>  0  88289  37  8696984  25   8696984  25 str
>  1  5  22  4897352  14  13594336  39 tuple
>  2   5083   2  2790664   8  16385000  47 dict (no owner)
>  3   1939   1  1749656   5  18134656  52 type
>  4699   0  1723272   5  19857928  57 dict of module
>  5  12337   5  1579136   5  21437064  61 types.CodeType
>  6  12403   5  1488360   4  22925424  66 function
>  7   1939   1  1452616   4  24378040  70 dict of type
>  8677   0   709496   2  25087536  72 dict of 0x1e4d880
>  9  25603  11   614472   2  25702008  73 int
> <1103 more rows. Type e.g. '_.more' to view.>
> {noformat}
> after several hours of running
> {noformat}
> INFO *** MemoryReport Heap:
>  Partition of a set of 1255662 objects. Total size = 315029632 bytes.
>  Index  Count   % Size   % Cumulative  % Kind (class / dict of class)
>  0  95554   8 99755056  32  99755056  32 dict of
>  
> apache_beam.runners.direct.bundle_factory._Bundle
>  1 117943   9 54193192  17 153948248  49 dict (no owner)
>  2 161068  13 27169296   9 181117544  57 unicode
>  3  94571   8 26479880   8 207597424  66 dict of apache_beam.pvalue.PBegin
>  4 126461  10 12715336   4 220312760  70 str
>  5  44374   4 12424720   4 232737480  74 dict of 
> apitools.base.protorpclite.messages.FieldList
>  6  44374   4  6348624   2 239086104  76 
> apitools.base.protorpclite.messages.FieldList
>  7  95556   8  6115584   2 245201688  78 
> apache_beam.runners.direct.bundle_factory._Bundle
>  8  94571   8  6052544   2 251254232  80 apache_beam.pvalue.PBegin
>  9  57371   5  5218424   2 256472656  81 tuple
> <1187 more rows. Type e.g. '_.more' to view.>
> {noformat}
>  
> I see that every bundle still sits in memory and all its data too. why aren't 
> the gc-ed?
> What is the policy for gc for the dataflow processes?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3622) DirectRunner memory issue with Python SDK

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-3622:
-

Assignee: Charles Chen  (was: Thomas Groh)

> DirectRunner memory issue with Python SDK
> -
>
> Key: BEAM-3622
> URL: https://issues.apache.org/jira/browse/BEAM-3622
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: yuri krnr
>Assignee: Charles Chen
>Priority: Major
>
> After running pipeline for a while in a streaming mode (reading from Pub/Sub 
> and writing to BigQuery, Datastore and another Pub/Sub) I noticed drastic 
> memory usage of a process. Using guppy as a profiler I got the following 
> results:
> start
> {noformat}
>  INFO *** MemoryReport Heap:
>  Partition of a set of 240208 objects. Total size = 34988840 bytes.
>  Index  Count   % Size   % Cumulative  % Kind (class / dict of class)
>  0  88289  37  8696984  25   8696984  25 str
>  1  5  22  4897352  14  13594336  39 tuple
>  2   5083   2  2790664   8  16385000  47 dict (no owner)
>  3   1939   1  1749656   5  18134656  52 type
>  4699   0  1723272   5  19857928  57 dict of module
>  5  12337   5  1579136   5  21437064  61 types.CodeType
>  6  12403   5  1488360   4  22925424  66 function
>  7   1939   1  1452616   4  24378040  70 dict of type
>  8677   0   709496   2  25087536  72 dict of 0x1e4d880
>  9  25603  11   614472   2  25702008  73 int
> <1103 more rows. Type e.g. '_.more' to view.>
> {noformat}
> after several hours of running
> {noformat}
> INFO *** MemoryReport Heap:
>  Partition of a set of 1255662 objects. Total size = 315029632 bytes.
>  Index  Count   % Size   % Cumulative  % Kind (class / dict of class)
>  0  95554   8 99755056  32  99755056  32 dict of
>  
> apache_beam.runners.direct.bundle_factory._Bundle
>  1 117943   9 54193192  17 153948248  49 dict (no owner)
>  2 161068  13 27169296   9 181117544  57 unicode
>  3  94571   8 26479880   8 207597424  66 dict of apache_beam.pvalue.PBegin
>  4 126461  10 12715336   4 220312760  70 str
>  5  44374   4 12424720   4 232737480  74 dict of 
> apitools.base.protorpclite.messages.FieldList
>  6  44374   4  6348624   2 239086104  76 
> apitools.base.protorpclite.messages.FieldList
>  7  95556   8  6115584   2 245201688  78 
> apache_beam.runners.direct.bundle_factory._Bundle
>  8  94571   8  6052544   2 251254232  80 apache_beam.pvalue.PBegin
>  9  57371   5  5218424   2 256472656  81 tuple
> <1187 more rows. Type e.g. '_.more' to view.>
> {noformat}
>  
> I see that every bundle still sits in memory and all its data too. why aren't 
> the gc-ed?
> What is the policy for gc for the dataflow processes?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3646) Add comments about appropriate use of DoFn.Teardown

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-3646:
-

 Summary: Add comments about appropriate use of DoFn.Teardown
 Key: BEAM-3646
 URL: https://issues.apache.org/jira/browse/BEAM-3646
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Thomas Groh
Assignee: Thomas Groh


Because the {{Teardown}} method has no relation to the atomicity of processing 
and commiting of output, it is EXTREMELY DANGEROUS to use to flush outputs, and 
buffered data there is extremely likely to never be flushed. If a DoFn instance 
with buffered data is lost (for example, via worker/machine failure), and the 
runner has committed the result of processing that input, the data is lost.

 

Not commenting on this being the case can cause users to believe that 
(especially if running a batch pipeline) that their data will be flushed on 
pipeline completion. This is very dangerous behavior that we do not warn of 
sufficiently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #5906

[Apache Jenkins Server]

See 

[beam] branch revert-4139-depsupdate deleted (was bb6db87)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch revert-4139-depsupdate
in repository https://gitbox.apache.org/repos/asf/beam.git.


 was bb6db87  Revert "Update cloud spanner library to 0.29.0"

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (b809dd0 -> f2c87bd)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b809dd0  [BEAM-3457] Add Go Gradle precommit
 add bb6db87  Revert "Update cloud spanner library to 0.29.0"
 new f2c87bd  Merge pull request #4635 from apache/revert-4139-depsupdate

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 pom.xml| 28 +++---
 .../construction/ArtifactServiceStagerTest.java|  4 ++--
 .../LocalFileSystemArtifactStagerServiceTest.java  |  4 ++--
 sdks/java/fn-execution/pom.xml |  4 
 sdks/java/io/google-cloud-platform/pom.xml |  2 +-
 .../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java  |  4 +---
 6 files changed, 14 insertions(+), 32 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #4635 from apache/revert-4139-depsupdate

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f2c87bd385da2399938d6ff414176ea953c3c776
Merge: b809dd0 bb6db87
Author: Eugene Kirpichov 
AuthorDate: Wed Feb 7 18:23:21 2018 -0800

Merge pull request #4635 from apache/revert-4139-depsupdate

Revert "Update cloud spanner library to 0.29.0"

 pom.xml| 28 +++---
 .../construction/ArtifactServiceStagerTest.java|  4 ++--
 .../LocalFileSystemArtifactStagerServiceTest.java  |  4 ++--
 sdks/java/fn-execution/pom.xml |  4 
 sdks/java/io/google-cloud-platform/pom.xml |  2 +-
 .../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java  |  4 +---
 6 files changed, 14 insertions(+), 32 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: [BEAM-3457] Add Go Gradle precommit

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b809dd074219ba191b3bd54cd85f7f5024584202
Merge: 37e1db7 578c3ad
Author: Lukasz Cwik 
AuthorDate: Wed Feb 7 17:40:00 2018 -0800

[BEAM-3457] Add Go Gradle precommit

 groovy => job_beam_PreCommit_Go_GradleBuild.groovy} | 17 ++---
 .../jenkins/job_beam_PreCommit_Java_GradleBuild.groovy  |  2 +-
 build.gradle|  9 -
 3 files changed, 15 insertions(+), 13 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (37e1db7 -> b809dd0)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 37e1db7  Merge pull request #4602 from charlesccychen/snippets-pickling
 add 578c3ad  [BEAM-3457] Add Go Gradle precommit
 new b809dd0  [BEAM-3457] Add Go Gradle precommit

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 groovy => job_beam_PreCommit_Go_GradleBuild.groovy} | 17 ++---
 .../jenkins/job_beam_PreCommit_Java_GradleBuild.groovy  |  2 +-
 build.gradle|  9 -
 3 files changed, 15 insertions(+), 13 deletions(-)
 copy .test-infra/jenkins/{job_beam_PreCommit_Java_GradleBuild.groovy => 
job_beam_PreCommit_Go_GradleBuild.groovy} (81%)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: Merge pull request #4602 from charlesccychen/snippets-pickling

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 37e1db7e33f99fb0a6147137841d682c7d4afbf9
Merge: 6745a61 8f9edbc
Author: Robert Bradshaw 
AuthorDate: Wed Feb 7 17:11:08 2018 -0800

Merge pull request #4602 from charlesccychen/snippets-pickling

Update snippets to fix pickling and clarify encoding issues

 .../apache_beam/examples/snippets/snippets.py  | 208 +++--
 .../apache_beam/examples/snippets/snippets_test.py |  16 +-
 2 files changed, 115 insertions(+), 109 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] branch master updated (6745a61 -> 37e1db7)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6745a61  [BEAM-3639] Update to gradle 4.5.1
 add 8f9edbc  Update snippets to fix pickling and clarify encoding issues
 new 37e1db7  Merge pull request #4602 from charlesccychen/snippets-pickling

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache_beam/examples/snippets/snippets.py  | 208 +++--
 .../apache_beam/examples/snippets/snippets_test.py |  16 +-
 2 files changed, 115 insertions(+), 109 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5905

[Apache Jenkins Server]

See 


--
[...truncated 2.29 MB...]
2018-02-08T00:56:56.940 [INFO] Excluding 
org.apache.beam:beam-model-job-management:jar:2.4.0-SNAPSHOT from the shaded 
jar.
2018-02-08T00:56:56.940 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding com.google.code.gson:gson:jar:2.7 from 
the shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding io.grpc:grpc-core:jar:1.7.0 from the 
shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding io.grpc:grpc-context:jar:1.7.0 from 
the shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding 
com.google.instrumentation:instrumentation-api:jar:0.4.3 from the shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding io.opencensus:opencensus-api:jar:0.6.0 
from the shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding io.grpc:grpc-stub:jar:1.7.0 from the 
shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding 
org.apache.beam:beam-runners-core-java:jar:2.4.0-SNAPSHOT from the shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding 
org.apache.commons:commons-lang3:jar:3.6 from the shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.7.0 from 
the shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding 
com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from the shaded jar.
2018-02-08T00:56:56.940 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.7.0 
from the shaded jar.
2018-02-08T00:57:00.541 [INFO] Replacing original artifact with shaded artifact.
2018-02-08T00:57:00.541 [INFO] Replacing 

 with 

2018-02-08T00:57:00.541 [INFO] Replacing original test artifact with shaded 
test artifact.
2018-02-08T00:57:00.541 [INFO] Replacing 

 with 

2018-02-08T00:57:00.748 [INFO] 
2018-02-08T00:57:00.748 [INFO] --- maven-assembly-plugin:3.1.0:single 
(source-release-assembly) @ beam-runners-apex ---
2018-02-08T00:57:00.751 [INFO] Skipping the assembly in this project because 
it's not the Execution Root
2018-02-08T00:57:00.859 [INFO] 
2018-02-08T00:57:00.859 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-runners-apex ---
2018-02-08T00:57:00.863 [INFO] Building jar: 

2018-02-08T00:57:01.000 [INFO] 
2018-02-08T00:57:01.000 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-runners-apex ---
2018-02-08T00:57:01.003 [INFO] Building jar: 

2018-02-08T00:57:01.145 [INFO] 
2018-02-08T00:57:01.146 [INFO] --- maven-javadoc-plugin:3.0.0-M1:jar 
(attach-javadocs) @ beam-runners-apex ---
2018-02-08T00:57:06.817 [INFO] 
2 warnings
2018-02-08T00:57:06.818 [WARNING] Javadoc Warnings
2018-02-08T00:57:06.818 [WARNING] 
:33:
 warning: no description for @param
2018-02-08T00:57:06.818 [WARNING] * @param 
2018-02-08T00:57:06.818 [WARNING] ^
2018-02-08T00:57:06.818 [WARNING] 
:48:
 warning: no description for @param
2018-02-08T00:57:06.818 [WARNING] * @param 
2018-02-08T00:57:06.818 [WARNING] ^
2018-02-08T00:57:06.832 [INFO] Building jar: 

2018-02-08T00:57:07.027 [INFO] 
2018-02-08T00:57:07.027 [INFO] --- 
reproducible-build-maven-plugin:0.4:strip-jar (default) @ beam-runners-apex ---
2018-02-08T00:57:07.028 [INFO] Stripping 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #4941

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #114

[Apache Jenkins Server]

See 


Changes:

[mairbek] Update cloud spanner library to 0.29.0

[mairbek] Fix test

[mairbek] More google-cloud-platform whitelisting

[mairbek] pom updates to make maven happy

[mairbek] Update netty deps

[mairbek] Fixed broken test

[ehudm] Add and migrate to HdfsCLI library for Python SDK.

[klk] Fix typo in gradle idea hints

[dkulp] [BEAM-3639] Update to gradle 4.5.1

--
[...truncated 13.30 KB...]
Requirement already satisfied: monotonic>=0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth<2.0.0dev,>=0.4.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth-httplib2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-gax<0.16dev,>=0.15.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: 
proto-google-cloud-pubsub-v1[grpc]<0.16dev,>=0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: cachetools>=2.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: hdfs, apache-beam
  Found existing installation: apache-beam 2.4.0.dev0
Not uninstalling apache-beam at 
/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python,
 outside environment 

  Running setup.py develop for apache-beam
Successfully installed apache-beam hdfs-2.1.0
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5354667756257548827.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.tfrecordioit_pkb_results 
--temp_dir=
 --official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 
--beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java 
--beam_it_module=sdks/java/io/file-based-io-tests 
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT 
'--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/114/]'
 '--beam_extra_mvn_properties=[filesystem=gcs]'
2018-02-08 00:00:51,540 2647a52d MainThread INFO Verbose logging to: 

2018-02-08 00:00:51,541 2647a52d MainThread INFO PerfKitBenchmarker 
version: v1.12.0-260-ge324e61
2018-02-08 00:00:51,542 2647a52d MainThread INFO Flag values:
--beam_extra_mvn_properties=[filesystem=gcs]
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT
--beam_it_timeout=1200
--beam_sdk=java
--temp_dir=
--maven_binary=/home/jenkins/tools/maven/latest/bin/mvn
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/114/]
--beam_prebuilt
--project=apache-beam-testing
--bigquery_table=beam_performance.tfrecordioit_pkb_results
--official
--beam_it_module=sdks/java/io/file-based-io-tests
--dpb_log_level=INFO
--beam_it_profile=io-it
--benchmarks=beam_integration_benchmark
2018-02-08 00:00:51,762 2647a52d MainThread WARNING  The key "flags" 

Build failed in Jenkins: beam_PerformanceTests_Python #887

[Apache Jenkins Server]

See 


Changes:

[mairbek] Update cloud spanner library to 0.29.0

[mairbek] Fix test

[mairbek] More google-cloud-platform whitelisting

[mairbek] pom updates to make maven happy

[mairbek] Update netty deps

[mairbek] Fixed broken test

[ehudm] Add and migrate to HdfsCLI library for Python SDK.

[klk] Fix typo in gradle idea hints

[dkulp] [BEAM-3639] Update to gradle 4.5.1

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 6745a6119d609b4008e961912e84e932915bdb13 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6745a6119d609b4008e961912e84e932915bdb13
Commit message: "[BEAM-3639] Update to gradle 4.5.1"
 > git rev-list fab8ca0b1e07837d0a0d63d1957468c0401e01e7 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1234899201070624103.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1860962156758772611.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6079312268737955276.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8141931892018767063.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4060032387073966504.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8282164355878853193.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #20

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #856

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-3644) Speed up Python DirectRunner execution by using the FnApiRunner when possible

[Robert Bradshaw (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Bradshaw updated BEAM-3644:
--
Description: 
Local execution of Beam pipelines on the current Python DirectRunner currently 
suffers from performance issues, which makes it hard for pipeline authors to 
iterate, especially on medium to large size datasets. We would like to optimize 
and make this a better experience for Beam users.

The FnApiRunner was written as a way of leveraging the portability framework 
execution code path for local execution for portability development. We've 
found it also offers great speedups in batch execution, so we propose to switch 
to use this runner in batch pipelines. For example, WordCount on the 
Shakespeare dataset with a single CPU core now takes 50 seconds to run, 
compared to 12 minutes before, a 15x performance improvement that users can get 
for free, with no pipeline changes.

  was:
Local execution of Beam pipelines on the current Python DirectRunner currently 
suffers from performance issues, which makes it hard for pipeline authors to 
iterate, especially on medium to large size datasets. We would like to optimize 
and make this a better experience for Beam users.

In the past few months, Robert implemented the FnApiRunner as a way of 
leveraging the portability framework execution code path for local execution. 
We've found great speedups in batch execution, so we propose to switch to use 
this runner in batch pipelines. For example, WordCount on the Shakespeare 
dataset with a single CPU core now takes 50 seconds to run, compared to 12 
minutes before, a 15x performance improvement that users can get for free, with 
no pipeline changes.


> Speed up Python DirectRunner execution by using the FnApiRunner when possible
> -
>
> Key: BEAM-3644
> URL: https://issues.apache.org/jira/browse/BEAM-3644
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>
> Local execution of Beam pipelines on the current Python DirectRunner 
> currently suffers from performance issues, which makes it hard for pipeline 
> authors to iterate, especially on medium to large size datasets. We would 
> like to optimize and make this a better experience for Beam users.
> The FnApiRunner was written as a way of leveraging the portability framework 
> execution code path for local execution for portability development. We've 
> found it also offers great speedups in batch execution, so we propose to 
> switch to use this runner in batch pipelines. For example, WordCount on the 
> Shakespeare dataset with a single CPU core now takes 50 seconds to run, 
> compared to 12 minutes before, a 15x performance improvement that users can 
> get for free, with no pipeline changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3645) Support multi-process execution on the FnApiRunner

[Charles Chen (JIRA)]

Charles Chen created BEAM-3645:
--

 Summary: Support multi-process execution on the FnApiRunner
 Key: BEAM-3645
 URL: https://issues.apache.org/jira/browse/BEAM-3645
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Affects Versions: 2.2.0, 2.3.0
Reporter: Charles Chen
Assignee: Charles Chen


https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance gain 
over the previous DirectRunner.  We can do even better in multi-core 
environments by supporting multi-process execution in the FnApiRunner, to scale 
past Python GIL limitations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3644) Speed up Python DirectRunner execution by using the FnApiRunner when possible

[Charles Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Charles Chen updated BEAM-3644:
---
Summary: Speed up Python DirectRunner execution by using the FnApiRunner 
when possible  (was: Speeding up Python DirectRunner execution by using the 
FnApiRunner when possible)

> Speed up Python DirectRunner execution by using the FnApiRunner when possible
> -
>
> Key: BEAM-3644
> URL: https://issues.apache.org/jira/browse/BEAM-3644
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>
> Local execution of Beam pipelines on the current Python DirectRunner 
> currently suffers from performance issues, which makes it hard for pipeline 
> authors to iterate, especially on medium to large size datasets. We would 
> like to optimize and make this a better experience for Beam users.
> In the past few months, Robert implemented the FnApiRunner as a way of 
> leveraging the portability framework execution code path for local execution. 
> We've found great speedups in batch execution, so we propose to switch to use 
> this runner in batch pipelines. For example, WordCount on the Shakespeare 
> dataset with a single CPU core now takes 50 seconds to run, compared to 12 
> minutes before, a 15x performance improvement that users can get for free, 
> with no pipeline changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3644) Speeding up Python DirectRunner execution by using the FnApiRunner when possible

[Charles Chen (JIRA)]

Charles Chen created BEAM-3644:
--

 Summary: Speeding up Python DirectRunner execution by using the 
FnApiRunner when possible
 Key: BEAM-3644
 URL: https://issues.apache.org/jira/browse/BEAM-3644
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Affects Versions: 2.2.0, 2.3.0
Reporter: Charles Chen
Assignee: Charles Chen


Local execution of Beam pipelines on the current Python DirectRunner currently 
suffers from performance issues, which makes it hard for pipeline authors to 
iterate, especially on medium to large size datasets. We would like to optimize 
and make this a better experience for Beam users.

In the past few months, Robert implemented the FnApiRunner as a way of 
leveraging the portability framework execution code path for local execution. 
We've found great speedups in batch execution, so we propose to switch to use 
this runner in batch pipelines. For example, WordCount on the Shakespeare 
dataset with a single CPU core now takes 50 seconds to run, compared to 12 
minutes before, a 15x performance improvement that users can get for free, with 
no pipeline changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #4940

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3643) Remove documentation for creating custom sinks

[Udi Meiri (JIRA)]

Udi Meiri created BEAM-3643:
---

 Summary: Remove documentation for creating custom sinks
 Key: BEAM-3643
 URL: https://issues.apache.org/jira/browse/BEAM-3643
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Udi Meiri
Assignee: Reuven Lax


I believe this whole section should be removed:

[https://beam.apache.org/documentation/sdks/python-custom-io/#creating-a-new-sink]

The reason is that class Sink is deprecated:

[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/iobase.py#L578]

Associated code snippets should also be removed:

[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py#L782]

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3639) Upgrade to gradle 4.5.1

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3639.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Upgrade to gradle 4.5.1
> ---
>
> Key: BEAM-3639
> URL: https://issues.apache.org/jira/browse/BEAM-3639
> Project: Beam
>  Issue Type: Task
>  Components: build-system
>Reporter: Daniel Kulp
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The Gradle 4.5.1 release notes mention two things of interest to us (along 
> with a ton of other things):
> 1) Lower memory usage and better use of build caches (aka: faster builds)
> 2) Updates to gpg plugin to enable agent use - likely will be required if we 
> eventually switch to using gradle for release builds.
> https://docs.gradle.org/current/release-notes.html
> The 4.4 update also includes updates to eclipse plugin to help setup eclipse 
> via gradle, 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: [BEAM-3639] Update to gradle 4.5.1

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6745a6119d609b4008e961912e84e932915bdb13
Merge: 9d72e81 e29cc6e
Author: Lukasz Cwik 
AuthorDate: Wed Feb 7 14:44:18 2018 -0800

[BEAM-3639] Update to gradle 4.5.1

 gradle/wrapper/gradle-wrapper.jar| Bin 54712 -> 54333 bytes
 gradle/wrapper/gradle-wrapper.properties |   2 +-
 2 files changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (9d72e81 -> 6745a61)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9d72e81  Fix typo in gradle idea hints
 add e29cc6e  [BEAM-3639] Update to gradle 4.5.1
 new 6745a61  [BEAM-3639] Update to gradle 4.5.1

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 gradle/wrapper/gradle-wrapper.jar| Bin 54712 -> 54333 bytes
 gradle/wrapper/gradle-wrapper.properties |   2 +-
 2 files changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (3371be2 -> 9d72e81)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3371be2  Merge pull request #4139: Update cloud spanner library to 
0.29.0
 add 506ddcf  Fix typo in gradle idea hints
 new 9d72e81  Fix typo in gradle idea hints

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 build_rules.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: Fix typo in gradle idea hints

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 9d72e81b5216f3fbe8067dcf940f38ca9c703976
Merge: 3371be2 506ddcf
Author: Lukasz Cwik 
AuthorDate: Wed Feb 7 14:40:00 2018 -0800

Fix typo in gradle idea hints

 build_rules.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Created] (BEAM-3642) Support bundle retries in the FnApiRunner

[Charles Chen (JIRA)]

Charles Chen created BEAM-3642:
--

 Summary: Support bundle retries in the FnApiRunner
 Key: BEAM-3642
 URL: https://issues.apache.org/jira/browse/BEAM-3642
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Charles Chen
Assignee: Ahmet Altay


The Python DirectRunner supports bundle retries, but the FnApiRunner currently 
does not support this mechanism.  We should support bundle retries in the 
FnApiRunner as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3641) Clean up logging in the FnApiRunner

[Charles Chen (JIRA)]

Charles Chen created BEAM-3641:
--

 Summary: Clean up logging in the FnApiRunner
 Key: BEAM-3641
 URL: https://issues.apache.org/jira/browse/BEAM-3641
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-harness
Reporter: Charles Chen
Assignee: Robert Bradshaw


We should consider cleaning up the logging emitted through execution of 
pipelines with the local FnApiRunner.  We currently do some pretty verbose 
debug logging, and we should audit each logging invocation to retain only the 
ones necessary.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3635) Infer type hints on PTransformOverrides

[Charles Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Charles Chen resolved BEAM-3635.

   Resolution: Fixed
Fix Version/s: 2.4.0

> Infer type hints on PTransformOverrides
> ---
>
> Key: BEAM-3635
> URL: https://issues.apache.org/jira/browse/BEAM-3635
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, type hints on transforms created by PTransformOverrides are 
> ignored.  We should instead infer the output types for these transforms using 
> any type hints associated to the transform.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #855

[Apache Jenkins Server]

See 


--
[...truncated 559.25 KB...]
{
  "@type": "kind:stream", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:even/Group/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s28"
}, 
"serialized_fn": 
"%0AD%22B%0A%1Dref_Coder_GlobalWindowCoder_1%12%21%0A%1F%0A%1D%0A%1Bbeam%3Acoder%3Aglobal_window%3Av1jT%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
 
"user_name": "assert:even/Group/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s30", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert:even/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s29"
}, 
"serialized_fn": "", 
"user_name": "assert:even/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s31", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #5901

[Apache Jenkins Server]

See 

[beam] annotated tag v2.3.0-RC2 created (now 5b22779)

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a change to annotated tag v2.3.0-RC2
in repository https://gitbox.apache.org/repos/asf/beam.git.


  at 5b22779  (tag)
 tagging 6cdc2f561d9dcafca33cc0e1754dab9c34382705 (commit)
 replaces v2.3.0-RC1
  by Jean-Baptiste Onofré
  on Wed Feb 7 21:08:28 2018 +0100

- Log -
[maven-release-plugin] copy for tag v2.3.0-RC2
---

No new revisions were added by this update.

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[beam] branch release-2.3.0 updated: [maven-release-plugin] rollback changes from release preparation of v2.3.0-RC2

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch release-2.3.0
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/release-2.3.0 by this push:
 new c42cc8c  [maven-release-plugin] rollback changes from release 
preparation of v2.3.0-RC2
c42cc8c is described below

commit c42cc8c5d13a4ef02d4320794ba9329594b7056b
Author: Jean-Baptiste Onofré 
AuthorDate: Wed Feb 7 21:08:48 2018 +0100

[maven-release-plugin] rollback changes from release preparation of 
v2.3.0-RC2
---
 examples/java/pom.xml   | 2 +-
 examples/pom.xml| 2 +-
 model/fn-execution/pom.xml  | 2 +-
 model/job-management/pom.xml| 2 +-
 model/pipeline/pom.xml  | 2 +-
 model/pom.xml   | 2 +-
 pom.xml | 4 ++--
 runners/apex/pom.xml| 2 +-
 runners/core-construction-java/pom.xml  | 2 +-
 runners/core-java/pom.xml   | 2 +-
 runners/direct-java/pom.xml | 2 +-
 runners/flink/pom.xml   | 2 +-
 runners/gcp/gcemd/pom.xml   | 2 +-
 runners/gcp/gcsproxy/pom.xml| 2 +-
 runners/gcp/pom.xml | 2 +-
 runners/gearpump/pom.xml| 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 runners/java-fn-execution/pom.xml   | 2 +-
 runners/local-artifact-service-java/pom.xml | 2 +-
 runners/local-java/pom.xml  | 2 +-
 runners/pom.xml | 2 +-
 runners/reference/java/pom.xml  | 2 +-
 runners/reference/job-server/pom.xml| 2 +-
 runners/reference/pom.xml   | 2 +-
 runners/spark/pom.xml   | 2 +-
 sdks/go/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml   | 2 +-
 sdks/java/container/pom.xml | 2 +-
 sdks/java/core/pom.xml  | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml | 2 +-
 sdks/java/extensions/jackson/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml   | 2 +-
 sdks/java/extensions/pom.xml| 2 +-
 sdks/java/extensions/protobuf/pom.xml   | 2 +-
 sdks/java/extensions/sketching/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml | 2 +-
 sdks/java/extensions/sql/pom.xml| 2 +-
 sdks/java/fn-execution/pom.xml  | 2 +-
 sdks/java/harness/pom.xml   | 2 +-
 sdks/java/io/amazon-web-services/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml   | 2 +-
 sdks/java/io/cassandra/pom.xml  | 2 +-
 sdks/java/io/common/pom.xml | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/pom.xml | 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml| 2 +-
 sdks/java/io/elasticsearch/pom.xml  | 2 +-
 sdks/java/io/file-based-io-tests/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml  | 2 +-
 sdks/java/io/hadoop-common/pom.xml  | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml | 2 +-
 sdks/java/io/hadoop-input-format/pom.xml| 2 +-
 sdks/java/io/hbase/pom.xml  | 2 +-
 sdks/java/io/hcatalog/pom.xml   | 2 +-
 sdks/java/io/jdbc/pom.xml   | 2 +-
 sdks/java/io/jms/pom.xml| 2 +-
 sdks/java/io/kafka/pom.xml  

[beam] branch release-2.3.0 updated: [maven-release-plugin] prepare release v2.3.0-RC2

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch release-2.3.0
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/release-2.3.0 by this push:
 new 6cdc2f5  [maven-release-plugin] prepare release v2.3.0-RC2
6cdc2f5 is described below

commit 6cdc2f561d9dcafca33cc0e1754dab9c34382705
Author: Jean-Baptiste Onofré 
AuthorDate: Wed Feb 7 21:08:03 2018 +0100

[maven-release-plugin] prepare release v2.3.0-RC2
---
 examples/java/pom.xml   | 2 +-
 examples/pom.xml| 2 +-
 model/fn-execution/pom.xml  | 2 +-
 model/job-management/pom.xml| 2 +-
 model/pipeline/pom.xml  | 2 +-
 model/pom.xml   | 2 +-
 pom.xml | 4 ++--
 runners/apex/pom.xml| 2 +-
 runners/core-construction-java/pom.xml  | 2 +-
 runners/core-java/pom.xml   | 2 +-
 runners/direct-java/pom.xml | 2 +-
 runners/flink/pom.xml   | 2 +-
 runners/gcp/gcemd/pom.xml   | 2 +-
 runners/gcp/gcsproxy/pom.xml| 2 +-
 runners/gcp/pom.xml | 2 +-
 runners/gearpump/pom.xml| 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 runners/java-fn-execution/pom.xml   | 2 +-
 runners/local-artifact-service-java/pom.xml | 2 +-
 runners/local-java/pom.xml  | 2 +-
 runners/pom.xml | 2 +-
 runners/reference/java/pom.xml  | 2 +-
 runners/reference/job-server/pom.xml| 2 +-
 runners/reference/pom.xml   | 2 +-
 runners/spark/pom.xml   | 2 +-
 sdks/go/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml   | 2 +-
 sdks/java/container/pom.xml | 2 +-
 sdks/java/core/pom.xml  | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml | 2 +-
 sdks/java/extensions/jackson/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml   | 2 +-
 sdks/java/extensions/pom.xml| 2 +-
 sdks/java/extensions/protobuf/pom.xml   | 2 +-
 sdks/java/extensions/sketching/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml | 2 +-
 sdks/java/extensions/sql/pom.xml| 2 +-
 sdks/java/fn-execution/pom.xml  | 2 +-
 sdks/java/harness/pom.xml   | 2 +-
 sdks/java/io/amazon-web-services/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml   | 2 +-
 sdks/java/io/cassandra/pom.xml  | 2 +-
 sdks/java/io/common/pom.xml | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/pom.xml | 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml| 2 +-
 sdks/java/io/elasticsearch/pom.xml  | 2 +-
 sdks/java/io/file-based-io-tests/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml  | 2 +-
 sdks/java/io/hadoop-common/pom.xml  | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml | 2 +-
 sdks/java/io/hadoop-input-format/pom.xml| 2 +-
 sdks/java/io/hbase/pom.xml  | 2 +-
 sdks/java/io/hcatalog/pom.xml   | 2 +-
 sdks/java/io/jdbc/pom.xml   | 2 +-
 sdks/java/io/jms/pom.xml| 2 +-
 sdks/java/io/kafka/pom.xml  | 2 +-
 sdks/java/io/kinesis/pom.xml

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesContainer_Dataflow #13

[Apache Jenkins Server]

See 

[jira] [Closed] (BEAM-3582) BeamRecord should be called BeamRow instead

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin closed BEAM-3582.
-
   Resolution: Fixed
Fix Version/s: Not applicable

[https://github.com/apache/beam/pull/4628] merged

> BeamRecord should be called BeamRow instead
> ---
>
> Key: BEAM-3582
> URL: https://issues.apache.org/jira/browse/BEAM-3582
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> All elements in Beam are referred to as "records," so the current class name 
> is confusing. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3639) Upgrade to gradle 4.5.1

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355934#comment-16355934
 ] 

Jean-Baptiste Onofré commented on BEAM-3639:


Note that sometime the wrapper might use a different gradle version.

> Upgrade to gradle 4.5.1
> ---
>
> Key: BEAM-3639
> URL: https://issues.apache.org/jira/browse/BEAM-3639
> Project: Beam
>  Issue Type: Task
>  Components: build-system
>Reporter: Daniel Kulp
>Assignee: Daniel Kulp
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The Gradle 4.5.1 release notes mention two things of interest to us (along 
> with a ton of other things):
> 1) Lower memory usage and better use of build caches (aka: faster builds)
> 2) Updates to gpg plugin to enable agent use - likely will be required if we 
> eventually switch to using gradle for release builds.
> https://docs.gradle.org/current/release-notes.html
> The 4.4 update also includes updates to eclipse plugin to help setup eclipse 
> via gradle, 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5900

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3640) Checkstyle enforcement of blank lines

[Daniel Kulp (JIRA)]

Daniel Kulp created BEAM-3640:
-

 Summary: Checkstyle enforcement of blank lines
 Key: BEAM-3640
 URL: https://issues.apache.org/jira/browse/BEAM-3640
 Project: Beam
  Issue Type: Task
  Components: build-system
Reporter: Daniel Kulp
Assignee: Daniel Kulp



Checkstyle has a rule (EmptyLineSeparator) we currently don't use to enforce 
the use of blank lines in various places.   In order to reduce the noise in 
reviews of PR's, I like to start using it.

This will likely be accomplished via a series of PR's.   Just adding the 
default EmptyLineSeparator to the checkstyle rules results in a THOUSANDS of 
violations which would be hard to review.   I will likely enable it for one 
token type at a time, fix the violations on that particular token type, submit, 
add then next token type, etc... That should  (hopefully) keep each PR down to 
something that can reasonably be reviewed. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (cf33dba -> 3371be2)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from cf33dba  Merge pull request #4581: [BEAM-3099] Add and migrate to 
HdfsCLI library for Python SDK
 add 5d4da50  Update cloud spanner library to 0.29.0
 add eeff6be  Fix test
 add f712e3e  More google-cloud-platform whitelisting
 add 4808ca9  pom updates to make maven happy
 add f9eac5e  Update netty deps
 add d3715c3  Fixed broken test
 new 3371be2  Merge pull request #4139: Update cloud spanner library to 
0.29.0

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 pom.xml| 28 +++---
 .../construction/ArtifactServiceStagerTest.java|  4 ++--
 .../LocalFileSystemArtifactStagerServiceTest.java  |  4 ++--
 sdks/java/fn-execution/pom.xml |  4 
 sdks/java/io/google-cloud-platform/pom.xml |  2 +-
 .../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java  |  4 +++-
 6 files changed, 32 insertions(+), 14 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] 01/01: Merge pull request #4139: Update cloud spanner library to 0.29.0

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 3371be2860341c93a09c76331dbaefb4544e2c29
Merge: cf33dba d3715c3
Author: Eugene Kirpichov 
AuthorDate: Wed Feb 7 11:05:25 2018 -0800

Merge pull request #4139: Update cloud spanner library to 0.29.0

Update cloud spanner library to 0.29.0

 pom.xml| 28 +++---
 .../construction/ArtifactServiceStagerTest.java|  4 ++--
 .../LocalFileSystemArtifactStagerServiceTest.java  |  4 ++--
 sdks/java/fn-execution/pom.xml |  4 
 sdks/java/io/google-cloud-platform/pom.xml |  2 +-
 .../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java  |  4 +++-
 6 files changed, 32 insertions(+), 14 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #5897

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #886

[Apache Jenkins Server]

See 


Changes:

[jbonofre] Revert "Reinstate proto round trip in Java DirectRunner"

[tgroh] Update Assign Window URN Constant Name

[kedin] Rename BeamRecord -> Row, BeamRecordType -> RowType

[wcn] Modify BufferedElementCountingOutputStream to use a buffer pool for its

[klk] Fix stable name errors in HBaseIOTest

[aromanenko.dev] [BEAM-3291] Add Kinesis write transform

[coheigea] Remove unused private variables.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision fab8ca0b1e07837d0a0d63d1957468c0401e01e7 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fab8ca0b1e07837d0a0d63d1957468c0401e01e7
Commit message: "BEAM-3572: Modify BufferedElementCountingOutputStream to use a 
buffer pool"
 > git rev-list 92a42020c9683db2fdc95a1c14023480d114b22c # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3207206500187824607.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5514076754637105647.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8388522437644174647.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1024611615388094064.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8254005689068186400.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7465521318077960460.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))

[beam] branch master updated (fab8ca0 -> cf33dba)

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from fab8ca0  BEAM-3572: Modify BufferedElementCountingOutputStream to use 
a buffer pool
 add 5143c43  Add and migrate to HdfsCLI library for Python SDK.
 new cf33dba  Merge pull request #4581: [BEAM-3099] Add and migrate to 
HdfsCLI library for Python SDK

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/io/filesystemio.py |   5 +-
 sdks/python/apache_beam/io/filesystems.py  |   2 +-
 sdks/python/apache_beam/io/gcp/gcsio.py|   1 -
 sdks/python/apache_beam/io/hadoopfilesystem.py | 126 +
 .../python/apache_beam/io/hadoopfilesystem_test.py | 153 -
 .../python/apache_beam/options/pipeline_options.py |  28 
 sdks/python/generate_pydoc.sh  |   1 -
 sdks/python/setup.py   |   2 +-
 8 files changed, 222 insertions(+), 96 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[beam] 01/01: Merge pull request #4581: [BEAM-3099] Add and migrate to HdfsCLI library for Python SDK

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit cf33dba157b6edaeb90f425fdf5dfa820bacc749
Merge: fab8ca0 5143c43
Author: Chamikara Jayalath 
AuthorDate: Wed Feb 7 10:39:19 2018 -0800

Merge pull request #4581: [BEAM-3099] Add and migrate to HdfsCLI library 
for Python SDK

 sdks/python/apache_beam/io/filesystemio.py |   5 +-
 sdks/python/apache_beam/io/filesystems.py  |   2 +-
 sdks/python/apache_beam/io/gcp/gcsio.py|   1 -
 sdks/python/apache_beam/io/hadoopfilesystem.py | 126 +
 .../python/apache_beam/io/hadoopfilesystem_test.py | 153 -
 .../python/apache_beam/options/pipeline_options.py |  28 
 sdks/python/generate_pydoc.sh  |   1 -
 sdks/python/setup.py   |   2 +-
 8 files changed, 222 insertions(+), 96 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
chamik...@apache.org.

[jira] [Commented] (BEAM-3561) Provide kubernetes cluster instance for IOITs.

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355855#comment-16355855
 ] 

Chamikara Jayalath commented on BEAM-3561:
--

This cluster is not used by any tests so should be fine to cleanup and reuse (I 
think you are correct that it was configured before to run the JDBC test).

Pls note that this is a single node cluster, we are in the process of setting 
up a larger cluster that would accommodate more tests. But it's great if we can 
enable one/some of the tests against this cluster as a proof of concept.

> Provide kubernetes cluster instance for IOITs.
> --
>
> Key: BEAM-3561
> URL: https://issues.apache.org/jira/browse/BEAM-3561
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Jason Kuster
>Priority: Major
>
> Performance tests that require running Kubernetes scripts currently cannot be 
> run on Jenkins. This is due to the fact that there is no dedicated kubernetes 
> cluster for them so Jenkins jobs cannot setup the needed infrastructure 
> anywhere.
> To allow running such tests we should provide an instance of kubernetes 
> cluster (for example a cluster hosted on GKE) and all necessary credentials 
> to connect with it from Jenkins executors (proper kubeconfig file on all 
> Jenkins executors). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3561) Provide kubernetes cluster instance for IOITs.

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath reassigned BEAM-3561:


Assignee: Alan Myrvold  (was: Jason Kuster)

> Provide kubernetes cluster instance for IOITs.
> --
>
> Key: BEAM-3561
> URL: https://issues.apache.org/jira/browse/BEAM-3561
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Alan Myrvold
>Priority: Major
>
> Performance tests that require running Kubernetes scripts currently cannot be 
> run on Jenkins. This is due to the fact that there is no dedicated kubernetes 
> cluster for them so Jenkins jobs cannot setup the needed infrastructure 
> anywhere.
> To allow running such tests we should provide an instance of kubernetes 
> cluster (for example a cluster hosted on GKE) and all necessary credentials 
> to connect with it from Jenkins executors (proper kubeconfig file on all 
> Jenkins executors). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3639) Upgrade to gradle 4.5.1

[Daniel Kulp (JIRA)]

Daniel Kulp created BEAM-3639:
-

 Summary: Upgrade to gradle 4.5.1
 Key: BEAM-3639
 URL: https://issues.apache.org/jira/browse/BEAM-3639
 Project: Beam
  Issue Type: Task
  Components: build-system
Reporter: Daniel Kulp
Assignee: Daniel Kulp


The Gradle 4.5.1 release notes mention two things of interest to us (along with 
a ton of other things):

1) Lower memory usage and better use of build caches (aka: faster builds)

2) Updates to gpg plugin to enable agent use - likely will be required if we 
eventually switch to using gradle for release builds.

https://docs.gradle.org/current/release-notes.html

The 4.4 update also includes updates to eclipse plugin to help setup eclipse 
via gradle, 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3572) Reduce inefficient allocations in coders

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3572.
-
   Resolution: Fixed
Fix Version/s: 2.4.0

> Reduce inefficient allocations in coders
> 
>
> Key: BEAM-3572
> URL: https://issues.apache.org/jira/browse/BEAM-3572
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Minor
> Fix For: 2.4.0
>
>   Original Estimate: 24h
>  Time Spent: 10m
>  Remaining Estimate: 23h 50m
>
> BufferedElementCountingOutputStream's constructor allocates a new buffer to 
> wrap the input OutputStream. This gets called on each invocation of encode() 
> from IterableLikeCoder. Since Coder is designed to be stateless, but this 
> buffer holds state and isn't threadsafe, we can't just have the caller manage 
> the buffer. Modifying the constructor to use a pool of buffers to reduce the 
> number of allocations will help performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: BEAM-3572: Modify BufferedElementCountingOutputStream to use a buffer pool

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit fab8ca0b1e07837d0a0d63d1957468c0401e01e7
Merge: a13c7d9 4959ef8
Author: Lukasz Cwik 
AuthorDate: Wed Feb 7 09:52:17 2018 -0800

BEAM-3572: Modify BufferedElementCountingOutputStream to use a buffer pool

 .../util/BufferedElementCountingOutputStream.java  | 15 -
 .../BufferedElementCountingOutputStreamTest.java   | 36 ++
 .../data/BeamFnDataBufferingOutboundObserver.java  |  4 ++-
 3 files changed, 53 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (a13c7d9 -> fab8ca0)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a13c7d9  Merge pull request #4628: [BEAM-3582] Rename BeamRecord -> 
Row, BeamRecordType -> RowType
 add 4959ef8  Modify BufferedElementCountingOutputStream to use a buffer 
pool for its internal allocations. This improves the worker's performance for 
workloads with large iterables since it avoids a allocation/deletion on each 
invocation of encode() in IterableLikeCoder.
 new fab8ca0  BEAM-3572: Modify BufferedElementCountingOutputStream to use 
a buffer pool

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../util/BufferedElementCountingOutputStream.java  | 15 -
 .../BufferedElementCountingOutputStreamTest.java   | 36 ++
 .../data/BeamFnDataBufferingOutboundObserver.java  |  4 ++-
 3 files changed, 53 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: Merge pull request #4628: [BEAM-3582] Rename BeamRecord -> Row, BeamRecordType -> RowType

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a13c7d9177c34017955cbfbfc29a7f4a698047c1
Merge: a957246 c4ac00a
Author: Kenn Knowles 
AuthorDate: Wed Feb 7 09:47:56 2018 -0800

Merge pull request #4628: [BEAM-3582] Rename BeamRecord -> Row, 
BeamRecordType -> RowType

 .../coders/{BeamRecordCoder.java => RowCoder.java} |  44 +++
 .../beam/sdk/values/{BeamRecord.java => Row.java}  |  54 -
 .../values/{BeamRecordType.java => RowType.java}   |  26 ++--
 .../org/apache/beam/sdk/values/BeamRecordTest.java | 113 -
 .../java/org/apache/beam/sdk/values/RowTest.java   | 113 +
 .../{BeamRecordTypeTest.java => RowTypeTest.java}  |  38 +++---
 .../apache/beam/sdk/extensions/sql/BeamSql.java|  18 +--
 .../apache/beam/sdk/extensions/sql/BeamSqlCli.java |   8 +-
 .../sdk/extensions/sql/BeamSqlSeekableTable.java   |   6 +-
 .../beam/sdk/extensions/sql/BeamSqlTable.java  |  10 +-
 .../{BeamSqlRecordHelper.java => RowHelper.java}   |   6 +-
 .../{BeamRecordSqlType.java => RowSqlType.java}|  20 +--
 .../beam/sdk/extensions/sql/SqlTypeCoder.java  |  14 +--
 .../sdk/extensions/sql/example/BeamSqlExample.java |  30 ++---
 .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java   |   6 +-
 .../interpreter/BeamSqlExpressionExecutor.java |   6 +-
 .../sql/impl/interpreter/BeamSqlFnExecutor.java|   6 +-
 .../operator/BeamSqlCaseExpression.java|   4 +-
 .../operator/BeamSqlCastExpression.java|   4 +-
 .../interpreter/operator/BeamSqlExpression.java|   8 +-
 .../operator/BeamSqlInputRefExpression.java|   4 +-
 .../interpreter/operator/BeamSqlPrimitive.java |   6 +-
 .../interpreter/operator/BeamSqlUdfExpression.java |   4 +-
 .../operator/BeamSqlWindowEndExpression.java   |   4 +-
 .../operator/BeamSqlWindowExpression.java  |   4 +-
 .../operator/BeamSqlWindowStartExpression.java |   4 +-
 .../arithmetic/BeamSqlArithmeticExpression.java|   4 +-
 .../comparison/BeamSqlCompareExpression.java   |   4 +-
 .../comparison/BeamSqlIsNotNullExpression.java |   4 +-
 .../comparison/BeamSqlIsNullExpression.java|   4 +-
 .../date/BeamSqlCurrentDateExpression.java |   4 +-
 .../date/BeamSqlCurrentTimeExpression.java |   4 +-
 .../date/BeamSqlCurrentTimestampExpression.java|   4 +-
 .../operator/date/BeamSqlDateCeilExpression.java   |   4 +-
 .../operator/date/BeamSqlDateFloorExpression.java  |   4 +-
 .../date/BeamSqlDatetimeMinusExpression.java   |   4 +-
 .../date/BeamSqlDatetimePlusExpression.java|   8 +-
 .../operator/date/BeamSqlExtractExpression.java|   4 +-
 .../date/BeamSqlIntervalMultiplyExpression.java|   4 +-
 .../BeamSqlTimestampMinusIntervalExpression.java   |   4 +-
 .../BeamSqlTimestampMinusTimestampExpression.java  |   4 +-
 .../operator/logical/BeamSqlAndExpression.java |   4 +-
 .../operator/logical/BeamSqlNotExpression.java |   4 +-
 .../operator/logical/BeamSqlOrExpression.java  |   4 +-
 .../operator/math/BeamSqlMathBinaryExpression.java |   4 +-
 .../operator/math/BeamSqlMathUnaryExpression.java  |   4 +-
 .../operator/math/BeamSqlPiExpression.java |   4 +-
 .../operator/math/BeamSqlRandExpression.java   |   6 +-
 .../math/BeamSqlRandIntegerExpression.java |   8 +-
 .../reinterpret/BeamSqlReinterpretExpression.java  |   4 +-
 .../string/BeamSqlCharLengthExpression.java|   4 +-
 .../operator/string/BeamSqlConcatExpression.java   |   4 +-
 .../operator/string/BeamSqlInitCapExpression.java  |   4 +-
 .../operator/string/BeamSqlLowerExpression.java|   4 +-
 .../operator/string/BeamSqlOverlayExpression.java  |   4 +-
 .../operator/string/BeamSqlPositionExpression.java |   4 +-
 .../string/BeamSqlSubstringExpression.java |   4 +-
 .../operator/string/BeamSqlTrimExpression.java |   4 +-
 .../operator/string/BeamSqlUpperExpression.java|   4 +-
 .../sql/impl/planner/BeamQueryPlanner.java |   4 +-
 .../sql/impl/rel/BeamAggregationRel.java   |  50 
 .../sdk/extensions/sql/impl/rel/BeamFilterRel.java |  14 ++-
 .../sdk/extensions/sql/impl/rel/BeamIOSinkRel.java |   6 +-
 .../extensions/sql/impl/rel/BeamIOSourceRel.java   |  12 +-
 .../extensions/sql/impl/rel/BeamIntersectRel.java  |   4 +-
 .../sdk/extensions/sql/impl/rel/BeamJoinRel.java   |  90 +++---
 .../sdk/extensions/sql/impl/rel/BeamMinusRel.java  |   4 +-
 .../extensions/sql/impl/rel/BeamProjectRel.java|  10 +-
 .../sdk/extensions/sql/impl/rel/BeamRelNode.java   |   4 +-
 .../sql/impl/rel/BeamSetOperatorRelBase.java   |  16 +--
 .../sdk/extensions/sql/impl/rel/BeamSortRel.java   |  18 +--
 .../sdk/extensions/sql/impl/rel/BeamUnionRel.java  |   4 +-
 .../sdk/extensions/sql/impl/rel/BeamValuesRel.java |  22 ++--
 .../sql/impl/rule/AggregateWindowField.java

[beam] branch master updated (a957246 -> a13c7d9)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a957246  Update Window Into URN Constant Name
 add c4ac00a  Rename BeamRecord -> Row, BeamRecordType -> RowType
 new a13c7d9  Merge pull request #4628: [BEAM-3582] Rename BeamRecord -> 
Row, BeamRecordType -> RowType

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../coders/{BeamRecordCoder.java => RowCoder.java} |  44 +++
 .../beam/sdk/values/{BeamRecord.java => Row.java}  |  54 -
 .../values/{BeamRecordType.java => RowType.java}   |  26 ++--
 .../org/apache/beam/sdk/values/BeamRecordTest.java | 113 -
 .../java/org/apache/beam/sdk/values/RowTest.java   | 113 +
 .../{BeamRecordTypeTest.java => RowTypeTest.java}  |  38 +++---
 .../apache/beam/sdk/extensions/sql/BeamSql.java|  18 +--
 .../apache/beam/sdk/extensions/sql/BeamSqlCli.java |   8 +-
 .../sdk/extensions/sql/BeamSqlSeekableTable.java   |   6 +-
 .../beam/sdk/extensions/sql/BeamSqlTable.java  |  10 +-
 .../{BeamSqlRecordHelper.java => RowHelper.java}   |   6 +-
 .../{BeamRecordSqlType.java => RowSqlType.java}|  20 +--
 .../beam/sdk/extensions/sql/SqlTypeCoder.java  |  14 +--
 .../sdk/extensions/sql/example/BeamSqlExample.java |  30 ++---
 .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java   |   6 +-
 .../interpreter/BeamSqlExpressionExecutor.java |   6 +-
 .../sql/impl/interpreter/BeamSqlFnExecutor.java|   6 +-
 .../operator/BeamSqlCaseExpression.java|   4 +-
 .../operator/BeamSqlCastExpression.java|   4 +-
 .../interpreter/operator/BeamSqlExpression.java|   8 +-
 .../operator/BeamSqlInputRefExpression.java|   4 +-
 .../interpreter/operator/BeamSqlPrimitive.java |   6 +-
 .../interpreter/operator/BeamSqlUdfExpression.java |   4 +-
 .../operator/BeamSqlWindowEndExpression.java   |   4 +-
 .../operator/BeamSqlWindowExpression.java  |   4 +-
 .../operator/BeamSqlWindowStartExpression.java |   4 +-
 .../arithmetic/BeamSqlArithmeticExpression.java|   4 +-
 .../comparison/BeamSqlCompareExpression.java   |   4 +-
 .../comparison/BeamSqlIsNotNullExpression.java |   4 +-
 .../comparison/BeamSqlIsNullExpression.java|   4 +-
 .../date/BeamSqlCurrentDateExpression.java |   4 +-
 .../date/BeamSqlCurrentTimeExpression.java |   4 +-
 .../date/BeamSqlCurrentTimestampExpression.java|   4 +-
 .../operator/date/BeamSqlDateCeilExpression.java   |   4 +-
 .../operator/date/BeamSqlDateFloorExpression.java  |   4 +-
 .../date/BeamSqlDatetimeMinusExpression.java   |   4 +-
 .../date/BeamSqlDatetimePlusExpression.java|   8 +-
 .../operator/date/BeamSqlExtractExpression.java|   4 +-
 .../date/BeamSqlIntervalMultiplyExpression.java|   4 +-
 .../BeamSqlTimestampMinusIntervalExpression.java   |   4 +-
 .../BeamSqlTimestampMinusTimestampExpression.java  |   4 +-
 .../operator/logical/BeamSqlAndExpression.java |   4 +-
 .../operator/logical/BeamSqlNotExpression.java |   4 +-
 .../operator/logical/BeamSqlOrExpression.java  |   4 +-
 .../operator/math/BeamSqlMathBinaryExpression.java |   4 +-
 .../operator/math/BeamSqlMathUnaryExpression.java  |   4 +-
 .../operator/math/BeamSqlPiExpression.java |   4 +-
 .../operator/math/BeamSqlRandExpression.java   |   6 +-
 .../math/BeamSqlRandIntegerExpression.java |   8 +-
 .../reinterpret/BeamSqlReinterpretExpression.java  |   4 +-
 .../string/BeamSqlCharLengthExpression.java|   4 +-
 .../operator/string/BeamSqlConcatExpression.java   |   4 +-
 .../operator/string/BeamSqlInitCapExpression.java  |   4 +-
 .../operator/string/BeamSqlLowerExpression.java|   4 +-
 .../operator/string/BeamSqlOverlayExpression.java  |   4 +-
 .../operator/string/BeamSqlPositionExpression.java |   4 +-
 .../string/BeamSqlSubstringExpression.java |   4 +-
 .../operator/string/BeamSqlTrimExpression.java |   4 +-
 .../operator/string/BeamSqlUpperExpression.java|   4 +-
 .../sql/impl/planner/BeamQueryPlanner.java |   4 +-
 .../sql/impl/rel/BeamAggregationRel.java   |  50 
 .../sdk/extensions/sql/impl/rel/BeamFilterRel.java |  14 ++-
 .../sdk/extensions/sql/impl/rel/BeamIOSinkRel.java |   6 +-
 .../extensions/sql/impl/rel/BeamIOSourceRel.java   |  12 +-
 .../extensions/sql/impl/rel/BeamIntersectRel.java  |   4 +-
 .../sdk/extensions/sql/impl/rel/BeamJoinRel.java   |  90 +++---
 .../sdk/extensions/sql/impl/rel/BeamMinusRel.java  |   4 +-
 .../extensions/sql/impl/rel/BeamProjectRel.java|  10 +-
 .../sdk/extensions/sql/impl/rel/BeamRelNode.java   |   4 +-
 .../sql/impl/rel/BeamSetOperatorRelBase.java   |  16 

[beam] branch master updated (ed32c91 -> a957246)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from ed32c91  Remove unused private variables.
 add 6c8e637  Update Assign Window URN Constant Name
 new a957246  Update Window Into URN Constant Name

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/runners/core/construction/Environments.java  | 2 +-
 .../beam/runners/core/construction/PTransformTranslation.java| 2 +-
 .../beam/runners/core/construction/WindowIntoTranslation.java| 9 +
 .../apache/beam/runners/core/construction/EnvironmentsTest.java  | 4 ++--
 .../apache/beam/runners/direct/TransformEvaluatorRegistry.java   | 4 ++--
 .../beam/runners/flink/FlinkBatchTransformTranslators.java   | 3 ++-
 .../beam/runners/flink/FlinkStreamingTransformTranslators.java   | 3 ++-
 7 files changed, 15 insertions(+), 12 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: Update Window Into URN Constant Name

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a95724651222e01feaca281a8fdc1f75cd722d97
Merge: ed32c91 6c8e637
Author: Lukasz Cwik 
AuthorDate: Wed Feb 7 09:43:38 2018 -0800

Update Window Into URN Constant Name

 .../org/apache/beam/runners/core/construction/Environments.java  | 2 +-
 .../beam/runners/core/construction/PTransformTranslation.java| 2 +-
 .../beam/runners/core/construction/WindowIntoTranslation.java| 9 +
 .../apache/beam/runners/core/construction/EnvironmentsTest.java  | 4 ++--
 .../apache/beam/runners/direct/TransformEvaluatorRegistry.java   | 4 ++--
 .../beam/runners/flink/FlinkBatchTransformTranslators.java   | 3 ++-
 .../beam/runners/flink/FlinkStreamingTransformTranslators.java   | 3 ++-
 7 files changed, 15 insertions(+), 12 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (6b4b8fa -> ed32c91)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6b4b8fa  Merge pull request #4468: [BEAM-3291] Add Kinesis write 
transform
 add 1bff066  Remove unused private variables.
 new ed32c91  Remove unused private variables.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/examples/WindowedWordCountIT.java |  5 -
 .../beam/examples/complete/game/StatefulTeamScoreTest.java |  2 --
 .../runners/core/construction/metrics/MetricFilteringTest.java |  2 --
 .../beam/runners/direct/UnboundedReadEvaluatorFactory.java |  3 ---
 .../org/apache/beam/runners/dataflow/DataflowMetricsTest.java  |  2 --
 .../beam/runners/reference/job/ReferenceRunnerJobServer.java   |  3 ---
 .../core/src/main/java/org/apache/beam/sdk/io/AvroSink.java|  2 --
 .../src/main/java/org/apache/beam/sdk/transforms/Distinct.java |  3 ---
 .../java/org/apache/beam/sdk/coders/DelegateCoderTest.java |  3 ---
 sdks/java/extensions/sketching/pom.xml |  5 -
 .../beam/sdk/extensions/sketching/ApproximateDistinctTest.java |  4 
 .../extensions/sql/meta/provider/text/BeamTextCSVTable.java|  4 
 .../operator/date/BeamSqlDatetimeMinusExpressionTest.java  |  3 ---
 .../BeamSqlArithmeticOperatorsIntegrationTest.java |  1 -
 .../java/org/apache/beam/sdk/io/cassandra/CassandraIO.java |  4 
 .../java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java |  4 
 .../apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java   |  3 ---
 .../beam/sdk/io/gcp/bigquery/WriteGroupedRecordsToFiles.java   |  3 ---
 .../java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java  |  3 ---
 sdks/java/io/jms/pom.xml   | 10 --
 .../io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java |  4 
 .../main/java/org/apache/beam/sdk/nexmark/NexmarkUtils.java|  5 -
 .../main/java/org/apache/beam/sdk/nexmark/queries/Query10.java |  1 -
 23 files changed, 79 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: Remove unused private variables.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit ed32c91c3e2d1c6982a9ff908cd8727aafbd6bbe
Merge: 6b4b8fa 1bff066
Author: Lukasz Cwik 
AuthorDate: Wed Feb 7 09:39:28 2018 -0800

Remove unused private variables.

 .../java/org/apache/beam/examples/WindowedWordCountIT.java |  5 -
 .../beam/examples/complete/game/StatefulTeamScoreTest.java |  2 --
 .../runners/core/construction/metrics/MetricFilteringTest.java |  2 --
 .../beam/runners/direct/UnboundedReadEvaluatorFactory.java |  3 ---
 .../org/apache/beam/runners/dataflow/DataflowMetricsTest.java  |  2 --
 .../beam/runners/reference/job/ReferenceRunnerJobServer.java   |  3 ---
 .../core/src/main/java/org/apache/beam/sdk/io/AvroSink.java|  2 --
 .../src/main/java/org/apache/beam/sdk/transforms/Distinct.java |  3 ---
 .../java/org/apache/beam/sdk/coders/DelegateCoderTest.java |  3 ---
 sdks/java/extensions/sketching/pom.xml |  5 -
 .../beam/sdk/extensions/sketching/ApproximateDistinctTest.java |  4 
 .../extensions/sql/meta/provider/text/BeamTextCSVTable.java|  4 
 .../operator/date/BeamSqlDatetimeMinusExpressionTest.java  |  3 ---
 .../BeamSqlArithmeticOperatorsIntegrationTest.java |  1 -
 .../java/org/apache/beam/sdk/io/cassandra/CassandraIO.java |  4 
 .../java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java |  4 
 .../apache/beam/sdk/io/gcp/bigquery/WriteBundlesToFiles.java   |  3 ---
 .../beam/sdk/io/gcp/bigquery/WriteGroupedRecordsToFiles.java   |  3 ---
 .../java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java  |  3 ---
 sdks/java/io/jms/pom.xml   | 10 --
 .../io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java |  4 
 .../main/java/org/apache/beam/sdk/nexmark/NexmarkUtils.java|  5 -
 .../main/java/org/apache/beam/sdk/nexmark/queries/Query10.java |  1 -
 23 files changed, 79 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Closed] (BEAM-3638) S3 occasionally fails to delete

[Jacob Marble (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacob Marble closed BEAM-3638.
--
Resolution: Invalid

> S3 occasionally fails to delete
> ---
>
> Key: BEAM-3638
> URL: https://issues.apache.org/jira/browse/BEAM-3638
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.3.0
>Reporter: Jacob Marble
>Assignee: Jacob Marble
>Priority: Critical
> Fix For: 2.3.0
>
>
> When AmazonS3Client.deleteObjects() is called, the S3 service occasionally 
> returns a retryable error, and also deletes the object. Beam retries the 
> request, and the S3 service correctly returns a "404 Not Found" error. The 
> job fails needlessly.
> I propose to fix this by logging and allowing "404 Not Found" errors; if the 
> object doesn't exist then the current state is also the desired state. Should 
> have a PR later today.
> The first error looks like this:
> {{java.io.IOException: 
> com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more 
> objects could not be deleted (Service: null; Status Code: 200; Error Code: 
> null; Request ID: [redacted]; S3 Extended Request ID: [redacted]), S3 
> Extended Request ID: [redacted]}}
> Subsequent errors:
> {{java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: Not 
> Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; 
> Request ID: [redacted]; S3 Extended Request ID: [redacted]), S3 Extended 
> Request ID: [redacted]}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3638) S3 occasionally fails to delete

[Jacob Marble (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355751#comment-16355751
 ] 

Jacob Marble commented on BEAM-3638:


I believe this is a mistake. The "subsequent errors" are emitted by 
Filesystems.copy(). Closing this for now, will re-open if I find something 
concrete.

> S3 occasionally fails to delete
> ---
>
> Key: BEAM-3638
> URL: https://issues.apache.org/jira/browse/BEAM-3638
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.3.0
>Reporter: Jacob Marble
>Assignee: Jacob Marble
>Priority: Critical
> Fix For: 2.3.0
>
>
> When AmazonS3Client.deleteObjects() is called, the S3 service occasionally 
> returns a retryable error, and also deletes the object. Beam retries the 
> request, and the S3 service correctly returns a "404 Not Found" error. The 
> job fails needlessly.
> I propose to fix this by logging and allowing "404 Not Found" errors; if the 
> object doesn't exist then the current state is also the desired state. Should 
> have a PR later today.
> The first error looks like this:
> {{java.io.IOException: 
> com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more 
> objects could not be deleted (Service: null; Status Code: 200; Error Code: 
> null; Request ID: [redacted]; S3 Extended Request ID: [redacted]), S3 
> Extended Request ID: [redacted]}}
> Subsequent errors:
> {{java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: Not 
> Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; 
> Request ID: [redacted]; S3 Extended Request ID: [redacted]), S3 Extended 
> Request ID: [redacted]}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[Alexey Romanenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355726#comment-16355726
 ] 

Alexey Romanenko edited comment on BEAM-3605 at 2/7/18 5:11 PM:


Hi [~pawel.kaczmarczyk],

Thank you for joining this thread.

There are several opened jiras related to these flaky tests: 

_ShardReadersPoolTest_:

https://issues.apache.org/jira/browse/BEAM-3598

https://issues.apache.org/jira/browse/BEAM-3599

https://issues.apache.org/jira/browse/BEAM-3605 (this one)

_KinesisReaderTest_

https://issues.apache.org/jira/browse/BEAM-3317 

But potentially, there can be more since some other tests also rely on 
_Thread.sleep()_. So, perhaps, it's just a question of probability how rare 
they would fail.

Do you think that we can avoid using _sleep()_ in these tests because, seems, 
it's not always reliable way for testing multithreading code? 
 I'm ok if you already have a fix for other tests and can fix them.

 


was (Author: aromanenko):
Hi [~pawel.kaczmarczyk],

Thank you for joining this thread.

There are several opened jiras related to these flaky tests: 

_ShardReadersPoolTest_:

https://issues.apache.org/jira/browse/BEAM-3598

https://issues.apache.org/jira/browse/BEAM-3599

https://issues.apache.org/jira/browse/BEAM-3605 (this one)

_KinesisReaderTest_

https://issues.apache.org/jira/browse/BEAM-3317 

But potentially, there can be more since some other tests also rely 
on_Thread.sleep(). S_o, perhaps, it's just a question of probability how rare 
they would fail_._

Do you think that we can avoid using _sleep()_ in these tests because, seems, 
it's not always reliable way for testing multithreading code? 
 I'm ok if you already have a fix for other tests and can fix them

 

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake, sickbay
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> 

[jira] [Comment Edited] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[Alexey Romanenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355726#comment-16355726
 ] 

Alexey Romanenko edited comment on BEAM-3605 at 2/7/18 5:09 PM:


Hi [~pawel.kaczmarczyk],

Thank you for joining this thread.

There are several opened jiras related to these flaky tests: 

_ShardReadersPoolTest_:

https://issues.apache.org/jira/browse/BEAM-3598

https://issues.apache.org/jira/browse/BEAM-3599

https://issues.apache.org/jira/browse/BEAM-3605 (this one)

_KinesisReaderTest_

https://issues.apache.org/jira/browse/BEAM-3317 

But potentially, there can be more since some other tests also rely 
on_Thread.sleep(). S_o, perhaps, it's just a question of probability how rare 
they would fail_._

Do you think that we can avoid using _sleep()_ in these tests because, seems, 
it's not always reliable way for testing multithreading code? 
 I'm ok if you already have a fix for other tests and can fix them

 


was (Author: aromanenko):
There are several opened jiras related to these flaky tests: 

_ShardReadersPoolTest_:

https://issues.apache.org/jira/browse/BEAM-3598

https://issues.apache.org/jira/browse/BEAM-3599

https://issues.apache.org/jira/browse/BEAM-3605 (this one)

_KinesisReaderTest_

https://issues.apache.org/jira/browse/BEAM-3317 

But potentially, there can be more since some other tests also rely 
on_Thread.sleep(). S_o, perhaps, it's just a question of probability how rare 
they would fail_._

Do you think that we can avoid using _sleep()_ in these tests because, seems, 
it's not always reliable way for testing multithreading code? 
I'm ok if you already have a fix for other tests and can fix them

 

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake, sickbay
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> 

[jira] [Commented] (BEAM-3605) Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure

[Alexey Romanenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355726#comment-16355726
 ] 

Alexey Romanenko commented on BEAM-3605:


There are several opened jiras related to these flaky tests: 

_ShardReadersPoolTest_:

https://issues.apache.org/jira/browse/BEAM-3598

https://issues.apache.org/jira/browse/BEAM-3599

https://issues.apache.org/jira/browse/BEAM-3605 (this one)

_KinesisReaderTest_

https://issues.apache.org/jira/browse/BEAM-3317 

But potentially, there can be more since some other tests also rely 
on_Thread.sleep(). S_o, perhaps, it's just a question of probability how rare 
they would fail_._

Do you think that we can avoid using _sleep()_ in these tests because, seems, 
it's not always reliable way for testing multithreading code? 
I'm ok if you already have a fix for other tests and can fix them

 

> Kinesis ShardReadersPoolTest shouldForgetClosedShardIterator failure
> 
>
> Key: BEAM-3605
> URL: https://issues.apache.org/jira/browse/BEAM-3605
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Alexey Romanenko
>Priority: Critical
>  Labels: flake, sickbay
>
> Here's one:
> https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/1758/testReport/junit/org.apache.beam.sdk.io.kinesis/ShardReadersPoolTest/shouldForgetClosedShardIterator/
> Filing all test failures as "Critical" so we can sickbay or fix.
> The Jenkins build will get GC'd so here is the error:
> {code}
> java.lang.AssertionError: 
> Expecting:
>   <["shard1", "shard2"]>
> to contain only:
>   <["shard2"]>
> but the following elements were unexpected:
>   <["shard1"]>
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPoolTest.shouldForgetClosedShardIterator(ShardReadersPoolTest.java:270)
> {code}
> and stderr
> {code}
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:16 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException
> Feb 01, 2018 11:24:19 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard1 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard1
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Kinesis Shard read loop has finished
> Feb 01, 2018 11:24:23 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool 
> readLoop
> INFO: Shard iterator for shard2 shard is closed, finishing the read loop
> org.apache.beam.sdk.io.kinesis.KinesisShardClosedException: Shard iterator 
> reached end of the shard: streamName=null, shardId=shard2
>   at 
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:70)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:121)
>   at 
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:112)
>   at 
> 

[jira] [Commented] (BEAM-3637) HBaseIOTest methods do not clean up tables

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355721#comment-16355721
 ] 

Jean-Baptiste Onofré commented on BEAM-3637:


Yup, I meant that a first fix has been done. Now we need a better one.

> HBaseIOTest methods do not clean up tables
> --
>
> Key: BEAM-3637
> URL: https://issues.apache.org/jira/browse/BEAM-3637
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3638) S3 occasionally fails to delete

[Jacob Marble (JIRA)]

Jacob Marble created BEAM-3638:
--

 Summary: S3 occasionally fails to delete
 Key: BEAM-3638
 URL: https://issues.apache.org/jira/browse/BEAM-3638
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Affects Versions: 2.3.0
Reporter: Jacob Marble
Assignee: Jacob Marble
 Fix For: 2.3.0


When AmazonS3Client.deleteObjects() is called, the S3 service occasionally 
returns a retryable error, and also deletes the object. Beam retries the 
request, and the S3 service correctly returns a "404 Not Found" error. The job 
fails needlessly.

I propose to fix this by logging and allowing "404 Not Found" errors; if the 
object doesn't exist then the current state is also the desired state. Should 
have a PR later today.

The first error looks like this:

{{java.io.IOException: 
com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects 
could not be deleted (Service: null; Status Code: 200; Error Code: null; 
Request ID: [redacted]; S3 Extended Request ID: [redacted]), S3 Extended 
Request ID: [redacted]}}

Subsequent errors:

{{java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: Not 
Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request 
ID: [redacted]; S3 Extended Request ID: [redacted]), S3 Extended Request ID: 
[redacted]}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3637) HBaseIOTest methods do not clean up tables

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355683#comment-16355683
 ] 

Kenneth Knowles commented on BEAM-3637:
---

Actually what I did was split the tests that didn't have stable names. This 
ticket is still valid - the tables created in each test method have different 
names otherwise they fail. It would be a little nicer if they dropped the table.

> HBaseIOTest methods do not clean up tables
> --
>
> Key: BEAM-3637
> URL: https://issues.apache.org/jira/browse/BEAM-3637
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3562) Update to Checkstyle 8.7

[Daniel Kulp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Kulp resolved BEAM-3562.
---
   Resolution: Fixed
Fix Version/s: 2.4.0

Branch was merged.

> Update to Checkstyle 8.7
> 
>
> Key: BEAM-3562
> URL: https://issues.apache.org/jira/browse/BEAM-3562
> Project: Beam
>  Issue Type: Task
>  Components: build-system
>Reporter: Daniel Kulp
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 2.4.0
>
>
> The latest IDE's all incorporate much newer versions of Checkstyle by 
> default.   While older versions can sometimes be used, it's a bit of a 
> hassle.  It would be good to update to Checkstyle 8.7 (latest as of now). 
> This involves two main things:
> 1) Update the checkstyle.xml file to be parseable by the newer checkstyle 
> 2) Update all code to then pass with the new file
>  
> This will likely resolve BEAM-3033, BEAM-3036, and BEAM-3077



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3446) RedisIO non-prefix read operations

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355702#comment-16355702
 ] 

Jean-Baptiste Onofré commented on BEAM-3446:


It sounds good to me.

> RedisIO non-prefix read operations
> --
>
> Key: BEAM-3446
> URL: https://issues.apache.org/jira/browse/BEAM-3446
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Vinay varma
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>
> Read operation in RedisIO is for prefix based look ups. While this can be 
> used for exact key matches as well, the number of operations limits the 
> through put of the function.
> I suggest exposing current readAll operation as readbyprefix and using more 
> simpler operations for readAll functionality.
> ex:
> {code:java}
> String output = jedis.get(element);
> if (output != null) {
> processContext.output(KV.of(element, output));
> }
> {code}
> instead of:
> https://github.com/apache/beam/blob/7d240c0bb171af6868f1a6e95196c9dcfc9ac640/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java#L292



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3446) RedisIO non-prefix read operations

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-3446:
--

Assignee: Jean-Baptiste Onofré  (was: Reuven Lax)

> RedisIO non-prefix read operations
> --
>
> Key: BEAM-3446
> URL: https://issues.apache.org/jira/browse/BEAM-3446
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Vinay varma
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>
> Read operation in RedisIO is for prefix based look ups. While this can be 
> used for exact key matches as well, the number of operations limits the 
> through put of the function.
> I suggest exposing current readAll operation as readbyprefix and using more 
> simpler operations for readAll functionality.
> ex:
> {code:java}
> String output = jedis.get(element);
> if (output != null) {
> processContext.output(KV.of(element, output));
> }
> {code}
> instead of:
> https://github.com/apache/beam/blob/7d240c0bb171af6868f1a6e95196c9dcfc9ac640/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java#L292



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3446) RedisIO non-prefix read operations

[Vinay varma (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355689#comment-16355689
 ] 

Vinay varma commented on BEAM-3446:
---

[~reuvenlax] I could put in a PR for this if you think the idea is fine.

> RedisIO non-prefix read operations
> --
>
> Key: BEAM-3446
> URL: https://issues.apache.org/jira/browse/BEAM-3446
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Vinay varma
>Assignee: Reuven Lax
>Priority: Major
>
> Read operation in RedisIO is for prefix based look ups. While this can be 
> used for exact key matches as well, the number of operations limits the 
> through put of the function.
> I suggest exposing current readAll operation as readbyprefix and using more 
> simpler operations for readAll functionality.
> ex:
> {code:java}
> String output = jedis.get(element);
> if (output != null) {
> processContext.output(KV.of(element, output));
> }
> {code}
> instead of:
> https://github.com/apache/beam/blob/7d240c0bb171af6868f1a6e95196c9dcfc9ac640/sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java#L292



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3637) HBaseIOTest methods do not clean up tables

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355679#comment-16355679
 ] 

Jean-Baptiste Onofré commented on BEAM-3637:


Actually, it's already merged as part of Kenn's last PR.

> HBaseIOTest methods do not clean up tables
> --
>
> Key: BEAM-3637
> URL: https://issues.apache.org/jira/browse/BEAM-3637
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3637) HBaseIOTest methods do not clean up tables

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-3637:
--

Assignee: Jean-Baptiste Onofré  (was: Alexey Romanenko)

> HBaseIOTest methods do not clean up tables
> --
>
> Key: BEAM-3637
> URL: https://issues.apache.org/jira/browse/BEAM-3637
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3637) HBaseIOTest methods do not clean up tables

[Alexey Romanenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Romanenko reassigned BEAM-3637:
--

Assignee: Alexey Romanenko

> HBaseIOTest methods do not clean up tables
> --
>
> Key: BEAM-3637
> URL: https://issues.apache.org/jira/browse/BEAM-3637
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Kenneth Knowles
>Assignee: Alexey Romanenko
>Priority: Minor
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3617) Restore proto round trip for Java DirectRunner (was: Performance degradation on the direct runner)

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3617:
--
Summary: Restore proto round trip for Java DirectRunner (was: Performance 
degradation on the direct runner)  (was: Performance degradation on the direct 
runner)

> Restore proto round trip for Java DirectRunner (was: Performance degradation 
> on the direct runner)
> --
>
> Key: BEAM-3617
> URL: https://issues.apache.org/jira/browse/BEAM-3617
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Jean-Baptiste Onofré
>Assignee: Kenneth Knowles
>Priority: Minor
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
> shows a performance degradation:
> {code}
> 
>  Beam 2.2.0   Beam 2.3.0
>   Query  Runtime(sec) Runtime(sec)
> 
>      6.410.6
>   0001   5.110.2
>   0002   3.0 5.8
>   0003   3.8 6.2
>   0004   0.9 1.4
>   0005   5.811.4
>   0006   0.8 1.4
>   0007 193.8  1249.1
>   0008   3.9 6.9
>   0009   0.9 1.3
>   0010   6.4 8.2
>   0011   5.0 9.4
>   0012   4.7 9.1
> {code}
> We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3617) Performance degradation on the direct runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3617:
--
Priority: Minor  (was: Blocker)

> Performance degradation on the direct runner
> 
>
> Key: BEAM-3617
> URL: https://issues.apache.org/jira/browse/BEAM-3617
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Jean-Baptiste Onofré
>Assignee: Kenneth Knowles
>Priority: Minor
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
> shows a performance degradation:
> {code}
> 
>  Beam 2.2.0   Beam 2.3.0
>   Query  Runtime(sec) Runtime(sec)
> 
>      6.410.6
>   0001   5.110.2
>   0002   3.0 5.8
>   0003   3.8 6.2
>   0004   0.9 1.4
>   0005   5.811.4
>   0006   0.8 1.4
>   0007 193.8  1249.1
>   0008   3.9 6.9
>   0009   0.9 1.3
>   0010   6.4 8.2
>   0011   5.0 9.4
>   0012   4.7 9.1
> {code}
> We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3617) Performance degradation on the direct runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3617:
--
Fix Version/s: (was: 2.3.0)

> Performance degradation on the direct runner
> 
>
> Key: BEAM-3617
> URL: https://issues.apache.org/jira/browse/BEAM-3617
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Jean-Baptiste Onofré
>Assignee: Kenneth Knowles
>Priority: Minor
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
> shows a performance degradation:
> {code}
> 
>  Beam 2.2.0   Beam 2.3.0
>   Query  Runtime(sec) Runtime(sec)
> 
>      6.410.6
>   0001   5.110.2
>   0002   3.0 5.8
>   0003   3.8 6.2
>   0004   0.9 1.4
>   0005   5.811.4
>   0006   0.8 1.4
>   0007 193.8  1249.1
>   0008   3.9 6.9
>   0009   0.9 1.3
>   0010   6.4 8.2
>   0011   5.0 9.4
>   0012   4.7 9.1
> {code}
> We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-3617) Performance degradation on the direct runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reopened BEAM-3617:
---

> Performance degradation on the direct runner
> 
>
> Key: BEAM-3617
> URL: https://issues.apache.org/jira/browse/BEAM-3617
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Jean-Baptiste Onofré
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 2.3.0
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
> shows a performance degradation:
> {code}
> 
>  Beam 2.2.0   Beam 2.3.0
>   Query  Runtime(sec) Runtime(sec)
> 
>      6.410.6
>   0001   5.110.2
>   0002   3.0 5.8
>   0003   3.8 6.2
>   0004   0.9 1.4
>   0005   5.811.4
>   0006   0.8 1.4
>   0007 193.8  1249.1
>   0008   3.9 6.9
>   0009   0.9 1.3
>   0010   6.4 8.2
>   0011   5.0 9.4
>   0012   4.7 9.1
> {code}
> We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3617) Performance degradation on the direct runner

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-3617.

Resolution: Fixed

> Performance degradation on the direct runner
> 
>
> Key: BEAM-3617
> URL: https://issues.apache.org/jira/browse/BEAM-3617
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Jean-Baptiste Onofré
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 2.3.0
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
> shows a performance degradation:
> {code}
> 
>  Beam 2.2.0   Beam 2.3.0
>   Query  Runtime(sec) Runtime(sec)
> 
>      6.410.6
>   0001   5.110.2
>   0002   3.0 5.8
>   0003   3.8 6.2
>   0004   0.9 1.4
>   0005   5.811.4
>   0006   0.8 1.4
>   0007 193.8  1249.1
>   0008   3.9 6.9
>   0009   0.9 1.3
>   0010   6.4 8.2
>   0011   5.0 9.4
>   0012   4.7 9.1
> {code}
> We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3617) Performance degradation on the direct runner

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3617:
---
Fix Version/s: (was: 2.4.0)

> Performance degradation on the direct runner
> 
>
> Key: BEAM-3617
> URL: https://issues.apache.org/jira/browse/BEAM-3617
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Jean-Baptiste Onofré
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: 2.3.0
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Running Nexmark queries with the direct runner between Beam 2.2.0 and 2.3.0 
> shows a performance degradation:
> {code}
> 
>  Beam 2.2.0   Beam 2.3.0
>   Query  Runtime(sec) Runtime(sec)
> 
>      6.410.6
>   0001   5.110.2
>   0002   3.0 5.8
>   0003   3.8 6.2
>   0004   0.9 1.4
>   0005   5.811.4
>   0006   0.8 1.4
>   0007 193.8  1249.1
>   0008   3.9 6.9
>   0009   0.9 1.3
>   0010   6.4 8.2
>   0011   5.0 9.4
>   0012   4.7 9.1
> {code}
> We can see especially Query 7 that is 10 times longer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch release-2.3.0 updated (76b8fd8 -> 23214c2)

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a change to branch release-2.3.0
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 76b8fd8  Merge pull request #4607 from iemejia/release-2.3.0
 add a71bb2e  Revert "Reinstate proto round trip in Java DirectRunner"
 add 358629a  Fix stable name errors in HBaseIOTest
 new 23214c2  Merge pull request #4621 from kennknowles/proto-round-trip

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/runners/direct/DirectOptions.java  |  8 +
 .../apache/beam/runners/direct/DirectRunner.java   | 15 +
 .../org/apache/beam/sdk/io/hbase/HBaseIOTest.java  | 36 --
 3 files changed, 51 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[beam] 01/01: Merge pull request #4621 from kennknowles/proto-round-trip

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch release-2.3.0
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 23214c24eb940d89185a4a0cc229fbcbb29d53d6
Merge: 76b8fd8 358629a
Author: Jean-Baptiste Onofré 
AuthorDate: Wed Feb 7 16:28:41 2018 +0100

Merge pull request #4621 from kennknowles/proto-round-trip

[2.3.0 Cherry-pick][BEAM-3617] Revert "Reinstate proto round trip in Java 
DirectRunner" and fixed masked errors in HBaseIOTest

 .../apache/beam/runners/direct/DirectOptions.java  |  8 +
 .../apache/beam/runners/direct/DirectRunner.java   | 15 +
 .../org/apache/beam/sdk/io/hbase/HBaseIOTest.java  | 36 --
 3 files changed, 51 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5896

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (BEAM-3561) Provide kubernetes cluster instance for IOITs.

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355502#comment-16355502
 ] 

Łukasz Gajowy edited comment on BEAM-3561 at 2/7/18 2:23 PM:
-

[~chamikara] you are right. I was able to connect to a cluster named 
"io-datastores".

There seem to be some pods left. i think those are from the times, when the 
JDBC job was running successfully. I think we should delete them. Jenkins logs: 
 
[https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_JDBC/236/console]

Can you (anybody) confirm that this cluster is not used by anything else so we 
can remove the above-mentioned pods?


was (Author: łukaszg):
[~chamikara] you are right. I was able to connect to a cluster named 
"io-datastores". 

There seem to be some pods left. i think those are from the times, when the 
JDBC job was running successfully. I think we should delete them. Jenkins logs: 
[https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_JDBC/236/console]

Can you (anybody) confirm that this cluster is not used by anything else so we 
can remove the abovementioned pods?

> Provide kubernetes cluster instance for IOITs.
> --
>
> Key: BEAM-3561
> URL: https://issues.apache.org/jira/browse/BEAM-3561
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Jason Kuster
>Priority: Major
>
> Performance tests that require running Kubernetes scripts currently cannot be 
> run on Jenkins. This is due to the fact that there is no dedicated kubernetes 
> cluster for them so Jenkins jobs cannot setup the needed infrastructure 
> anywhere.
> To allow running such tests we should provide an instance of kubernetes 
> cluster (for example a cluster hosted on GKE) and all necessary credentials 
> to connect with it from Jenkins executors (proper kubeconfig file on all 
> Jenkins executors). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3561) Provide kubernetes cluster instance for IOITs.

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355502#comment-16355502
 ] 

Łukasz Gajowy commented on BEAM-3561:
-

[~chamikara] you are right. I was able to connect to a cluster named 
"io-datastores". 

There seem to be some pods left. i think those are from the times, when the 
JDBC job was running successfully. I think we should delete them. Jenkins logs: 
[https://builds.apache.org/view/A-D/view/Beam/job/beam_PerformanceTests_JDBC/236/console]

Can you (anybody) confirm that this cluster is not used by anything else so we 
can remove the abovementioned pods?

> Provide kubernetes cluster instance for IOITs.
> --
>
> Key: BEAM-3561
> URL: https://issues.apache.org/jira/browse/BEAM-3561
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Jason Kuster
>Priority: Major
>
> Performance tests that require running Kubernetes scripts currently cannot be 
> run on Jenkins. This is due to the fact that there is no dedicated kubernetes 
> cluster for them so Jenkins jobs cannot setup the needed infrastructure 
> anywhere.
> To allow running such tests we should provide an instance of kubernetes 
> cluster (for example a cluster hosted on GKE) and all necessary credentials 
> to connect with it from Jenkins executors (proper kubeconfig file on all 
> Jenkins executors). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3291) Add Kinesis Write transform

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-3291.

   Resolution: Fixed
Fix Version/s: 2.4.0

> Add Kinesis Write transform
> ---
>
> Key: BEAM-3291
> URL: https://issues.apache.org/jira/browse/BEAM-3291
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: Not applicable
>Reporter: Ismaël Mejía
>Assignee: Alexey Romanenko
>Priority: Minor
> Fix For: 2.4.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Currently KinesisIO only has a Read transform, we need to provide a Write too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4468: [BEAM-3291] Add Kinesis write transform

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6b4b8fa07e4e88f45ec1d4a4c692cc3f6fb545ac
Merge: 30dbf5c 2baeeb9
Author: Ismaël Mejía 
AuthorDate: Wed Feb 7 15:02:27 2018 +0100

Merge pull request #4468: [BEAM-3291] Add Kinesis write transform

 sdks/java/io/kinesis/build.gradle  |   1 +
 sdks/java/io/kinesis/pom.xml   |  34 +-
 .../beam/sdk/io/kinesis/AWSClientsProvider.java|   9 +-
 .../beam/sdk/io/kinesis/BasicKinesisProvider.java  |  90 +
 .../org/apache/beam/sdk/io/kinesis/KinesisIO.java  | 446 ++---
 ...lientsProvider.java => KinesisPartitioner.java} |  16 +-
 .../beam/sdk/io/kinesis/AmazonKinesisMock.java |  11 +
 .../apache/beam/sdk/io/kinesis/KinesisIOIT.java| 126 ++
 .../beam/sdk/io/kinesis/KinesisMockReadTest.java   |   2 +-
 .../beam/sdk/io/kinesis/KinesisMockWriteTest.java  | 289 +
 .../beam/sdk/io/kinesis/KinesisProducerMock.java   | 120 ++
 .../beam/sdk/io/kinesis/KinesisReaderIT.java   | 121 --
 .../beam/sdk/io/kinesis/KinesisServiceMock.java|  88 
 13 files changed, 1159 insertions(+), 194 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[beam] branch master updated (30dbf5c -> 6b4b8fa)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 30dbf5c  Merge pull request #4627 from kennknowles/revertAndHBase
 add 2baeeb9  [BEAM-3291] Add Kinesis write transform
 new 6b4b8fa  Merge pull request #4468: [BEAM-3291] Add Kinesis write 
transform

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/io/kinesis/build.gradle  |   1 +
 sdks/java/io/kinesis/pom.xml   |  34 +-
 .../beam/sdk/io/kinesis/AWSClientsProvider.java|   9 +-
 .../beam/sdk/io/kinesis/BasicKinesisProvider.java  |  90 +
 .../org/apache/beam/sdk/io/kinesis/KinesisIO.java  | 446 ++---
 ...losedException.java => KinesisPartitioner.java} |  13 +-
 .../beam/sdk/io/kinesis/AmazonKinesisMock.java |  11 +
 .../apache/beam/sdk/io/kinesis/KinesisIOIT.java| 126 ++
 .../beam/sdk/io/kinesis/KinesisMockReadTest.java   |   2 +-
 .../beam/sdk/io/kinesis/KinesisMockWriteTest.java  | 289 +
 .../beam/sdk/io/kinesis/KinesisProducerMock.java   | 120 ++
 .../beam/sdk/io/kinesis/KinesisReaderIT.java   | 121 --
 .../beam/sdk/io/kinesis/KinesisServiceMock.java|  88 
 13 files changed, 1161 insertions(+), 189 deletions(-)
 create mode 100644 
sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/BasicKinesisProvider.java
 copy 
sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/{KinesisShardClosedException.java
 => KinesisPartitioner.java} (79%)
 create mode 100644 
sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisIOIT.java
 create mode 100644 
sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisMockWriteTest.java
 create mode 100644 
sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisProducerMock.java
 delete mode 100644 
sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisReaderIT.java
 create mode 100644 
sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisServiceMock.java

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[jira] [Assigned] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-3409:
--

Assignee: Romain Manni-Bucau  (was: Jean-Baptiste Onofré)

> Unexpected behavior of DoFn teardown method running in unit tests 
> --
>
> Key: BEAM-3409
> URL: https://issues.apache.org/jira/browse/BEAM-3409
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.3.0
>Reporter: Alexey Romanenko
>Assignee: Romain Manni-Bucau
>Priority: Minor
>  Labels: test
>
> Writing a unit test, I found out a strange behaviour of Teardown method of 
> DoFn implementation when I run this method in unit tests using TestPipeline.
> To be more precise, it doesn’t wait until teardown() method will be finished, 
> it just exits from this method after about 1 sec (on my machine) even if it 
> should take longer (very simple example - running infinite loop inside this 
> method or put thread in sleep). In the same time, when I run the same code 
> from main() with ordinary Pipeline and direct runner, then it’s ok and it 
> works as expected - teardown() method will be performed completely despite 
> how much time it will take.
> I created two test cases to reproduce this issue - the first one to run with 
> main() and the second one to run with junit. They use the same implementation 
> of DoFn (class LongTearDownFn) and expects that teardown method will be 
> running at least for SLEEP_TIME ms. In case of running as junit test it's not 
> a case (see output log).
> - run with main()
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/main/java/TearDown.java
> - run with junit
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/test/java/TearDownTest.java



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3409) Unexpected behavior of DoFn teardown method running in unit tests

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3409:
---
Component/s: (was: testing)

> Unexpected behavior of DoFn teardown method running in unit tests 
> --
>
> Key: BEAM-3409
> URL: https://issues.apache.org/jira/browse/BEAM-3409
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.3.0
>Reporter: Alexey Romanenko
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Labels: test
>
> Writing a unit test, I found out a strange behaviour of Teardown method of 
> DoFn implementation when I run this method in unit tests using TestPipeline.
> To be more precise, it doesn’t wait until teardown() method will be finished, 
> it just exits from this method after about 1 sec (on my machine) even if it 
> should take longer (very simple example - running infinite loop inside this 
> method or put thread in sleep). In the same time, when I run the same code 
> from main() with ordinary Pipeline and direct runner, then it’s ok and it 
> works as expected - teardown() method will be performed completely despite 
> how much time it will take.
> I created two test cases to reproduce this issue - the first one to run with 
> main() and the second one to run with junit. They use the same implementation 
> of DoFn (class LongTearDownFn) and expects that teardown method will be 
> running at least for SLEEP_TIME ms. In case of running as junit test it's not 
> a case (see output log).
> - run with main()
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/main/java/TearDown.java
> - run with junit
> https://github.com/aromanenko-dev/beam-samples/blob/master/runners-tests/src/test/java/TearDownTest.java



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4627 from kennknowles/revertAndHBase

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 30dbf5c93872e6ab463add400e2e7e52215e5bec
Merge: 92a4202 5acb913
Author: Jean-Baptiste Onofré 
AuthorDate: Wed Feb 7 14:54:07 2018 +0100

Merge pull request #4627 from kennknowles/revertAndHBase

[BEAM-3617] This closes #4627

 .../apache/beam/runners/direct/DirectOptions.java  |  8 +
 .../apache/beam/runners/direct/DirectRunner.java   | 15 +
 .../org/apache/beam/sdk/io/hbase/HBaseIOTest.java  | 36 --
 3 files changed, 51 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[beam] branch master updated (92a4202 -> 30dbf5c)

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 92a4202  Merge pull request #4620: Add missing gradle build config for 
sdks/java/extensions/sketching
 add b70b024  Revert "Reinstate proto round trip in Java DirectRunner"
 add 5acb913  Fix stable name errors in HBaseIOTest
 new 30dbf5c  Merge pull request #4627 from kennknowles/revertAndHBase

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/runners/direct/DirectOptions.java  |  8 +
 .../apache/beam/runners/direct/DirectRunner.java   | 15 +
 .../org/apache/beam/sdk/io/hbase/HBaseIOTest.java  | 36 --
 3 files changed, 51 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.