Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #2557

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Dataflow #2219

[Apache Jenkins Server]

See 


Changes:

[younghee.kwon] Improve DirectRunner performance by tuning BoundedReadEvaluator.

[altay] Removes deprecated fileio.TextFileSink.

--
[...truncated 27 lines...]
Modules changed, recalculating dependency graph
Established TCP socket on 57813
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_RunnableOnService_Dataflow] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 57813
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 -B -e clean verify -am -pl runners/google-cloud-dataflow-java -DforkCount=0 
-DrunnableOnServicePipelineOptions=[ 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner", 
"--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-runnable-on-service-tests/" ]
2017-02-07T07:29:11.313 [INFO] Error stacktraces are turned on.
2017-02-07T07:29:11.415 [INFO] Scanning for projects...
2017-02-07T07:29:12.564 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
2017-02-07T07:29:12.966 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
 (7 KB at 14.8 KB/sec)
2017-02-07T07:29:12.981 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
2017-02-07T07:29:13.018 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
 (7 KB at 168.8 KB/sec)
2017-02-07T07:29:13.034 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
2017-02-07T07:29:13.068 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
 (4 KB at 97.1 KB/sec)
2017-02-07T07:29:13.071 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
2017-02-07T07:29:13.121 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
 (23 KB at 450.2 KB/sec)
2017-02-07T07:29:13.124 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
2017-02-07T07:29:13.163 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
 (32 KB at 816.0 KB/sec)
2017-02-07T07:29:13.168 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom
2017-02-07T07:29:13.202 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom (14 KB 
at 401.4 KB/sec)
2017-02-07T07:29:13.215 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
2017-02-07T07:29:13.246 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
 (5 KB at 126.4 KB/sec)
2017-02-07T07:29:13.255 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
2017-02-07T07:29:13.288 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
 (4 KB at 100.5 KB/sec)
2017-02-07T07:29:13.292 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
2017-02-07T07:29:13.329 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
 (20 KB at 539.6 KB/sec)
2017-02-07T07:29:13.334 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
2017-02-07T07:29:13.365 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
 (7 KB at 212.9 KB/sec)
2017-02-07T07:29:13.369 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
2017-02-07T07:29:13.404 [INFO] Downloaded: 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Flink #1562

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Spark #870

[Apache Jenkins Server]

See 


Changes:

[altay] Removes deprecated fileio.TextFileSink.

--
[...truncated 26 lines...]
Modules changed, recalculating dependency graph
Established TCP socket on 42320
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_RunnableOnService_Spark] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 42320
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 -B -e clean verify -am -pl runners/spark -Prunnable-on-service-tests 
-Plocal-runnable-on-service-tests -Dspark.port.maxRetries=64 
-Dspark.ui.enabled=false
2017-02-07T06:59:29.922 [INFO] Error stacktraces are turned on.
2017-02-07T06:59:30.003 [INFO] Scanning for projects...
2017-02-07T06:59:30.884 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
2017-02-07T06:59:31.246 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
 (7 KB at 16.4 KB/sec)
2017-02-07T06:59:31.259 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
2017-02-07T06:59:31.301 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
 (7 KB at 152.7 KB/sec)
2017-02-07T06:59:31.315 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
2017-02-07T06:59:31.352 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
 (4 KB at 91.7 KB/sec)
2017-02-07T06:59:31.354 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
2017-02-07T06:59:31.409 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
 (23 KB at 408.5 KB/sec)
2017-02-07T06:59:31.413 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
2017-02-07T06:59:31.455 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
 (32 KB at 757.7 KB/sec)
2017-02-07T06:59:31.463 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom
2017-02-07T06:59:31.497 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom (14 KB 
at 413.5 KB/sec)
2017-02-07T06:59:31.507 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
2017-02-07T06:59:31.538 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
 (5 KB at 130.4 KB/sec)
2017-02-07T06:59:31.544 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
2017-02-07T06:59:31.575 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
 (4 KB at 106.9 KB/sec)
2017-02-07T06:59:31.578 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
2017-02-07T06:59:31.616 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
 (20 KB at 525.4 KB/sec)
2017-02-07T06:59:31.620 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
2017-02-07T06:59:31.652 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
 (7 KB at 206.3 KB/sec)
2017-02-07T06:59:31.655 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
2017-02-07T06:59:31.690 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
 (14 KB at 389.5 KB/sec)
2017-02-07T06:59:31.696 [INFO] Downloading: 

[jira] [Commented] (BEAM-1406) Remove deprecated fileio.TextFileSink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855448#comment-15855448
 ] 

ASF GitHub Bot commented on BEAM-1406:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1932


> Remove deprecated fileio.TextFileSink
> -
>
> Key: BEAM-1406
> URL: https://issues.apache.org/jira/browse/BEAM-1406
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> Users should be using textio.WriteToText() instead of fileio.TextFileSink.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1932: [BEAM-1406] Removes deprecated fileio.TextFileSink

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1932


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1932

[altay]

This closes #1932


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9dad73c2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9dad73c2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9dad73c2

Branch: refs/heads/master
Commit: 9dad73c29d2d939074f9597accbec8c8aa59830d
Parents: 2328697 a164323
Author: Ahmet Altay 
Authored: Mon Feb 6 22:46:09 2017 -0800
Committer: Ahmet Altay 
Committed: Mon Feb 6 22:46:09 2017 -0800

--
 sdks/python/apache_beam/__init__.py  |  4 +-
 sdks/python/apache_beam/io/fileio.py | 77 ---
 sdks/python/apache_beam/io/textio.py | 70 ++--
 3 files changed, 69 insertions(+), 82 deletions(-)
--

[1/2] beam git commit: Removes deprecated fileio.TextFileSink.

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 232869747 -> 9dad73c29


Removes deprecated fileio.TextFileSink.

Users should be using textio.WriteToText() transform instead of
fileio.TextFileSink.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a1643232
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a1643232
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a1643232

Branch: refs/heads/master
Commit: a164323266f8e8548466bb6de535c90b7d0bd8bc
Parents: 2328697
Author: Chamikara Jayalath 
Authored: Mon Feb 6 16:01:11 2017 -0800
Committer: Ahmet Altay 
Committed: Mon Feb 6 22:46:04 2017 -0800

--
 sdks/python/apache_beam/__init__.py  |  4 +-
 sdks/python/apache_beam/io/fileio.py | 77 ---
 sdks/python/apache_beam/io/textio.py | 70 ++--
 3 files changed, 69 insertions(+), 82 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a1643232/sdks/python/apache_beam/__init__.py
--
diff --git a/sdks/python/apache_beam/__init__.py 
b/sdks/python/apache_beam/__init__.py
index eed251b..ce382b8 100644
--- a/sdks/python/apache_beam/__init__.py
+++ b/sdks/python/apache_beam/__init__.py
@@ -53,8 +53,8 @@ After this import statement
 
 * transform classes are available as beam.FlatMap, beam.GroupByKey, etc.
 * Pipeline class is available as beam.Pipeline
-* text source/sink classes are available as beam.io.TextFileSource,
-  beam.io.TextFileSink
+* text read/write transforms are available as beam.io.ReadfromText,
+  beam.io.WriteToText
 
 Examples
 

http://git-wip-us.apache.org/repos/asf/beam/blob/a1643232/sdks/python/apache_beam/io/fileio.py
--
diff --git a/sdks/python/apache_beam/io/fileio.py 
b/sdks/python/apache_beam/io/fileio.py
index 97cf387..59fabb3 100644
--- a/sdks/python/apache_beam/io/fileio.py
+++ b/sdks/python/apache_beam/io/fileio.py
@@ -28,15 +28,12 @@ import shutil
 import time
 import zlib
 
-from apache_beam import coders
 from apache_beam.internal import util
 from apache_beam.io import gcsio
 from apache_beam.io import iobase
 from apache_beam.transforms.display import DisplayDataItem
 
 
-__all__ = ['TextFileSink']
-
 DEFAULT_SHARD_NAME_TEMPLATE = '-S-of-N'
 
 
@@ -721,77 +718,3 @@ class FileSinkWriter(iobase.Writer):
   def close(self):
 self.sink.close(self.temp_handle)
 return self.temp_shard_path
-
-
-class TextFileSink(FileSink):
-  """A sink to a GCS or local text file or files."""
-
-  def __init__(self,
-   file_path_prefix,
-   file_name_suffix='',
-   append_trailing_newlines=True,
-   num_shards=0,
-   shard_name_template=None,
-   coder=coders.ToStringCoder(),
-   compression_type=CompressionTypes.AUTO):
-"""Initialize a TextFileSink.
-
-Args:
-  file_path_prefix: The file path to write to. The files written will begin
-with this prefix, followed by a shard identifier (see num_shards), and
-end in a common extension, if given by file_name_suffix. In most cases,
-only this argument is specified and num_shards, shard_name_template, 
and
-file_name_suffix use default values.
-  file_name_suffix: Suffix for the files written.
-  append_trailing_newlines: indicate whether this sink should write an
-additional newline char after writing each element.
-  num_shards: The number of files (shards) used for output. If not set, the
-service will decide on the optimal number of shards.
-Constraining the number of shards is likely to reduce
-the performance of a pipeline.  Setting this value is not recommended
-unless you require a specific number of output files.
-  shard_name_template: A template string containing placeholders for
-the shard number and shard count. Currently only '' and
-'-S-of-N' are patterns accepted by the service.
-When constructing a filename for a particular shard number, the
-upper-case letters 'S' and 'N' are replaced with the 0-padded shard
-number and shard count respectively.  This argument can be '' in which
-case it behaves as if num_shards was set to 1 and only one file will be
-generated. The default pattern used is '-S-of-N'.
-  coder: Coder used to encode each line.
-  compression_type: Used to handle compressed output files. Typical value
-  is CompressionTypes.AUTO, in which case the final file path's
-  extension (as determined by file_path_prefix, file_name_suffix,
-  num_shards and 

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Spark #869

[Apache Jenkins Server]

See 


Changes:

[younghee.kwon] Improve DirectRunner performance by tuning BoundedReadEvaluator.

--
[...truncated 26 lines...]
Modules changed, recalculating dependency graph
Established TCP socket on 49819
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_RunnableOnService_Spark] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 49819
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 -B -e clean verify -am -pl runners/spark -Prunnable-on-service-tests 
-Plocal-runnable-on-service-tests -Dspark.port.maxRetries=64 
-Dspark.ui.enabled=false
2017-02-07T06:47:21.219 [INFO] Error stacktraces are turned on.
2017-02-07T06:47:21.334 [INFO] Scanning for projects...
2017-02-07T06:47:22.512 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
2017-02-07T06:47:22.856 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
 (7 KB at 17.4 KB/sec)
2017-02-07T06:47:22.867 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
2017-02-07T06:47:22.908 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
 (7 KB at 156.5 KB/sec)
2017-02-07T06:47:22.923 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
2017-02-07T06:47:22.959 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
 (4 KB at 91.7 KB/sec)
2017-02-07T06:47:22.962 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
2017-02-07T06:47:23.017 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
 (23 KB at 401.1 KB/sec)
2017-02-07T06:47:23.024 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
2017-02-07T06:47:23.066 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
 (32 KB at 757.7 KB/sec)
2017-02-07T06:47:23.072 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom
2017-02-07T06:47:23.106 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom (14 KB 
at 413.5 KB/sec)
2017-02-07T06:47:23.115 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
2017-02-07T06:47:23.147 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
 (5 KB at 126.4 KB/sec)
2017-02-07T06:47:23.154 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
2017-02-07T06:47:23.186 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
 (4 KB at 103.6 KB/sec)
2017-02-07T06:47:23.189 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
2017-02-07T06:47:23.225 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
 (20 KB at 554.6 KB/sec)
2017-02-07T06:47:23.230 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
2017-02-07T06:47:23.260 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
 (7 KB at 220.0 KB/sec)
2017-02-07T06:47:23.263 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
2017-02-07T06:47:23.296 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
 (14 KB at 401.3 KB/sec)
2017-02-07T06:47:23.302 [INFO] Downloading: 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2556

[Apache Jenkins Server]

See 


Changes:

[younghee.kwon] Improve DirectRunner performance by tuning BoundedReadEvaluator.

--
[...truncated 26 lines...]
Modules changed, recalculating dependency graph
Established TCP socket on 45655
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_MavenInstall] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 45655
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 
 
-Dmaven.repo.local=
 -B -e -P release,dataflow-runner clean install coveralls:report 
-DrepoToken= -DskipITs=false -DintegrationTestPipelineOptions=[ 
"--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests", 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]
2017-02-07T06:47:02.691 [INFO] Error stacktraces are turned on.
2017-02-07T06:47:02.783 [INFO] Scanning for projects...
2017-02-07T06:47:04.825 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
2017-02-07T06:47:05.410 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
 (7 KB at 10.2 KB/sec)
2017-02-07T06:47:05.449 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
2017-02-07T06:47:05.501 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
 (7 KB at 116.6 KB/sec)
2017-02-07T06:47:05.546 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
2017-02-07T06:47:05.594 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
 (4 KB at 68.8 KB/sec)
2017-02-07T06:47:05.598 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
2017-02-07T06:47:05.665 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
 (23 KB at 329.2 KB/sec)
2017-02-07T06:47:05.675 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
2017-02-07T06:47:05.738 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
 (32 KB at 505.1 KB/sec)
2017-02-07T06:47:05.746 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom
2017-02-07T06:47:05.790 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom (14 KB 
at 310.1 KB/sec)
2017-02-07T06:47:05.801 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
2017-02-07T06:47:05.841 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
 (5 KB at 101.1 KB/sec)
2017-02-07T06:47:05.851 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
2017-02-07T06:47:05.894 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
 (4 KB at 77.1 KB/sec)
2017-02-07T06:47:05.898 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
2017-02-07T06:47:05.941 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
 (20 KB at 475.4 KB/sec)
2017-02-07T06:47:05.945 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
2017-02-07T06:47:06.001 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
 (7 KB at 117.9 KB/sec)
2017-02-07T06:47:06.005 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
2017-02-07T06:47:06.041 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
 (14 KB at 

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Flink #1561

[Apache Jenkins Server]

See 


Changes:

[younghee.kwon] Improve DirectRunner performance by tuning BoundedReadEvaluator.

--
[...truncated 26 lines...]
Modules changed, recalculating dependency graph
Established TCP socket on 34776
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_RunnableOnService_Flink] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 34776
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 -B -e clean verify -am -pl runners/flink/runner 
-Plocal-runnable-on-service-tests -Prunnable-on-service-tests
2017-02-07T06:46:58.418 [INFO] Error stacktraces are turned on.
2017-02-07T06:46:58.538 [INFO] Scanning for projects...
2017-02-07T06:47:00.463 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
2017-02-07T06:47:00.928 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.4.0.Final/os-maven-plugin-1.4.0.Final.pom
 (7 KB at 12.8 KB/sec)
2017-02-07T06:47:00.950 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
2017-02-07T06:47:00.997 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
 (7 KB at 128.3 KB/sec)
2017-02-07T06:47:01.030 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
2017-02-07T06:47:01.076 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
 (4 KB at 71.8 KB/sec)
2017-02-07T06:47:01.083 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
2017-02-07T06:47:01.146 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
 (23 KB at 350.2 KB/sec)
2017-02-07T06:47:01.153 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
2017-02-07T06:47:01.207 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
 (32 KB at 589.3 KB/sec)
2017-02-07T06:47:01.217 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom
2017-02-07T06:47:01.259 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom (14 KB 
at 324.9 KB/sec)
2017-02-07T06:47:01.282 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
2017-02-07T06:47:01.317 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
 (5 KB at 115.5 KB/sec)
2017-02-07T06:47:01.334 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
2017-02-07T06:47:01.370 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.17/plexus-utils-3.0.17.pom
 (4 KB at 92.1 KB/sec)
2017-02-07T06:47:01.377 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
2017-02-07T06:47:01.420 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.3.1/plexus-3.3.1.pom
 (20 KB at 453.8 KB/sec)
2017-02-07T06:47:01.427 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
2017-02-07T06:47:01.465 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/spice/spice-parent/17/spice-parent-17.pom
 (7 KB at 169.2 KB/sec)
2017-02-07T06:47:01.477 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
2017-02-07T06:47:01.514 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/10/forge-parent-10.pom
 (14 KB at 367.8 KB/sec)
2017-02-07T06:47:01.521 [INFO] Downloading: 

[jira] [Commented] (BEAM-1410) Reduce sdk-py DirectRunner running time and memory consumption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855435#comment-15855435
 ] 

ASF GitHub Bot commented on BEAM-1410:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1933


> Reduce sdk-py DirectRunner running time and memory consumption
> --
>
> Key: BEAM-1410
> URL: https://issues.apache.org/jira/browse/BEAM-1410
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Younghee Kwon
>Assignee: Ahmet Altay
>Priority: Minor
>  Labels: performance, python
>
> Some experimental benchmarks shows that DirectRunner can improve performance 
> in cpu and memory. 
> I will roll out some CLs to improve them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1933: [BEAM-1410] Improve DirectRunner performance by tun...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1933


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Improve DirectRunner performance by tuning BoundedReadEvaluator.

[altay]

Repository: beam
Updated Branches:
  refs/heads/master f90558c34 -> 232869747


Improve DirectRunner performance by tuning BoundedReadEvaluator.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d34145ad
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d34145ad
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d34145ad

Branch: refs/heads/master
Commit: d34145adf81dd50aa00d8e968fb2843bb624b23c
Parents: f90558c
Author: Younghee Kwon 
Authored: Mon Feb 6 21:49:49 2017 -0800
Committer: Younghee Kwon 
Committed: Mon Feb 6 21:49:49 2017 -0800

--
 sdks/python/apache_beam/runners/direct/transform_evaluator.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d34145ad/sdks/python/apache_beam/runners/direct/transform_evaluator.py
--
diff --git a/sdks/python/apache_beam/runners/direct/transform_evaluator.py 
b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
index 1700de6..6ae5697 100644
--- a/sdks/python/apache_beam/runners/direct/transform_evaluator.py
+++ b/sdks/python/apache_beam/runners/direct/transform_evaluator.py
@@ -175,7 +175,8 @@ class _TransformEvaluator(object):
 class _BoundedReadEvaluator(_TransformEvaluator):
   """TransformEvaluator for bounded Read transform."""
 
-  MAX_ELEMENT_PER_BUNDLE = 100
+  # After some benchmarks, 1000 was optimal among {100,1000,1}
+  MAX_ELEMENT_PER_BUNDLE = 1000
 
   def __init__(self, evaluation_context, applied_ptransform,
input_committed_bundle, side_inputs, scoped_metrics_container):

[jira] [Commented] (BEAM-1410) Reduce sdk-py DirectRunner running time and memory consumption

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855386#comment-15855386
 ] 

ASF GitHub Bot commented on BEAM-1410:
--

GitHub user yk5 opened a pull request:

https://github.com/apache/beam/pull/1933

[BEAM-1410] Improve DirectRunner performance by tuning BoundedReadEvaluator.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yk5/beam performance

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1933.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1933


commit d34145adf81dd50aa00d8e968fb2843bb624b23c
Author: Younghee Kwon 
Date:   2017-02-07T05:49:49Z

Improve DirectRunner performance by tuning BoundedReadEvaluator.




> Reduce sdk-py DirectRunner running time and memory consumption
> --
>
> Key: BEAM-1410
> URL: https://issues.apache.org/jira/browse/BEAM-1410
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Younghee Kwon
>Assignee: Ahmet Altay
>Priority: Minor
>  Labels: performance, python
>
> Some experimental benchmarks shows that DirectRunner can improve performance 
> in cpu and memory. 
> I will roll out some CLs to improve them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1933: [BEAM-1410] Improve DirectRunner performance by tun...

[yk5]

GitHub user yk5 opened a pull request:

https://github.com/apache/beam/pull/1933

[BEAM-1410] Improve DirectRunner performance by tuning BoundedReadEvaluator.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yk5/beam performance

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1933.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1933


commit d34145adf81dd50aa00d8e968fb2843bb624b23c
Author: Younghee Kwon 
Date:   2017-02-07T05:49:49Z

Improve DirectRunner performance by tuning BoundedReadEvaluator.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1410) Reduce sdk-py DirectRunner running time and memory consumption

[Younghee Kwon (JIRA)]

Younghee Kwon created BEAM-1410:
---

 Summary: Reduce sdk-py DirectRunner running time and memory 
consumption
 Key: BEAM-1410
 URL: https://issues.apache.org/jira/browse/BEAM-1410
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Younghee Kwon
Assignee: Ahmet Altay
Priority: Minor


Some experimental benchmarks shows that DirectRunner can improve performance in 
cpu and memory. 

I will roll out some CLs to improve them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1331) Add documentation from reading compressed text sources

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855301#comment-15855301
 ] 

ASF GitHub Bot commented on BEAM-1331:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1845


> Add documentation from reading compressed text sources
> --
>
> Key: BEAM-1331
> URL: https://issues.apache.org/jira/browse/BEAM-1331
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Trivial
> Fix For: Not applicable
>
>
> Add a snippet for reading from compressed text sources.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: Closes #1845

[robertwb]

Closes #1845


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f90558c3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f90558c3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f90558c3

Branch: refs/heads/master
Commit: f90558c34ec90c1f43b119b160971deddc8b2c61
Parents: a26fd1f 98e513b
Author: Robert Bradshaw 
Authored: Mon Feb 6 20:28:04 2017 -0800
Committer: Robert Bradshaw 
Committed: Mon Feb 6 20:28:04 2017 -0800

--
 .../apache_beam/examples/snippets/snippets.py   | 20 --
 .../examples/snippets/snippets_test.py  | 29 
 2 files changed, 40 insertions(+), 9 deletions(-)
--

[1/2] beam git commit: Add snippet for reading from compressed text sources

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master a26fd1ff3 -> f90558c34


Add snippet for reading from compressed text sources


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/98e513bc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/98e513bc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/98e513bc

Branch: refs/heads/master
Commit: 98e513bc64be3241d8a97902cba4808eec384a4b
Parents: a26fd1f
Author: Sourabh Bajaj 
Authored: Mon Feb 6 17:23:14 2017 -0800
Committer: Robert Bradshaw 
Committed: Mon Feb 6 20:28:03 2017 -0800

--
 .../apache_beam/examples/snippets/snippets.py   | 20 --
 .../examples/snippets/snippets_test.py  | 29 
 2 files changed, 40 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/98e513bc/sdks/python/apache_beam/examples/snippets/snippets.py
--
diff --git a/sdks/python/apache_beam/examples/snippets/snippets.py 
b/sdks/python/apache_beam/examples/snippets/snippets.py
index 42d194e..c3879dc 100644
--- a/sdks/python/apache_beam/examples/snippets/snippets.py
+++ b/sdks/python/apache_beam/examples/snippets/snippets.py
@@ -828,8 +828,7 @@ def model_textio(renames):
   # [START model_textio_read]
   p = beam.Pipeline(options=PipelineOptions())
   # [START model_pipelineio_read]
-  lines = p | 'ReadFromText' >> beam.io.ReadFromText(
-  'gs://my_bucket/path/to/input-*.csv')
+  lines = p | 'ReadFromText' >> beam.io.ReadFromText('path/to/input-*.csv')
   # [END model_pipelineio_read]
   # [END model_textio_read]
 
@@ -837,7 +836,7 @@ def model_textio(renames):
   filtered_words = lines | 'FilterWords' >> beam.FlatMap(filter_words)
   # [START model_pipelineio_write]
   filtered_words | 'WriteToText' >> beam.io.WriteToText(
-  'gs://my_bucket/path/to/numbers', file_name_suffix='.csv')
+  '/path/to/numbers', file_name_suffix='.csv')
   # [END model_pipelineio_write]
   # [END model_textio_write]
 
@@ -845,6 +844,21 @@ def model_textio(renames):
   p.run().wait_until_finish()
 
 
+def model_textio_compressed(renames, expected):
+  """Using a Read Transform to read compressed text files."""
+  p = TestPipeline()
+
+  # [START model_textio_write_compressed]
+  lines = p | 'ReadFromText' >> beam.io.ReadFromText(
+  '/path/to/input-*.csv.gz',
+  compression_type=beam.io.fileio.CompressionTypes.GZIP)
+  # [END model_textio_write_compressed]
+
+  beam.assert_that(lines, beam.equal_to(expected))
+  p.visit(SnippetUtils.RenameFiles(renames))
+  p.run().wait_until_finish()
+
+
 def model_datastoreio():
   """Using a Read and Write transform to read/write to Cloud Datastore."""
 

http://git-wip-us.apache.org/repos/asf/beam/blob/98e513bc/sdks/python/apache_beam/examples/snippets/snippets_test.py
--
diff --git a/sdks/python/apache_beam/examples/snippets/snippets_test.py 
b/sdks/python/apache_beam/examples/snippets/snippets_test.py
index a602b66..4827e94 100644
--- a/sdks/python/apache_beam/examples/snippets/snippets_test.py
+++ b/sdks/python/apache_beam/examples/snippets/snippets_test.py
@@ -18,6 +18,7 @@
 """Tests for all code snippets used in public docs."""
 
 import glob
+import gzip
 import logging
 import os
 import tempfile
@@ -355,13 +356,15 @@ class SnippetsTest(unittest.TestCase):
 To be used for testing.
 """
 
-def __init__(self, file_to_read=None):
+def __init__(self, file_to_read=None, compression_type=None):
   self.file_to_read = file_to_read
+  self.compression_type = compression_type
 
 class ReadDoFn(beam.DoFn):
 
-  def __init__(self, file_to_read):
+  def __init__(self, file_to_read, compression_type):
 self.file_to_read = file_to_read
+self.compression_type = compression_type
 self.coder = coders.StrUtf8Coder()
 
   def process(self, element):
@@ -370,13 +373,19 @@ class SnippetsTest(unittest.TestCase):
   def finish_bundle(self):
 assert self.file_to_read
 for file_name in glob.glob(self.file_to_read):
-  with open(file_name) as file:
-for record in file:
-  yield self.coder.decode(record.rstrip('\n'))
+  if self.compression_type is None:
+with open(file_name) as file:
+  for record in file:
+yield self.coder.decode(record.rstrip('\n'))
+  else:
+with gzip.open(file_name, 'r') as file:
+  for record in file:
+yield self.coder.decode(record.rstrip('\n'))
 
 def expand(self, pcoll):
   return pcoll | beam.Create([None]) | 'DummyReadForTesting' >> beam.ParDo(
-  

[GitHub] beam pull request #1845: [BEAM-1331] Add snippet for reading from compressed...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1845


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-722) Add Display Data to the Python SDK

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-722.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Add Display Data to the Python SDK
> --
>
> Key: BEAM-722
> URL: https://issues.apache.org/jira/browse/BEAM-722
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Frances Perry
> Fix For: Not applicable
>
>
> The DisplayData feature has been added to the Java SDK (see blog post 
> announcing it: 
> https://cloud.google.com/blog/big-data/2016/06/dataflow-updates-see-more-details-about-your-pipelines).
>  We need now to add it to the Python SDK.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-525) Verify that ParDo with multiple outputs with tags un declared in with_outputs() work

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-525.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Verify that ParDo with multiple outputs with tags un declared in 
> with_outputs() work 
> -
>
> Key: BEAM-525
> URL: https://issues.apache.org/jira/browse/BEAM-525
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
> Fix For: Not applicable
>
>
> test_undeclared_side_outputs was failing (when last checked) under certain 
> conditions:
> See this TODO:
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/dataflow_test.py#L202
> This is probably not failing any more but it needs to be verified.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-8) Import Python SDK

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-8.

   Resolution: Fixed
Fix Version/s: Not applicable

> Import Python SDK
> -
>
> Key: BEAM-8
> URL: https://issues.apache.org/jira/browse/BEAM-8
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Frances Perry
>Assignee: Frances Perry
> Fix For: Not applicable
>
>
> Google has an in progress Python SDK, which we will be contributing to Beam 
> in the near future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1090) High memory usage error

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-1090.
---
   Resolution: Fixed
 Assignee: Ahmet Altay
Fix Version/s: Not applicable

> High memory usage error
> ---
>
> Key: BEAM-1090
> URL: https://issues.apache.org/jira/browse/BEAM-1090
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 0.3.0-incubating
>Reporter: María GH
>Assignee: Ahmet Altay
>Priority: Minor
> Fix For: Not applicable
>
>
> Non-reproducible high memory usage test failure. It goes away on its own.
> RuntimeError: High memory usage: 201418866688 > 201008464768 [while running 
> 'oom:check']
> root: WARNING: A task failed with exception.
>  High memory usage: 201418866688 > 201008464768 [while running 'oom:check']
> ---
> Complete results at https://travis-ci.org/apache/incubator-beam/jobs/181011669



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1116) Support for new Timer API in Flink runner

[Jingsong Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingsong Lee reassigned BEAM-1116:
--

Assignee: Jingsong Lee

> Support for new Timer API in Flink runner
> -
>
> Key: BEAM-1116
> URL: https://issues.apache.org/jira/browse/BEAM-1116
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1408) outputWithTimestamp() accepts timestamps that will fail preconditions

[Andy Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy Xu updated BEAM-1408:
--
Description: 
We have accidentally created events with *wrong* timestamps in the future which 
are accepted by
outputWithTimestamp(), but will fail at a later step:

java.lang.IllegalStateException: Timer 472976-06-15T20:09:57.269Z is beyond 
end-of-time
at Preconditions.checkState(Preconditions.java:199)
at 
ReduceFnRunner.scheduleEndOfWindowOrGarbageCollectionTimer(ReduceFnRunner.java:1050)
[...]

Would it make sense to implement a check already at outputWithTimestamp() level 
to fail early?


  was:
We have accidentally created events with *wrong* timestamps in the future which 
are accepted by
outputWithTimestamp(), but will fail at a later step:

java.lang.IllegalStateException: Timer 472976-06-15T20:09:57.269Z is beyond 
end-of-time

atcom.google.cloud.dataflow.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:199)
at 
com.google.cloud.dataflow.sdk.util.ReduceFnRunner.scheduleEndOfWindowOrGarbageCollectionTimer(ReduceFnRunner.java:1050)
[...]

Would it make sense to implement a check already at outputWithTimestamp() level 
to fail early?



> outputWithTimestamp() accepts timestamps that will fail preconditions
> -
>
> Key: BEAM-1408
> URL: https://issues.apache.org/jira/browse/BEAM-1408
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Andy Xu
>Assignee: Davor Bonaci
>Priority: Minor
>
> We have accidentally created events with *wrong* timestamps in the future 
> which are accepted by
> outputWithTimestamp(), but will fail at a later step:
> java.lang.IllegalStateException: Timer 472976-06-15T20:09:57.269Z is beyond 
> end-of-time
> at Preconditions.checkState(Preconditions.java:199)
> at 
> ReduceFnRunner.scheduleEndOfWindowOrGarbageCollectionTimer(ReduceFnRunner.java:1050)
> [...]
> Would it make sense to implement a check already at outputWithTimestamp() 
> level to fail early?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1925: Add git .mailmap file

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/1925


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Add some more RunnableOnService tests for stateful ParDo

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 0c24286e1 -> a26fd1ff3


Add some more RunnableOnService tests for stateful ParDo


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6c00e912
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6c00e912
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6c00e912

Branch: refs/heads/master
Commit: 6c00e9121e6572fc06d0379802883c118acbed9f
Parents: f4e1097
Author: Kenneth Knowles 
Authored: Fri Jan 6 12:03:11 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 15:21:43 2017 -0800

--
 .../apache/beam/sdk/transforms/ParDoTest.java   | 138 +++
 1 file changed, 138 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6c00e912/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
index 2e3fb85..7381e06 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
@@ -80,6 +80,7 @@ import org.apache.beam.sdk.util.Timer;
 import org.apache.beam.sdk.util.TimerSpec;
 import org.apache.beam.sdk.util.TimerSpecs;
 import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
+import org.apache.beam.sdk.util.state.AccumulatorCombiningState;
 import org.apache.beam.sdk.util.state.BagState;
 import org.apache.beam.sdk.util.state.StateSpec;
 import org.apache.beam.sdk.util.state.StateSpecs;
@@ -1502,6 +1503,104 @@ public class ParDoTest implements Serializable {
 
   @Test
   @Category({RunnableOnService.class, UsesStatefulParDo.class})
+  public void testValueStateFixedWindows() {
+final String stateId = "foo";
+
+DoFn, Integer> fn =
+new DoFn, Integer>() {
+
+  @StateId(stateId)
+  private final StateSpec intState =
+  StateSpecs.value(VarIntCoder.of());
+
+  @ProcessElement
+  public void processElement(
+  ProcessContext c, @StateId(stateId) ValueState state) {
+Integer currentValue = MoreObjects.firstNonNull(state.read(), 0);
+c.output(currentValue);
+state.write(currentValue + 1);
+  }
+};
+
+IntervalWindow firstWindow = new IntervalWindow(new Instant(0), new 
Instant(10));
+IntervalWindow secondWindow = new IntervalWindow(new Instant(10), new 
Instant(20));
+
+PCollection output =
+pipeline
+.apply(
+Create.timestamped(
+// first window
+TimestampedValue.of(KV.of("hello", 7), new Instant(1)),
+TimestampedValue.of(KV.of("hello", 14), new Instant(2)),
+TimestampedValue.of(KV.of("hello", 21), new Instant(3)),
+
+// second window
+TimestampedValue.of(KV.of("hello", 28), new Instant(11)),
+TimestampedValue.of(KV.of("hello", 35), new Instant(13
+.apply(Window.>into(FixedWindows.of(Duration.millis(10
+.apply("Stateful ParDo", ParDo.of(fn));
+
+PAssert.that(output).inWindow(firstWindow).containsInAnyOrder(0, 1, 2);
+PAssert.that(output).inWindow(secondWindow).containsInAnyOrder(0, 1);
+pipeline.run();
+  }
+
+  /**
+   * Tests that there is no state bleeding between adjacent stateful {@link 
ParDo} transforms,
+   * which may (or may not) be executed in similar contexts after runner 
optimizations.
+   */
+  @Test
+  @Category({RunnableOnService.class, UsesStatefulParDo.class})
+  public void testValueStateSameId() {
+final String stateId = "foo";
+
+DoFn, KV> fn =
+new DoFn, KV>() {
+
+  @StateId(stateId)
+  private final StateSpec intState =
+  StateSpecs.value(VarIntCoder.of());
+
+  @ProcessElement
+  public void processElement(
+  ProcessContext c, @StateId(stateId) ValueState state) {
+Integer currentValue = MoreObjects.firstNonNull(state.read(), 0);
+c.output(KV.of("sizzle", currentValue));
+state.write(currentValue + 1);
+  }
+};
+
+DoFn, Integer> fn2 =
+new DoFn, Integer>() {
+
+  @StateId(stateId)
+  private final StateSpec intState =
+  StateSpecs.value(VarIntCoder.of());
+
+ 

[2/2] beam git commit: This closes #1748: Add some more RunnableOnService tests for stateful ParDo

[kenn]

This closes #1748: Add some more RunnableOnService tests for stateful ParDo


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a26fd1ff
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a26fd1ff
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a26fd1ff

Branch: refs/heads/master
Commit: a26fd1ff34bb5a51d6cc82ab9eb3c7e40cef49d3
Parents: 0c24286 6c00e91
Author: Kenneth Knowles 
Authored: Mon Feb 6 15:38:43 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 15:38:43 2017 -0800

--
 .../apache/beam/sdk/transforms/ParDoTest.java   | 138 +++
 1 file changed, 138 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a26fd1ff/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
--

[GitHub] beam pull request #1932: [BEAM-1406] Removes deprecated fileio.TextFileSink

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/1932

[BEAM-1406] Removes deprecated fileio.TextFileSink

Users should be using textio.WriteToText() transform instead of
fileio.TextFileSink.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam remove_textfilesink_fileio

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1932.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1932


commit 2fed64bf5f3e7eda2a3a372556851cdbffeb1a1a
Author: Chamikara Jayalath 
Date:   2017-02-07T00:01:11Z

Removes deprecated fileio.TextFileSink.

Users should be using textio.WriteToText() transform instead of
fileio.TextFileSink.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1406) Remove deprecated fileio.TextFileSink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855018#comment-15855018
 ] 

ASF GitHub Bot commented on BEAM-1406:
--

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/1932

[BEAM-1406] Removes deprecated fileio.TextFileSink

Users should be using textio.WriteToText() transform instead of
fileio.TextFileSink.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam remove_textfilesink_fileio

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1932.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1932


commit 2fed64bf5f3e7eda2a3a372556851cdbffeb1a1a
Author: Chamikara Jayalath 
Date:   2017-02-07T00:01:11Z

Removes deprecated fileio.TextFileSink.

Users should be using textio.WriteToText() transform instead of
fileio.TextFileSink.




> Remove deprecated fileio.TextFileSink
> -
>
> Key: BEAM-1406
> URL: https://issues.apache.org/jira/browse/BEAM-1406
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> Users should be using textio.WriteToText() instead of fileio.TextFileSink.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854993#comment-15854993
 ] 

ASF GitHub Bot commented on BEAM-25:


Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1748


> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1409) The DirectRunner should forbid elements outside of representable time.

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1409:
-

 Summary: The DirectRunner should forbid elements outside of 
representable time.
 Key: BEAM-1409
 URL: https://issues.apache.org/jira/browse/BEAM-1409
 Project: Beam
  Issue Type: Improvement
  Components: runner-direct
Reporter: Thomas Groh
Assignee: Thomas Groh


This ensures that even if the timestamps are representable in Java, they are 
also representable within the model.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1748: [BEAM-25] Some more RunnableOnService tests for sta...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1748


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1409) The DirectRunner should forbid elements outside of representable time.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855003#comment-15855003
 ] 

ASF GitHub Bot commented on BEAM-1409:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/1931

[BEAM-1409] Check that Elements, Timers have permitted Timestamps

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Add a checkArgument to ImmutableListBundleFactory that elements must
be timestamped earlier than the maximum representable timestamp.

Add a checkArgument to TimerData that Timers must fire before the
maximum representable timestamp.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam checkArgument_not_past_infinity

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1931.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1931


commit ebc973254a8e85eb4e9476ed1778a068dd7436f8
Author: Thomas Groh 
Date:   2017-02-06T23:49:05Z

Check that Elements, Timers have permitted Timestamps

Add a checkArgument to ImmutableListBundleFactory that elements must
be timestamped earlier than the maximum representable timestamp.

Add a checkArgument to TimerData that Timers must fire before the
maximum representable timestamp.




> The DirectRunner should forbid elements outside of representable time.
> --
>
> Key: BEAM-1409
> URL: https://issues.apache.org/jira/browse/BEAM-1409
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> This ensures that even if the timestamps are representable in Java, they are 
> also representable within the model.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1931: [BEAM-1409] Check that Elements, Timers have permit...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/1931

[BEAM-1409] Check that Elements, Timers have permitted Timestamps

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Add a checkArgument to ImmutableListBundleFactory that elements must
be timestamped earlier than the maximum representable timestamp.

Add a checkArgument to TimerData that Timers must fire before the
maximum representable timestamp.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam checkArgument_not_past_infinity

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1931.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1931


commit ebc973254a8e85eb4e9476ed1778a068dd7436f8
Author: Thomas Groh 
Date:   2017-02-06T23:49:05Z

Check that Elements, Timers have permitted Timestamps

Add a checkArgument to ImmutableListBundleFactory that elements must
be timestamped earlier than the maximum representable timestamp.

Add a checkArgument to TimerData that Timers must fire before the
maximum representable timestamp.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1930: Add support repeated template invocations for BQIO....

[sammcveety]

GitHub user sammcveety opened a pull request:

https://github.com/apache/beam/pull/1930

Add support repeated template invocations for BQIO.Write



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam 
add_side_inputs_to_bigquery_io

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1930.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1930


commit c346ae9808187c4c75571481dffbc34f05b4ef04
Author: Sam McVeety 
Date:   2017-02-06T21:40:12Z

Add support repeated template invocations for BQIO.Write




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1407) Support multiple Kafka client in KakaIO

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854963#comment-15854963
 ] 

Xu Mingmin commented on BEAM-1407:
--

Thanks! will add a design doc soon.

> Support multiple Kafka client in KakaIO
> ---
>
> Key: BEAM-1407
> URL: https://issues.apache.org/jira/browse/BEAM-1407
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> enhance KafkaIO to work with Kafka client 0.9 and 0.10, (maybe 0.8 as well 
> ?), 
> 1). to fully leverage new features in each version, like external 
> authentication, timestamp in 0.10;
> 2). hide kafka API changed, to support steamless switch between different 
> cluster versions;
> Scope of change:
> 1). add an abstract API for Kafka Consumer in existing KafkaIO. By default 
> it's kafka 0.9 to keep compatible, with an option for extension;
> 2). Wrapped API for Kafka 0.10, with Kafka message timestamp support;
> 3). wrapped API for Kafka 0.8 (TBD);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1388) Update default configuration of retry decorator so that wait times are more practical

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-1388.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Update default configuration of retry decorator so that wait times are more 
> practical
> -
>
> Key: BEAM-1388
> URL: https://issues.apache.org/jira/browse/BEAM-1388
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: Not applicable
>
>
> Currently retry decorator [1] has following default configuration.
> initial delay: 5 sec
> number of retries: 16
> factor: 2
> max delay: 4 hours
> I propose updating these values to following so that default values are more 
> practical.
> initial delay: 5 sec
> number of retries: 7
> factor: 2
> max delay: 1 hour
> With this update, for maximum number of retries, system will wait for 635 sec 
> while wait before last retry being 320 sec.
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/retry.py#L117



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854962#comment-15854962
 ] 

ASF GitHub Bot commented on BEAM-27:


Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1827


> Add user-ready API for interacting with timers
> --
>
> Key: BEAM-27
> URL: https://issues.apache.org/jira/browse/BEAM-27
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> Pipeline authors will benefit from a different factorization of interaction 
> with underlying timers. The current APIs are targeted at runner implementers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1827: [BEAM-27] Allow absolute timers

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1827


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1827: Allow absolute timers

[kenn]

This closes #1827: Allow absolute timers


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0c24286e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0c24286e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0c24286e

Branch: refs/heads/master
Commit: 0c24286e1bd586b9202b170dbca74188383aa13d
Parents: f4e1097 a93c5c0
Author: Kenneth Knowles 
Authored: Mon Feb 6 15:17:45 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 15:17:45 2017 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java | 83 +++--
 .../java/org/apache/beam/sdk/util/Timer.java| 11 +++
 .../apache/beam/sdk/transforms/ParDoTest.java   | 93 
 3 files changed, 178 insertions(+), 9 deletions(-)
--

[1/2] beam git commit: Allow absolute timers

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master f4e109767 -> 0c24286e1


Allow absolute timers


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a93c5c05
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a93c5c05
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a93c5c05

Branch: refs/heads/master
Commit: a93c5c0594dcd4519fcf4b842f2fe0b8244a81a3
Parents: f4e1097
Author: Kenneth Knowles 
Authored: Mon Jan 23 20:50:50 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 15:17:23 2017 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java | 83 +++--
 .../java/org/apache/beam/sdk/util/Timer.java| 11 +++
 .../apache/beam/sdk/transforms/ParDoTest.java   | 93 
 3 files changed, 178 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a93c5c05/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index 8c9b8b7..7a89389 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.runners.core;
 
+import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkNotNull;
 
 import com.google.common.collect.Iterables;
@@ -94,6 +95,8 @@ public class SimpleDoFnRunner implements 
DoFnRunner implements 
DoFnRunner(
@@ -182,7 +186,8 @@ public class SimpleDoFnRunner implements 
DoFnRunner argumentProvider =
-new OnTimerArgumentProvider<>(fn, context, window, effectiveTimestamp, 
timeDomain);
+new OnTimerArgumentProvider<>(
+fn, context, window, allowedLateness, effectiveTimestamp, 
timeDomain);
 invoker.invokeOnTimer(timerId, argumentProvider);
   }
 
@@ -210,7 +215,7 @@ public class SimpleDoFnRunner implements 
DoFnRunner 
createProcessContext(WindowedValue elem) {
-return new DoFnProcessContext(fn, context, elem);
+return new DoFnProcessContext(fn, context, elem, 
allowedLateness);
   }
 
   private RuntimeException wrapUserCodeException(Throwable t) {
@@ -465,6 +470,7 @@ public class SimpleDoFnRunner implements 
DoFnRunner fn;
 final DoFnContext context;
 final WindowedValue windowedValue;
+private final Duration allowedLateness;
 
 /** Lazily initialized; should only be accessed via {@link 
#getNamespace()}. */
 @Nullable private StateNamespace namespace;
@@ -486,11 +492,13 @@ public class SimpleDoFnRunner implements 
DoFnRunner fn,
 DoFnContext context,
-WindowedValue windowedValue) {
+WindowedValue windowedValue,
+Duration allowedLateness) {
   fn.super();
   this.fn = fn;
   this.context = context;
   this.windowedValue = windowedValue;
+  this.allowedLateness = allowedLateness;
 }
 
 @Override
@@ -633,7 +641,8 @@ public class SimpleDoFnRunner implements 
DoFnRunner

[jira] [Commented] (BEAM-1407) Support multiple Kafka client in KakaIO

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854947#comment-15854947
 ] 

Davor Bonaci commented on BEAM-1407:


Here you go -- enjoy! Please let me know if I can help in any way.

> Support multiple Kafka client in KakaIO
> ---
>
> Key: BEAM-1407
> URL: https://issues.apache.org/jira/browse/BEAM-1407
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> enhance KafkaIO to work with Kafka client 0.9 and 0.10, to fully leverage new 
> features in 0.10: external authentication, timestamp.
> Scope of change:
> 1). Add an adapter layer to hide the API differences between Kafka0.9 and 
> 0.10;
> 2). use Kafka message timestamp as default in mode 0.10;



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1407) Support multiple Kafka client in KakaIO

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1407:
--

Assignee: Xu Mingmin  (was: Davor Bonaci)

> Support multiple Kafka client in KakaIO
> ---
>
> Key: BEAM-1407
> URL: https://issues.apache.org/jira/browse/BEAM-1407
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>
> enhance KafkaIO to work with Kafka client 0.9 and 0.10, to fully leverage new 
> features in 0.10: external authentication, timestamp.
> Scope of change:
> 1). Add an adapter layer to hide the API differences between Kafka0.9 and 
> 0.10;
> 2). use Kafka message timestamp as default in mode 0.10;



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: [BEAM-65] ProcessFn: support setup/teardown

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 224e44765 -> f4e109767


[BEAM-65] ProcessFn: support setup/teardown

Previously, ProcessFn did not explicitly invoke the underlying fn's
@Setup and @Teardown methods - it was assuming that those methods would
get invoked on that fn externally. This was true in direct runner,
but is not necessarily true in other runners: e.g., Dataflow runner
will serialize the whole ProcessFn and treat it mostly as a regular
DoFn, so it makes more sense to have lifecycle methods of ProcessFn
delegate to the underlying fn.

Also, adds a getter for fn (a runner may need it to create a proper
ProcessContext when creating the SplittableProcessElementInvoker).


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b6877ce1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b6877ce1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b6877ce1

Branch: refs/heads/master
Commit: b6877ce1739650e7e7593c1d2c2858d60d2393fb
Parents: 4ccbdbc
Author: Eugene Kirpichov 
Authored: Wed Feb 1 16:07:44 2017 -0800
Committer: Eugene Kirpichov 
Committed: Fri Feb 3 10:41:47 2017 -0800

--
 .../beam/runners/core/SplittableParDo.java  | 16 -
 .../beam/runners/core/SplittableParDoTest.java  | 73 +++-
 .../runners/direct/ParDoEvaluatorFactory.java   |  5 +-
 ...littableProcessElementsEvaluatorFactory.java | 11 ++-
 4 files changed, 96 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b6877ce1/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
index 78acb19..664f334 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
@@ -375,7 +375,6 @@ public class SplittableParDo
 Coder restrictionCoder,
 Coder windowCoder) {
   this.fn = fn;
-  this.invoker = DoFnInvokers.invokerFor(fn);
   this.windowCoder = windowCoder;
   this.elementTag =
   StateTags.value("element", WindowedValue.getFullCoder(elementCoder, 
this.windowCoder));
@@ -395,6 +394,21 @@ public class SplittableParDo
   this.processElementInvoker = invoker;
 }
 
+public DoFn getFn() {
+  return fn;
+}
+
+@Setup
+public void setup() throws Exception {
+  invoker = DoFnInvokers.invokerFor(fn);
+  invoker.invokeSetup();
+}
+
+@Teardown
+public void tearDown() throws Exception {
+  invoker.invokeTeardown();
+}
+
 @StartBundle
 public void startBundle(Context c) throws Exception {
   invoker.invokeStartBundle(wrapContext(c));

http://git-wip-us.apache.org/repos/asf/beam/blob/b6877ce1/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
index bb7fd8c..96d65ae 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
@@ -200,7 +200,8 @@ public class SplittableParDoTest {
* {@link DoFn.ProcessElement} calls).
*/
   private static class ProcessFnTester<
-  InputT, OutputT, RestrictionT, TrackerT extends 
RestrictionTracker> {
+  InputT, OutputT, RestrictionT, TrackerT extends 
RestrictionTracker>
+  implements AutoCloseable {
 private final DoFnTester<
 KeyedWorkItem>, OutputT>
 tester;
@@ -270,6 +271,11 @@ public class SplittableParDoTest {
   this.currentProcessingTime = currentProcessingTime;
 }
 
+@Override
+public void close() throws Exception {
+  tester.close();
+}
+
 /** Performs a seed {@link DoFn.ProcessElement} call feeding the element 
and restriction. */
 void startElement(InputT element, RestrictionT restriction) throws 
Exception {
   startElement(
@@ -633,4 +639,69 @@ public class SplittableParDoTest {
 Instant.now().getMillis() - base.getMillis(),
 greaterThanOrEqualTo(maxBundleDuration.getMillis()));
   }
+
+  private static class LifecycleVerifyingFn extends 

[2/2] beam git commit: This closes #1895: ProcessFn: support setup/teardown

[kenn]

This closes #1895: ProcessFn: support setup/teardown


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f4e10976
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f4e10976
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f4e10976

Branch: refs/heads/master
Commit: f4e10976789aa5fe4b350be21df29544fa626251
Parents: 224e447 b6877ce
Author: Kenneth Knowles 
Authored: Mon Feb 6 14:50:36 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 14:50:36 2017 -0800

--
 .../beam/runners/core/SplittableParDo.java  | 16 -
 .../beam/runners/core/SplittableParDoTest.java  | 73 +++-
 .../runners/direct/ParDoEvaluatorFactory.java   |  5 +-
 ...littableProcessElementsEvaluatorFactory.java | 11 ++-
 4 files changed, 96 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f4e10976/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
--

http://git-wip-us.apache.org/repos/asf/beam/blob/f4e10976/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
--

http://git-wip-us.apache.org/repos/asf/beam/blob/f4e10976/runners/direct-java/src/main/java/org/apache/beam/runners/direct/SplittableProcessElementsEvaluatorFactory.java
--

[GitHub] beam pull request #1895: [BEAM-65] ProcessFn: create invoker in setup()

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1895


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-1405) Spark runner should not close the spark context when set to reuse it

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-1405:
--

Assignee: Jean-Baptiste Onofré  (was: Ismaël Mejía)

> Spark runner should not close the spark context when set to reuse it
> 
>
> Key: BEAM-1405
> URL: https://issues.apache.org/jira/browse/BEAM-1405
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Ismaël Mejía
>Assignee: Jean-Baptiste Onofré
>
> It seems that we don't take into account the options when we decide to close 
> the spark context, this produces an exception when trying to reuse a provided 
> context.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854925#comment-15854925
 ] 

ASF GitHub Bot commented on BEAM-362:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/1929

[BEAM-362] Move a few more remnants of internal state to runners/core-java

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam StateTag

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1929.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1929


commit f01d08c3012b254c563a27b675e5fd1c2bb48ab9
Author: Kenneth Knowles 
Date:   2017-02-04T03:40:46Z

Move StateMerging to runners/core-java

commit 913179f61a6c2706a1ae7de72c2f6b075d73b77c
Author: Kenneth Knowles 
Date:   2017-02-04T03:41:45Z

Remove dead code referencing StateTag

commit 3b634afab5e4f6fff5da52e6fd10b9888a9163cd
Author: Kenneth Knowles 
Date:   2017-02-04T03:46:29Z

Move StateAccessor, MergingStateAccessor to runners/core-java

commit acffac24f6c7bc5144c79a0220df03fb1b0e3e3c
Author: Kenneth Knowles 
Date:   2017-02-04T03:48:38Z

Move StateTag adapter code to StateTags

commit aede9a6d16bafc43ef3d3e3112904004a49dd91b
Author: Kenneth Knowles 
Date:   2017-02-04T03:51:18Z

Move StateTag and friends to runners/core-java




> Move shared runner functionality out of SDK and into runners/core-java
> --
>
> Key: BEAM-362
> URL: https://issues.apache.org/jira/browse/BEAM-362
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1929: [BEAM-362] Move a few more remnants of internal sta...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/1929

[BEAM-362] Move a few more remnants of internal state to runners/core-java

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam StateTag

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1929.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1929


commit f01d08c3012b254c563a27b675e5fd1c2bb48ab9
Author: Kenneth Knowles 
Date:   2017-02-04T03:40:46Z

Move StateMerging to runners/core-java

commit 913179f61a6c2706a1ae7de72c2f6b075d73b77c
Author: Kenneth Knowles 
Date:   2017-02-04T03:41:45Z

Remove dead code referencing StateTag

commit 3b634afab5e4f6fff5da52e6fd10b9888a9163cd
Author: Kenneth Knowles 
Date:   2017-02-04T03:46:29Z

Move StateAccessor, MergingStateAccessor to runners/core-java

commit acffac24f6c7bc5144c79a0220df03fb1b0e3e3c
Author: Kenneth Knowles 
Date:   2017-02-04T03:48:38Z

Move StateTag adapter code to StateTags

commit aede9a6d16bafc43ef3d3e3112904004a49dd91b
Author: Kenneth Knowles 
Date:   2017-02-04T03:51:18Z

Move StateTag and friends to runners/core-java




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1407) Support multiple Kafka client in KakaIO

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-1407:
-
Summary: Support multiple Kafka client in KakaIO  (was: Support Kafka 
client 0.10 in KakaIO)

> Support multiple Kafka client in KakaIO
> ---
>
> Key: BEAM-1407
> URL: https://issues.apache.org/jira/browse/BEAM-1407
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Davor Bonaci
>
> enhance KafkaIO to work with Kafka client 0.9 and 0.10, to fully leverage new 
> features in 0.10: external authentication, timestamp.
> Scope of change:
> 1). Add an adapter layer to hide the API differences between Kafka0.9 and 
> 0.10;
> 2). use Kafka message timestamp as default in mode 0.10;



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-425) Create Elasticsearch IO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854912#comment-15854912
 ] 

Jean-Baptiste Onofré commented on BEAM-425:
---

We already have a branch with a elasticsearch-5 module using the new REST 
client with elasticsearch-5 slice API.

> Create Elasticsearch IO
> ---
>
> Key: BEAM-425
> URL: https://issues.apache.org/jira/browse/BEAM-425
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.5.0
>
>
> I'm working on a new ElasticsearchIO providing both bounded source and sink.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854891#comment-15854891
 ] 

ASF GitHub Bot commented on BEAM-362:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1858


> Move shared runner functionality out of SDK and into runners/core-java
> --
>
> Key: BEAM-362
> URL: https://issues.apache.org/jira/browse/BEAM-362
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1858: [BEAM-362] Move state-oriented remnants of OldDoFn ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1858


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1407) Support Kafka client 0.10 in KakaIO

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-1407:


 Summary: Support Kafka client 0.10 in KakaIO
 Key: BEAM-1407
 URL: https://issues.apache.org/jira/browse/BEAM-1407
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Xu Mingmin
Assignee: Davor Bonaci


enhance KafkaIO to work with Kafka client 0.9 and 0.10, to fully leverage new 
features in 0.10: external authentication, timestamp.

Scope of change:
1). Add an adapter layer to hide the API differences between Kafka0.9 and 0.10;
2). use Kafka message timestamp as default in mode 0.10;




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[06/15] beam git commit: Delete long-deprecated StateContexts methods

[kenn]

Delete long-deprecated StateContexts methods


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b12e5ffb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b12e5ffb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b12e5ffb

Branch: refs/heads/master
Commit: b12e5ffbaf9bacebc5d80b3ed24c37a22e37524f
Parents: af391b8
Author: Kenneth Knowles 
Authored: Thu Jan 26 21:05:09 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 09:26:06 2017 -0800

--
 .../runners/core/SimpleOldDoFnRunnerTest.java   |  1 -
 .../core/triggers/NeverStateMachineTest.java|  1 -
 .../beam/sdk/util/state/StateContexts.java  | 63 
 3 files changed, 65 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b12e5ffb/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleOldDoFnRunnerTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleOldDoFnRunnerTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleOldDoFnRunnerTest.java
index 97da9ee..28698ca 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleOldDoFnRunnerTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SimpleOldDoFnRunnerTest.java
@@ -23,7 +23,6 @@ import static org.mockito.Mockito.mock;
 import java.util.Arrays;
 import java.util.List;
 import org.apache.beam.runners.core.BaseExecutionContext.StepContext;
-
 import org.apache.beam.sdk.util.SystemDoFnInternal;
 import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.util.WindowedValue;

http://git-wip-us.apache.org/repos/asf/beam/blob/b12e5ffb/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/NeverStateMachineTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/NeverStateMachineTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/NeverStateMachineTest.java
index 6d8a344..79147d4 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/NeverStateMachineTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/NeverStateMachineTest.java
@@ -25,7 +25,6 @@ import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.FixedWindows;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.values.TimestampedValue;
-
 import org.joda.time.Duration;
 import org.joda.time.Instant;
 import org.junit.Before;

http://git-wip-us.apache.org/repos/asf/beam/blob/b12e5ffb/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
index a9ce0b5..2ce9594 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
@@ -17,10 +17,8 @@
  */
 package org.apache.beam.sdk.util.state;
 
-import javax.annotation.Nullable;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.values.PCollectionView;
 
 /**
@@ -50,65 +48,4 @@ public class StateContexts {
   public static  StateContext nullContext() {
 return (StateContext) NULL_CONTEXT;
   }
-
-  /**
-   * @deprecated This exists only for temporary compatibility with Dataflow 
worker and should be
-   * deleted once a worker image is released that uses runners-core build 
after
-   * https://github.com/apache/beam/pull/1353.
-   */
-  @Deprecated
-  public static  StateContext windowOnly(final W 
window) {
-return new StateContext() {
-  @Override
-  public PipelineOptions getPipelineOptions() {
-throw new IllegalArgumentException(
-"cannot call getPipelineOptions() in a window only context");
-  }
-
-  @Override
-  public  T sideInput(PCollectionView view) {
-throw new IllegalArgumentException("cannot call sideInput() in a 
window only context");
-  }
-
-  @Override
-  public W window() {
-return window;
-  }
-};
-  }
-
-  /**
-   * @deprecated This exists only for temporary compatibility with Dataflow 
worker and should be
-   * deleted once a worker image is released that uses 

[14/15] beam git commit: Move StateInternals to runners/core-java

[kenn]

Move StateInternals to runners/core-java


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/144b1df8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/144b1df8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/144b1df8

Branch: refs/heads/master
Commit: 144b1df88176f32aec4991423eb225b844ff16c2
Parents: 4e4391c
Author: Kenneth Knowles 
Authored: Thu Jan 26 21:22:38 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 09:26:56 2017 -0800

--
 .../operators/ApexGroupByKeyOperator.java   |  2 +-
 .../operators/ApexParDoOperator.java|  2 +-
 .../translation/utils/ApexStateInternals.java   |  2 +-
 .../apex/translation/utils/NoOpStepContext.java |  2 +-
 .../beam/runners/core/BaseExecutionContext.java |  1 -
 .../beam/runners/core/ExecutionContext.java |  1 -
 .../GroupAlsoByWindowViaOutputBufferDoFn.java   |  1 -
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |  1 -
 .../GroupAlsoByWindowViaWindowSetNewDoFn.java   |  1 -
 .../runners/core/InMemoryStateInternals.java|  1 -
 .../runners/core/MergingActiveWindowSet.java|  1 -
 .../runners/core/ReduceFnContextFactory.java|  1 -
 .../beam/runners/core/ReduceFnRunner.java   |  1 -
 .../beam/runners/core/SideInputHandler.java |  1 -
 .../beam/runners/core/SimpleOldDoFnRunner.java  |  1 -
 .../beam/runners/core/SplittableParDo.java  |  1 -
 .../beam/runners/core/StateInternals.java   | 61 
 .../runners/core/StateInternalsFactory.java |  1 -
 .../beam/runners/core/WindowingInternals.java   |  1 -
 .../TriggerStateMachineContextFactory.java  |  2 +-
 .../core/GroupAlsoByWindowsProperties.java  |  1 -
 .../core/MergingActiveWindowSetTest.java|  1 -
 .../beam/runners/core/SplittableParDoTest.java  |  1 -
 .../triggers/TriggerStateMachineTester.java |  2 +-
 .../CopyOnAccessInMemoryStateInternals.java |  2 +-
 ...littableProcessElementsEvaluatorFactory.java |  2 +-
 .../StatefulParDoEvaluatorFactoryTest.java  |  2 +-
 .../functions/FlinkProcessContextBase.java  |  2 +-
 .../wrappers/streaming/DoFnOperator.java|  2 +-
 .../wrappers/streaming/FlinkStateInternals.java |  2 +-
 .../wrappers/streaming/WindowDoFnOperator.java  |  2 +-
 .../translation/SparkGroupAlsoByWindowFn.java   |  4 +-
 .../spark/translation/SparkProcessContext.java  |  2 +-
 .../spark/translation/TranslationUtils.java |  2 +-
 .../beam/sdk/util/state/StateInternals.java | 57 --
 .../beam/fn/harness/fake/FakeStepContext.java   |  2 +-
 36 files changed, 78 insertions(+), 93 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/144b1df8/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
index 7891b34..274e807 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
@@ -45,6 +45,7 @@ import 
org.apache.beam.runners.core.GroupAlsoByWindowViaWindowSetDoFn;
 import org.apache.beam.runners.core.KeyedWorkItem;
 import org.apache.beam.runners.core.KeyedWorkItems;
 import org.apache.beam.runners.core.OldDoFn;
+import org.apache.beam.runners.core.StateInternals;
 import org.apache.beam.runners.core.StateInternalsFactory;
 import org.apache.beam.runners.core.SystemReduceFn;
 import org.apache.beam.runners.core.TimerInternals;
@@ -62,7 +63,6 @@ import org.apache.beam.sdk.util.CoderUtils;
 import org.apache.beam.sdk.util.TimeDomain;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingStrategy;
-import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.util.state.StateNamespace;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;

http://git-wip-us.apache.org/repos/asf/beam/blob/144b1df8/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
index 13db40a..7f2512a 100644
--- 

[05/15] beam git commit: Move WindowingInternals to runners/core-java

[kenn]

Move WindowingInternals to runners/core-java


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/949ab3ac
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/949ab3ac
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/949ab3ac

Branch: refs/heads/master
Commit: 949ab3ac6d654a310a513d2e64e8dbf39fd4f388
Parents: b12e5ff
Author: Kenneth Knowles 
Authored: Thu Jan 26 21:06:10 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 09:26:06 2017 -0800

--
 .../operators/ApexGroupByKeyOperator.java   |  2 +-
 .../org/apache/beam/runners/core/OldDoFn.java   |  1 -
 .../beam/runners/core/SimpleOldDoFnRunner.java  |  1 -
 .../beam/runners/core/WindowingInternals.java   | 83 
 .../core/WindowingInternalsAdapters.java|  1 -
 .../core/GroupAlsoByWindowsProperties.java  |  1 -
 .../beam/runners/core/ReduceFnTester.java   |  1 -
 .../functions/FlinkProcessContextBase.java  |  2 +-
 .../beam/sdk/util/WindowingInternals.java   | 82 ---
 9 files changed, 85 insertions(+), 89 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/949ab3ac/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
index 4c2b461..c5da368 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
@@ -46,6 +46,7 @@ import org.apache.beam.runners.core.KeyedWorkItem;
 import org.apache.beam.runners.core.KeyedWorkItems;
 import org.apache.beam.runners.core.OldDoFn;
 import org.apache.beam.runners.core.SystemReduceFn;
+import org.apache.beam.runners.core.WindowingInternals;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
 import org.apache.beam.sdk.coders.KvCoder;
@@ -59,7 +60,6 @@ import org.apache.beam.sdk.util.CoderUtils;
 import org.apache.beam.sdk.util.TimeDomain;
 import org.apache.beam.sdk.util.TimerInternals;
 import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.util.state.StateInternalsFactory;

http://git-wip-us.apache.org/repos/asf/beam/blob/949ab3ac/runners/core-java/src/main/java/org/apache/beam/runners/core/OldDoFn.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/OldDoFn.java 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/OldDoFn.java
index b099721..4033260 100644
--- a/runners/core-java/src/main/java/org/apache/beam/runners/core/OldDoFn.java
+++ b/runners/core-java/src/main/java/org/apache/beam/runners/core/OldDoFn.java
@@ -42,7 +42,6 @@ import org.apache.beam.sdk.transforms.display.DisplayData;
 import org.apache.beam.sdk.transforms.display.HasDisplayData;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
 import org.joda.time.Duration;

http://git-wip-us.apache.org/repos/asf/beam/blob/949ab3ac/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleOldDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleOldDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleOldDoFnRunner.java
index 2fe9226..9f80bca 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleOldDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleOldDoFnRunner.java
@@ -42,7 +42,6 @@ import org.apache.beam.sdk.util.TimeDomain;
 import org.apache.beam.sdk.util.TimerInternals;
 import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.values.PCollectionView;

[08/15] beam git commit: Move StateNamespace and related to runners/core-java

[kenn]

http://git-wip-us.apache.org/repos/asf/beam/blob/e77e7f0d/sdks/java/core/src/test/java/org/apache/beam/sdk/util/state/StateNamespacesTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/state/StateNamespacesTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/state/StateNamespacesTest.java
deleted file mode 100644
index f546e56..000
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/state/StateNamespacesTest.java
+++ /dev/null
@@ -1,130 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util.state;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertThat;
-
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.hamcrest.Matchers;
-import org.joda.time.Instant;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-/**
- * Tests for {@link StateNamespaces}.
- */
-@RunWith(JUnit4.class)
-public class StateNamespacesTest {
-
-  private final Coder intervalCoder = 
IntervalWindow.getCoder();
-
-  private IntervalWindow intervalWindow(long start, long end) {
-return new IntervalWindow(new Instant(start), new Instant(end));
-  }
-
-  /**
-   * This test should not be changed. It verifies that the stringKey matches 
certain expectations.
-   * If this changes, the ability to reload any pipeline that has persisted 
these namespaces will
-   * be impacted.
-   */
-  @Test
-  public void testStability() {
-StateNamespace global = StateNamespaces.global();
-StateNamespace intervalWindow =
-StateNamespaces.window(intervalCoder, intervalWindow(1000, 87392));
-StateNamespace intervalWindowAndTrigger =
-StateNamespaces.windowAndTrigger(intervalCoder, intervalWindow(1000, 
87392), 57);
-StateNamespace globalWindow = StateNamespaces.window(
-GlobalWindow.Coder.INSTANCE, GlobalWindow.INSTANCE);
-StateNamespace globalWindowAndTrigger = StateNamespaces.windowAndTrigger(
-GlobalWindow.Coder.INSTANCE, GlobalWindow.INSTANCE, 12);
-
-assertEquals("/", global.stringKey());
-assertEquals("/gAABVWD4ogU/", intervalWindow.stringKey());
-assertEquals("/gAABVWD4ogU/1L/", intervalWindowAndTrigger.stringKey());
-assertEquals("//", globalWindow.stringKey());
-assertEquals("//C/", globalWindowAndTrigger.stringKey());
-  }
-
-  /**
-   * Test that WindowAndTrigger namespaces are prefixed by the related Window 
namespace.
-   */
-  @Test
-  public void testIntervalWindowPrefixing() {
-StateNamespace window =
-StateNamespaces.window(intervalCoder, intervalWindow(1000, 87392));
-StateNamespace windowAndTrigger = StateNamespaces.windowAndTrigger(
-intervalCoder, intervalWindow(1000, 87392), 57);
-assertThat(windowAndTrigger.stringKey(), 
Matchers.startsWith(window.stringKey()));
-assertThat(StateNamespaces.global().stringKey(),
-Matchers.not(Matchers.startsWith(window.stringKey(;
-  }
-
-  /**
-   * Test that WindowAndTrigger namespaces are prefixed by the related Window 
namespace.
-   */
-  @Test
-  public void testGlobalWindowPrefixing() {
-StateNamespace window =
-StateNamespaces.window(GlobalWindow.Coder.INSTANCE, 
GlobalWindow.INSTANCE);
-StateNamespace windowAndTrigger = StateNamespaces.windowAndTrigger(
-GlobalWindow.Coder.INSTANCE, GlobalWindow.INSTANCE, 57);
-assertThat(windowAndTrigger.stringKey(), 
Matchers.startsWith(window.stringKey()));
-assertThat(StateNamespaces.global().stringKey(),
-Matchers.not(Matchers.startsWith(window.stringKey(;
-  }
-
-  @Test
-  public void testFromStringGlobal() {
-assertStringKeyRoundTrips(intervalCoder, StateNamespaces.global());
-  }
-
-  @Test
-  public void testFromStringIntervalWindow() {
-assertStringKeyRoundTrips(
-intervalCoder, StateNamespaces.window(intervalCoder, 
intervalWindow(1000, 8000)));
-assertStringKeyRoundTrips(
-

[04/15] beam git commit: Move StateInternalsFactory to runners/core-java

[kenn]

Move StateInternalsFactory to runners/core-java


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2e6c1310
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2e6c1310
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2e6c1310

Branch: refs/heads/master
Commit: 2e6c131027caea3e4df5545ff1b65acc2a7f53a0
Parents: 1d7c6b0
Author: Kenneth Knowles 
Authored: Thu Jan 26 21:13:14 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 09:26:06 2017 -0800

--
 .../apex/translation/TranslationContext.java|  2 +-
 .../operators/ApexGroupByKeyOperator.java   |  2 +-
 .../operators/ApexParDoOperator.java|  2 +-
 .../translation/utils/ApexStateInternals.java   |  4 +--
 .../GroupAlsoByWindowViaOutputBufferDoFn.java   |  1 -
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |  1 -
 .../GroupAlsoByWindowViaWindowSetNewDoFn.java   |  1 -
 .../beam/runners/core/SplittableParDo.java  |  1 -
 .../runners/core/StateInternalsFactory.java | 36 
 ...roupAlsoByWindowViaOutputBufferDoFnTest.java |  1 -
 .../core/GroupAlsoByWindowsProperties.java  |  1 -
 .../beam/runners/core/SplittableParDoTest.java  |  1 -
 ...littableProcessElementsEvaluatorFactory.java |  2 +-
 .../wrappers/streaming/WindowDoFnOperator.java  |  3 +-
 .../translation/SparkGroupAlsoByWindowFn.java   |  2 +-
 .../spark/translation/TranslationUtils.java |  2 +-
 .../sdk/util/state/StateInternalsFactory.java   | 35 ---
 17 files changed, 44 insertions(+), 53 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2e6c1310/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
index 8c30bc6..acd8ab1 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/TranslationContext.java
@@ -33,11 +33,11 @@ import org.apache.beam.runners.apex.ApexPipelineOptions;
 import org.apache.beam.runners.apex.translation.utils.ApexStateInternals;
 import org.apache.beam.runners.apex.translation.utils.ApexStreamTuple;
 import org.apache.beam.runners.apex.translation.utils.CoderAdapterStreamCodec;
+import org.apache.beam.runners.core.StateInternalsFactory;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.runners.TransformHierarchy;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
-import org.apache.beam.sdk.util.state.StateInternalsFactory;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.PInput;

http://git-wip-us.apache.org/repos/asf/beam/blob/2e6c1310/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
index c5da368..6322796 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
@@ -45,6 +45,7 @@ import 
org.apache.beam.runners.core.GroupAlsoByWindowViaWindowSetDoFn;
 import org.apache.beam.runners.core.KeyedWorkItem;
 import org.apache.beam.runners.core.KeyedWorkItems;
 import org.apache.beam.runners.core.OldDoFn;
+import org.apache.beam.runners.core.StateInternalsFactory;
 import org.apache.beam.runners.core.SystemReduceFn;
 import org.apache.beam.runners.core.WindowingInternals;
 import org.apache.beam.sdk.coders.Coder;
@@ -62,7 +63,6 @@ import org.apache.beam.sdk.util.TimerInternals;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.util.state.StateInternals;
-import org.apache.beam.sdk.util.state.StateInternalsFactory;
 import org.apache.beam.sdk.util.state.StateNamespace;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;

http://git-wip-us.apache.org/repos/asf/beam/blob/2e6c1310/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java

[09/15] beam git commit: Move StateNamespace and related to runners/core-java

[kenn]

Move StateNamespace and related to runners/core-java


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e77e7f0d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e77e7f0d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e77e7f0d

Branch: refs/heads/master
Commit: e77e7f0d535eb8ad309f3210e3061566b55e7a21
Parents: 144b1df
Author: Kenneth Knowles 
Authored: Thu Jan 26 21:26:04 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 09:26:56 2017 -0800

--
 .../operators/ApexGroupByKeyOperator.java   |   2 +-
 .../translation/utils/ApexStateInternals.java   |   2 +-
 .../utils/ApexStateInternalsTest.java   |   6 +-
 .../runners/core/InMemoryStateInternals.java|   1 -
 .../runners/core/InMemoryTimerInternals.java|   1 -
 .../runners/core/MergingActiveWindowSet.java|   1 -
 .../runners/core/ReduceFnContextFactory.java|   4 +-
 .../beam/runners/core/ReduceFnRunner.java   |   2 +-
 .../beam/runners/core/SideInputHandler.java |   1 -
 .../beam/runners/core/SimpleDoFnRunner.java |   2 -
 .../beam/runners/core/SplittableParDo.java  |   2 -
 .../beam/runners/core/StateInternals.java   |   1 -
 .../beam/runners/core/StateNamespace.java   |  56 
 .../runners/core/StateNamespaceForTest.java |  65 +
 .../beam/runners/core/StateNamespaces.java  | 278 +++
 .../apache/beam/runners/core/StateTable.java|   1 -
 .../core/TestInMemoryStateInternals.java|   1 -
 .../beam/runners/core/TimerInternals.java   |   2 -
 .../TriggerStateMachineContextFactory.java  |   4 +-
 .../core/InMemoryStateInternalsTest.java|   2 -
 .../core/InMemoryTimerInternalsTest.java|   2 -
 .../runners/core/KeyedWorkItemCoderTest.java|   1 -
 .../core/PushbackSideInputDoFnRunnerTest.java   |   1 -
 .../beam/runners/core/ReduceFnTester.java   |   2 -
 .../beam/runners/core/SimpleDoFnRunnerTest.java |   1 -
 .../beam/runners/core/StateNamespacesTest.java  | 130 +
 .../beam/runners/core/TimerInternalsTest.java   |   2 -
 .../triggers/TriggerStateMachineTester.java |   8 +-
 .../CopyOnAccessInMemoryStateInternals.java |   2 +-
 .../runners/direct/DirectTimerInternals.java|   2 +-
 .../direct/StatefulParDoEvaluatorFactory.java   |   4 +-
 .../beam/runners/direct/WatermarkManager.java   |   2 +-
 .../CopyOnAccessInMemoryStateInternalsTest.java |   6 +-
 .../direct/DirectTimerInternalsTest.java|   2 +-
 ...leManagerRemovingTransformEvaluatorTest.java |   2 +-
 .../runners/direct/EvaluationContextTest.java   |   2 +-
 .../StatefulParDoEvaluatorFactoryTest.java  |   4 +-
 .../runners/direct/WatermarkManagerTest.java|   2 +-
 .../wrappers/streaming/FlinkStateInternals.java |   2 +-
 .../wrappers/streaming/WindowDoFnOperator.java  |   2 +-
 .../streaming/FlinkStateInternalsTest.java  |   4 +-
 .../beam/sdk/util/state/StateNamespace.java |  56 
 .../sdk/util/state/StateNamespaceForTest.java   |  65 -
 .../beam/sdk/util/state/StateNamespaces.java| 278 ---
 .../sdk/util/state/StateNamespacesTest.java | 130 -
 45 files changed, 559 insertions(+), 587 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e77e7f0d/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
index 274e807..3508c3e 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
@@ -47,6 +47,7 @@ import org.apache.beam.runners.core.KeyedWorkItems;
 import org.apache.beam.runners.core.OldDoFn;
 import org.apache.beam.runners.core.StateInternals;
 import org.apache.beam.runners.core.StateInternalsFactory;
+import org.apache.beam.runners.core.StateNamespace;
 import org.apache.beam.runners.core.SystemReduceFn;
 import org.apache.beam.runners.core.TimerInternals;
 import org.apache.beam.runners.core.WindowingInternals;
@@ -63,7 +64,6 @@ import org.apache.beam.sdk.util.CoderUtils;
 import org.apache.beam.sdk.util.TimeDomain;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingStrategy;
-import org.apache.beam.sdk.util.state.StateNamespace;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionView;

[11/15] beam git commit: Move TimerInternals to runners/core-java

[kenn]

Move TimerInternals to runners/core-java


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b086d2fd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b086d2fd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b086d2fd

Branch: refs/heads/master
Commit: b086d2fdf1dba2b0e4a3362bd3ee1a92bf21c56a
Parents: 92b33bc
Author: Kenneth Knowles 
Authored: Thu Jan 26 21:17:18 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 09:26:56 2017 -0800

--
 .../operators/ApexGroupByKeyOperator.java   |   2 +-
 .../apex/translation/utils/NoOpStepContext.java |   2 +-
 .../beam/runners/core/BaseExecutionContext.java |   1 -
 .../beam/runners/core/ExecutionContext.java |   1 -
 .../GroupAlsoByWindowViaOutputBufferDoFn.java   |   1 -
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |   1 -
 .../GroupAlsoByWindowViaWindowSetNewDoFn.java   |   2 -
 .../runners/core/InMemoryTimerInternals.java|   1 -
 .../apache/beam/runners/core/KeyedWorkItem.java |   2 +-
 .../beam/runners/core/KeyedWorkItemCoder.java   |   4 +-
 .../beam/runners/core/KeyedWorkItems.java   |   2 +-
 .../core/LateDataDroppingDoFnRunner.java|   1 -
 .../beam/runners/core/PaneInfoTracker.java  |   1 -
 .../runners/core/ReduceFnContextFactory.java|   3 +-
 .../beam/runners/core/ReduceFnRunner.java   |   3 +-
 .../beam/runners/core/SimpleDoFnRunner.java |   1 -
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   1 -
 .../beam/runners/core/SplittableParDo.java  |   1 -
 .../beam/runners/core/TimerInternals.java   | 288 +++
 .../runners/core/TimerInternalsFactory.java |   1 -
 .../apache/beam/runners/core/WatermarkHold.java |   1 -
 .../beam/runners/core/WindowingInternals.java   |   1 -
 .../core/GroupAlsoByWindowsProperties.java  |   1 -
 .../core/InMemoryTimerInternalsTest.java|   2 +-
 .../runners/core/KeyedWorkItemCoderTest.java|   2 +-
 .../core/LateDataDroppingDoFnRunnerTest.java|   1 -
 .../core/PushbackSideInputDoFnRunnerTest.java   |   2 +-
 .../beam/runners/core/ReduceFnTester.java   |   3 +-
 .../beam/runners/core/SimpleDoFnRunnerTest.java |   3 +-
 .../beam/runners/core/SplittableParDoTest.java  |   1 -
 .../beam/runners/core/TimerInternalsTest.java   | 107 +++
 .../triggers/TriggerStateMachineTester.java |   2 +-
 .../runners/direct/DirectExecutionContext.java  |   2 +-
 .../beam/runners/direct/DirectRunner.java   |   2 +-
 .../runners/direct/DirectTimerInternals.java|   2 +-
 ...ecycleManagerRemovingTransformEvaluator.java |   2 +-
 .../beam/runners/direct/EvaluationContext.java  |   2 +-
 .../direct/ExecutorServiceParallelExecutor.java |   2 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   2 +-
 .../beam/runners/direct/ParDoEvaluator.java |   2 +-
 ...littableProcessElementsEvaluatorFactory.java |   2 +-
 .../direct/StatefulParDoEvaluatorFactory.java   |   2 +-
 .../beam/runners/direct/WatermarkManager.java   |   4 +-
 .../direct/DirectTimerInternalsTest.java|   2 +-
 ...leManagerRemovingTransformEvaluatorTest.java |   2 +-
 .../runners/direct/EvaluationContextTest.java   |   2 +-
 .../runners/direct/WatermarkManagerTest.java|   2 +-
 .../functions/FlinkProcessContextBase.java  |   2 +-
 .../wrappers/streaming/DoFnOperator.java|   2 +-
 .../streaming/SingletonKeyedWorkItem.java   |   2 +-
 .../wrappers/streaming/WindowDoFnOperator.java  |   4 +-
 .../translation/SparkGroupAlsoByWindowFn.java   |   2 +-
 .../spark/translation/SparkProcessContext.java  |   2 +-
 .../apache/beam/sdk/util/TimerInternals.java| 286 --
 .../sdk/transforms/join/UnionCoderTest.java |   7 -
 .../beam/sdk/util/TimerInternalsTest.java   | 106 ---
 .../beam/fn/harness/fake/FakeStepContext.java   |   2 +-
 57 files changed, 433 insertions(+), 459 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b086d2fd/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
index 6322796..7891b34 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
@@ -47,6 +47,7 @@ import org.apache.beam.runners.core.KeyedWorkItems;
 import org.apache.beam.runners.core.OldDoFn;
 import org.apache.beam.runners.core.StateInternalsFactory;
 import 

[01/15] beam git commit: Move TimerInternalsFactory to runners/core-java

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master e0189f352 -> 224e44765


Move TimerInternalsFactory to runners/core-java


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/92b33bc7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/92b33bc7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/92b33bc7

Branch: refs/heads/master
Commit: 92b33bc7ec6d7c5c32a49c5750ac1d4381a478ce
Parents: 2e6c131
Author: Kenneth Knowles 
Authored: Thu Jan 26 21:13:42 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 09:26:06 2017 -0800

--
 .../beam/runners/core/SplittableParDo.java  |  1 -
 .../runners/core/TimerInternalsFactory.java | 36 
 .../beam/runners/core/SplittableParDoTest.java  |  1 -
 ...littableProcessElementsEvaluatorFactory.java |  2 +-
 .../sdk/util/state/TimerInternalsFactory.java   | 36 
 5 files changed, 37 insertions(+), 39 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/92b33bc7/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
index e414430..7368b2f 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
@@ -57,7 +57,6 @@ import org.apache.beam.sdk.util.state.StateNamespace;
 import org.apache.beam.sdk.util.state.StateNamespaces;
 import org.apache.beam.sdk.util.state.StateTag;
 import org.apache.beam.sdk.util.state.StateTags;
-import org.apache.beam.sdk.util.state.TimerInternalsFactory;
 import org.apache.beam.sdk.util.state.ValueState;
 import org.apache.beam.sdk.util.state.WatermarkHoldState;
 import org.apache.beam.sdk.values.KV;

http://git-wip-us.apache.org/repos/asf/beam/blob/92b33bc7/runners/core-java/src/main/java/org/apache/beam/runners/core/TimerInternalsFactory.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/TimerInternalsFactory.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/TimerInternalsFactory.java
new file mode 100644
index 000..e129aed
--- /dev/null
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/TimerInternalsFactory.java
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core;
+
+import java.io.Serializable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.util.TimerInternals;
+
+/**
+ * A factory for providing {@link TimerInternals} for a particular key.
+ *
+ * Because it will generally be embedded in a {@link 
org.apache.beam.sdk.transforms.DoFn DoFn},
+ * albeit at execution time, it is marked {@link Serializable}.
+ */
+@Experimental(Kind.STATE)
+public interface TimerInternalsFactory {
+
+  /** Returns {@link TimerInternals} for the provided key. */
+  TimerInternals timerInternalsForKey(K key);
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/92b33bc7/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
index b408d37..427e2f4 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
@@ -55,7 +55,6 @@ import org.apache.beam.sdk.util.SideInputReader;
 import 

[jira] [Commented] (BEAM-1407) Support Kafka client 0.10 in KakaIO

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854892#comment-15854892
 ] 

Xu Mingmin commented on BEAM-1407:
--

[~davor], can you assign this task to me?

> Support Kafka client 0.10 in KakaIO
> ---
>
> Key: BEAM-1407
> URL: https://issues.apache.org/jira/browse/BEAM-1407
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Davor Bonaci
>
> enhance KafkaIO to work with Kafka client 0.9 and 0.10, to fully leverage new 
> features in 0.10: external authentication, timestamp.
> Scope of change:
> 1). Add an adapter layer to hide the API differences between Kafka0.9 and 
> 0.10;
> 2). use Kafka message timestamp as default in mode 0.10;



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[15/15] beam git commit: This closes #1858: Move state-oriented remnants of OldDoFn to runners/core-java

[kenn]

This closes #1858: Move state-oriented remnants of OldDoFn to runners/core-java

  Upgrade Dataflow container version to beam-master-20170203
  Move StateNamespace and related to runners/core-java
  Move StateInternals to runners/core-java
  Move StateTable to runners/core-java
  Move TimerInternals to runners/core-java
  Move TimerInternalsFactory to runners/core-java
  Move StateInternalsFactory to runners/core-java
  Remove comment-only uses that block runners/core-java migrations
  Move WindowingInternals to runners/core-java
  Delete long-deprecated StateContexts methods
  Move InMemoryStateInternals to runners/core-java
  Eliminate use of InMemoryStateInternals in the Java SDK


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/224e4476
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/224e4476
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/224e4476

Branch: refs/heads/master
Commit: 224e447653e5fef60982a230127b974dcb700161
Parents: e0189f3 fd7e614
Author: Kenneth Knowles 
Authored: Mon Feb 6 14:30:06 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 14:30:06 2017 -0800

--
 .../apex/translation/TranslationContext.java|   2 +-
 .../operators/ApexGroupByKeyOperator.java   |  10 +-
 .../operators/ApexParDoOperator.java|   4 +-
 .../translation/utils/ApexStateInternals.java   |   8 +-
 .../apex/translation/utils/NoOpStepContext.java |   4 +-
 .../utils/ApexStateInternalsTest.java   |   6 +-
 .../beam/runners/core/BaseExecutionContext.java |   2 -
 .../beam/runners/core/ExecutionContext.java |   2 -
 .../GroupAlsoByWindowViaOutputBufferDoFn.java   |   3 -
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |   3 -
 .../GroupAlsoByWindowViaWindowSetNewDoFn.java   |   4 -
 .../runners/core/InMemoryStateInternals.java| 439 +++
 .../runners/core/InMemoryTimerInternals.java|   2 -
 .../apache/beam/runners/core/KeyedWorkItem.java |   2 +-
 .../beam/runners/core/KeyedWorkItemCoder.java   |   4 +-
 .../beam/runners/core/KeyedWorkItems.java   |   2 +-
 .../core/LateDataDroppingDoFnRunner.java|   1 -
 .../runners/core/MergingActiveWindowSet.java|   2 -
 .../org/apache/beam/runners/core/OldDoFn.java   |   1 -
 .../beam/runners/core/PaneInfoTracker.java  |   1 -
 .../runners/core/ReduceFnContextFactory.java|   8 +-
 .../beam/runners/core/ReduceFnRunner.java   |   6 +-
 .../beam/runners/core/SideInputHandler.java |   2 -
 .../beam/runners/core/SimpleDoFnRunner.java |   3 -
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   3 -
 .../beam/runners/core/SplittableParDo.java  |   6 -
 .../beam/runners/core/StateInternals.java   |  60 +++
 .../runners/core/StateInternalsFactory.java |  35 ++
 .../beam/runners/core/StateNamespace.java   |  56 +++
 .../runners/core/StateNamespaceForTest.java |  65 +++
 .../beam/runners/core/StateNamespaces.java  | 278 
 .../apache/beam/runners/core/StateTable.java|  85 
 .../core/TestInMemoryStateInternals.java|  64 +++
 .../beam/runners/core/TimerInternals.java   | 286 
 .../runners/core/TimerInternalsFactory.java |  35 ++
 .../apache/beam/runners/core/WatermarkHold.java |   1 -
 .../beam/runners/core/WindowingInternals.java   |  81 
 .../core/WindowingInternalsAdapters.java|   1 -
 .../TriggerStateMachineContextFactory.java  |   6 +-
 ...roupAlsoByWindowViaOutputBufferDoFnTest.java |   1 -
 .../core/GroupAlsoByWindowsProperties.java  |   5 -
 .../core/InMemoryStateInternalsTest.java| 357 +++
 .../core/InMemoryTimerInternalsTest.java|   4 +-
 .../runners/core/KeyedWorkItemCoderTest.java|   3 +-
 .../core/LateDataDroppingDoFnRunnerTest.java|   1 -
 .../core/MergingActiveWindowSetTest.java|   2 -
 .../core/PushbackSideInputDoFnRunnerTest.java   |   3 +-
 .../beam/runners/core/ReduceFnTester.java   |   7 +-
 .../beam/runners/core/SideInputHandlerTest.java |   1 -
 .../beam/runners/core/SimpleDoFnRunnerTest.java |   4 +-
 .../runners/core/SimpleOldDoFnRunnerTest.java   |   1 -
 .../beam/runners/core/SplittableParDoTest.java  |   8 +-
 .../beam/runners/core/StateNamespacesTest.java  | 130 ++
 .../beam/runners/core/TimerInternalsTest.java   | 105 +
 .../core/triggers/NeverStateMachineTest.java|   1 -
 .../triggers/TriggerStateMachineTester.java |  14 +-
 .../CopyOnAccessInMemoryStateInternals.java |  18 +-
 .../runners/direct/DirectExecutionContext.java  |   2 +-
 .../beam/runners/direct/DirectRunner.java   |   2 +-
 .../runners/direct/DirectTimerInternals.java|   4 +-
 ...ecycleManagerRemovingTransformEvaluator.java |   2 +-
 .../beam/runners/direct/EvaluationContext.java  |   2 +-
 .../direct/ExecutorServiceParallelExecutor.java |   2 +-
 

[02/15] beam git commit: Move InMemoryStateInternals to runners/core-java

[kenn]

Move InMemoryStateInternals to runners/core-java


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/af391b88
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/af391b88
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/af391b88

Branch: refs/heads/master
Commit: af391b88cfc38659f594799bb58b6090a7bcd3a4
Parents: 2b6698d
Author: Kenneth Knowles 
Authored: Thu Jan 26 21:03:42 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 09:26:06 2017 -0800

--
 .../runners/core/InMemoryStateInternals.java| 442 +++
 .../core/TestInMemoryStateInternals.java|  65 +++
 .../core/GroupAlsoByWindowsProperties.java  |   1 -
 .../core/InMemoryStateInternalsTest.java| 359 +++
 .../core/MergingActiveWindowSetTest.java|   1 -
 .../beam/runners/core/ReduceFnTester.java   |   1 -
 .../beam/runners/core/SideInputHandlerTest.java |   1 -
 .../beam/runners/core/SplittableParDoTest.java  |   1 -
 .../triggers/TriggerStateMachineTester.java |   2 +-
 .../CopyOnAccessInMemoryStateInternals.java |  12 +-
 .../spark/translation/TranslationUtils.java |   3 +-
 .../sdk/util/state/InMemoryStateInternals.java  | 430 --
 .../util/state/TestInMemoryStateInternals.java  |  61 ---
 .../util/state/InMemoryStateInternalsTest.java  | 348 ---
 14 files changed, 874 insertions(+), 853 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/af391b88/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryStateInternals.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryStateInternals.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryStateInternals.java
new file mode 100644
index 000..059e32d
--- /dev/null
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryStateInternals.java
@@ -0,0 +1,442 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.transforms.Combine.CombineFn;
+import org.apache.beam.sdk.transforms.Combine.KeyedCombineFn;
+import 
org.apache.beam.sdk.transforms.CombineWithContext.KeyedCombineFnWithContext;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.OutputTimeFn;
+import org.apache.beam.sdk.util.CombineFnUtil;
+import org.apache.beam.sdk.util.state.AccumulatorCombiningState;
+import org.apache.beam.sdk.util.state.BagState;
+import org.apache.beam.sdk.util.state.ReadableState;
+import org.apache.beam.sdk.util.state.State;
+import org.apache.beam.sdk.util.state.StateContext;
+import org.apache.beam.sdk.util.state.StateContexts;
+import org.apache.beam.sdk.util.state.StateInternals;
+import org.apache.beam.sdk.util.state.StateNamespace;
+import org.apache.beam.sdk.util.state.StateTable;
+import org.apache.beam.sdk.util.state.StateTag;
+import org.apache.beam.sdk.util.state.StateTag.StateBinder;
+import org.apache.beam.sdk.util.state.ValueState;
+import org.apache.beam.sdk.util.state.WatermarkHoldState;
+import org.joda.time.Instant;
+
+/**
+ * In-memory implementation of {@link StateInternals}. Used in {@code 
BatchModeExecutionContext}
+ * and for running tests that need state.
+ */
+@Experimental(Kind.STATE)
+public class InMemoryStateInternals implements StateInternals {
+
+  public static  InMemoryStateInternals forKey(K key) {
+return new InMemoryStateInternals<>(key);
+  }
+
+  private final K key;
+
+  protected InMemoryStateInternals(K key) {
+this.key = key;
+  }
+
+  @Override
+  public K getKey() {
+

[12/15] beam git commit: Upgrade Dataflow container version to beam-master-20170203

[kenn]

Upgrade Dataflow container version to beam-master-20170203


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fd7e6149
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fd7e6149
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fd7e6149

Branch: refs/heads/master
Commit: fd7e6149a038a6480b6d2f4f76a738bb83c47476
Parents: e77e7f0
Author: Kenneth Knowles 
Authored: Fri Feb 3 19:08:52 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 09:26:56 2017 -0800

--
 runners/google-cloud-dataflow-java/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/fd7e6149/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 35f31b9..0125e2e 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -33,7 +33,7 @@
   jar
 
   
-
beam-master-20170126
+
beam-master-20170203
 6
   
 

[03/15] beam git commit: Eliminate use of InMemoryStateInternals in the Java SDK

[kenn]

Eliminate use of InMemoryStateInternals in the Java SDK


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2b6698d1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2b6698d1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2b6698d1

Branch: refs/heads/master
Commit: 2b6698d1b136f251436aa8048cf145695ffb5f16
Parents: 7564486
Author: Kenneth Knowles 
Authored: Thu Jan 26 21:00:15 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 09:26:06 2017 -0800

--
 .../apache/beam/runners/core/SplittableParDoTest.java|  5 -
 .../java/org/apache/beam/sdk/transforms/DoFnTester.java  | 11 ---
 2 files changed, 4 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2b6698d1/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
index bb7fd8c..ad0b01d 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
@@ -54,6 +54,7 @@ import org.apache.beam.sdk.transforms.windowing.PaneInfo;
 import org.apache.beam.sdk.util.SideInputReader;
 import org.apache.beam.sdk.util.TimerInternals;
 import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.state.InMemoryStateInternals;
 import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.util.state.StateInternalsFactory;
 import org.apache.beam.sdk.util.state.TimerInternalsFactory;
@@ -207,6 +208,7 @@ public class SplittableParDoTest {
 private Instant currentProcessingTime;
 
 private InMemoryTimerInternals timerInternals;
+private InMemoryStateInternals stateInternals;
 
 ProcessFnTester(
 Instant currentProcessingTime,
@@ -221,11 +223,12 @@ public class SplittableParDoTest {
   fn, inputCoder, restrictionCoder, IntervalWindow.getCoder());
   this.tester = DoFnTester.of(processFn);
   this.timerInternals = new InMemoryTimerInternals();
+  this.stateInternals = InMemoryStateInternals.forKey("dummy");
   processFn.setStateInternalsFactory(
   new StateInternalsFactory() {
 @Override
 public StateInternals stateInternalsForKey(String key) {
-  return tester.getStateInternals();
+  return stateInternals;
 }
   });
   processFn.setTimerInternalsFactory(

http://git-wip-us.apache.org/repos/asf/beam/blob/2b6698d1/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
index 87ae1f5..cc5281c 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
@@ -48,8 +48,6 @@ import org.apache.beam.sdk.util.SerializableUtils;
 import org.apache.beam.sdk.util.Timer;
 import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.state.InMemoryStateInternals;
-import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TimestampedValue;
 import org.apache.beam.sdk.values.TupleTag;
@@ -135,11 +133,6 @@ public class DoFnTester implements 
AutoCloseable {
 windowValues.put(window, value);
   }
 
-  @SuppressWarnings("unchecked")
-  public  StateInternals getStateInternals() {
-return (StateInternals) stateInternals;
-  }
-
   public PipelineOptions getPipelineOptions() {
 return options;
   }
@@ -228,8 +221,6 @@ public class DoFnTester implements 
AutoCloseable {
 }
 TestContext context = new TestContext();
 context.setupDelegateAggregators();
-// State and timer internals are per-bundle.
-stateInternals = InMemoryStateInternals.forKey(new Object());
 try {
   fnInvoker.invokeStartBundle(context);
 } catch (UserCodeException e) {
@@ -768,8 +759,6 @@ public class DoFnTester implements 
AutoCloseable {
   /** The outputs from the {@link DoFn} under test. */
   private Map> outputs;
 
-  private InMemoryStateInternals stateInternals;
-
   /** The state of processing of the 

[07/15] beam git commit: Remove comment-only uses that block runners/core-java migrations

[kenn]

Remove comment-only uses that block runners/core-java migrations


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1d7c6b06
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1d7c6b06
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1d7c6b06

Branch: refs/heads/master
Commit: 1d7c6b0638d521cf221bb7456a0e976371ca1b07
Parents: 949ab3a
Author: Kenneth Knowles 
Authored: Thu Jan 26 21:09:41 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 6 09:26:06 2017 -0800

--
 .../java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java  | 2 --
 .../main/java/org/apache/beam/sdk/util/state/ReadableState.java  | 4 ++--
 2 files changed, 2 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1d7c6b06/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
index c9d178a..de5b1e1 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/PaneInfo.java
@@ -65,8 +65,6 @@ public final class PaneInfo {
* produces a final pane, it will not be merged into any new windows.
*
* The predictions above are made using the mechanism of watermarks.
-   * See {@link org.apache.beam.sdk.util.TimerInternals} for more information
-   * about watermarks.
*
* We can state some properties of {@code LATE} and {@code ON_TIME} 
panes, but first need some
* definitions:

http://git-wip-us.apache.org/repos/asf/beam/blob/1d7c6b06/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/ReadableState.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/ReadableState.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/ReadableState.java
index 3b4cb7b..c3e9936 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/ReadableState.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/ReadableState.java
@@ -35,14 +35,14 @@ public interface ReadableState {
*
* If there will be many calls to {@link #read} for different state in 
short succession,
* you should first call {@link #readLater} for all of them so the reads can 
potentially be
-   * batched (depending on the underlying {@link StateInternals} 
implementation}.
+   * batched (depending on the underlying implementation}.
*/
   T read();
 
   /**
* Indicate that the value will be read later.
*
-   * This allows a {@link StateInternals} implementation to start an 
asynchronous prefetch or
+   * This allows an implementation to start an asynchronous prefetch or
* to include this state in the next batch of reads.
*
* @return this for convenient chaining

[jira] [Commented] (BEAM-588) All runners should support ProfilingOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854826#comment-15854826
 ] 

ASF GitHub Bot commented on BEAM-588:
-

GitHub user yk5 opened a pull request:

https://github.com/apache/beam/pull/1928

[BEAM-588] Add MemoryReporter to python-sdk

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yk5/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1928.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1928


commit 1c9a36599f1ae9b86205c59ba3754dba139921d2
Author: Younghee Kwon 
Date:   2017-02-06T20:35:50Z

To add sdks/python/utils/profiler a MemoryReporter that tracks heap 
profiles.

commit 9525392a39234af4efd808c3cbe17e930d65bf94
Author: Younghee Kwon 
Date:   2017-02-06T21:55:36Z

added comment about guppy




> All runners should support ProfilingOptions
> ---
>
> Key: BEAM-588
> URL: https://issues.apache.org/jira/browse/BEAM-588
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/utils/options.py#L366
> This is useful for profiling pipelines in different environments.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1928: [BEAM-588] Add MemoryReporter to python-sdk

[yk5]

GitHub user yk5 opened a pull request:

https://github.com/apache/beam/pull/1928

[BEAM-588] Add MemoryReporter to python-sdk

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yk5/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1928.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1928


commit 1c9a36599f1ae9b86205c59ba3754dba139921d2
Author: Younghee Kwon 
Date:   2017-02-06T20:35:50Z

To add sdks/python/utils/profiler a MemoryReporter that tracks heap 
profiles.

commit 9525392a39234af4efd808c3cbe17e930d65bf94
Author: Younghee Kwon 
Date:   2017-02-06T21:55:36Z

added comment about guppy




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1406) Remove deprecated fileio.TextFileSink

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-1406:


 Summary: Remove deprecated fileio.TextFileSink
 Key: BEAM-1406
 URL: https://issues.apache.org/jira/browse/BEAM-1406
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Affects Versions: Not applicable
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath


Users should be using textio.WriteToText() instead of fileio.TextFileSink.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1404) Coveralls should not mark the build as "failed"

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854766#comment-15854766
 ] 

Davor Bonaci commented on BEAM-1404:


This is just a config number inside Coveralls, the default is zero. Easy to 
change once we correct the underlying data.

> Coveralls should not mark the build as "failed"
> ---
>
> Key: BEAM-1404
> URL: https://issues.apache.org/jira/browse/BEAM-1404
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Jason Kuster
>
> For those folks who use the GitHub summary of the status of a PR, lots of 
> good PRs are marked failed because of Coveralls believing the coverage 
> lowered by a few thousandths of a percent. In fact, usually the decline is 
> false / not possible, and even if it were, we should be exercising human 
> judgment about whether it was significant.
> So coveralls should not fail the build. Happy to leave every other bit in 
> place as we learn how to most effectively use it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1389) Remove or update Flink Runner README.md

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854761#comment-15854761
 ] 

Davor Bonaci commented on BEAM-1389:


Website is much more visible; I'd suggest that over README.md somewhere deep in 
the codebase.

> Remove or update Flink Runner README.md
> ---
>
> Key: BEAM-1389
> URL: https://issues.apache.org/jira/browse/BEAM-1389
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>
> The readme is outdated by now. We could either remove it or just put a link 
> to the Runner documentation on the website.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1395) SparkGroupAlsoByWindowFn not sorting grouped elements by timestamp

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela resolved BEAM-1395.
-
   Resolution: Fixed
Fix Version/s: 0.6.0

> SparkGroupAlsoByWindowFn not sorting grouped elements by timestamp
> --
>
> Key: BEAM-1395
> URL: https://issues.apache.org/jira/browse/BEAM-1395
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.6.0
>
>
> SparkGroupAlsoByWindowFn relies on the grouped elements (pre key) to be 
> sorted by their timestamp, which is not the case, and so could cause: 
> {code}
> IllegalStateException: Cannot move input watermark time backwards
> {code}
> We should sort the values first, just like with {{Combine}} implementations: 
> https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkKeyedCombineFn.java#L73



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1395) SparkGroupAlsoByWindowFn not sorting grouped elements by timestamp

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854652#comment-15854652
 ] 

ASF GitHub Bot commented on BEAM-1395:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1922


> SparkGroupAlsoByWindowFn not sorting grouped elements by timestamp
> --
>
> Key: BEAM-1395
> URL: https://issues.apache.org/jira/browse/BEAM-1395
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> SparkGroupAlsoByWindowFn relies on the grouped elements (pre key) to be 
> sorted by their timestamp, which is not the case, and so could cause: 
> {code}
> IllegalStateException: Cannot move input watermark time backwards
> {code}
> We should sort the values first, just like with {{Combine}} implementations: 
> https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkKeyedCombineFn.java#L73



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #1922

[amitsela]

This closes #1922


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e0189f35
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e0189f35
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e0189f35

Branch: refs/heads/master
Commit: e0189f35221b587fddfe8073a129fa061179eac9
Parents: 1e49ee8 71197ae
Author: Sela 
Authored: Mon Feb 6 21:39:29 2017 +0200
Committer: Sela 
Committed: Mon Feb 6 21:39:29 2017 +0200

--
 .../translation/SparkGroupAlsoByWindowFn.java   | 53 +++-
 1 file changed, 18 insertions(+), 35 deletions(-)
--

[GitHub] beam pull request #1922: [BEAM-1395] SparkGroupAlsoByWindowFn not sorting gr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1922


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-1395] Remove chunking.

[amitsela]

Repository: beam
Updated Branches:
  refs/heads/master 1e49ee8f2 -> e0189f352


[BEAM-1395] Remove chunking.

fixup! formatting.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/71197ae6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/71197ae6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/71197ae6

Branch: refs/heads/master
Commit: 71197ae61637d9f6317bd5abd7194d67044fee9d
Parents: 1e49ee8
Author: Sela 
Authored: Mon Feb 6 10:09:35 2017 +0200
Committer: Sela 
Committed: Mon Feb 6 21:38:51 2017 +0200

--
 .../translation/SparkGroupAlsoByWindowFn.java   | 53 +++-
 1 file changed, 18 insertions(+), 35 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/71197ae6/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGroupAlsoByWindowFn.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGroupAlsoByWindowFn.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGroupAlsoByWindowFn.java
index 34eea65..9d84481 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGroupAlsoByWindowFn.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGroupAlsoByWindowFn.java
@@ -18,7 +18,6 @@
 
 package org.apache.beam.runners.spark.translation;
 
-import com.google.common.collect.Iterables;
 import java.util.ArrayList;
 import java.util.Collection;
 import java.util.List;
@@ -86,7 +85,7 @@ public class SparkGroupAlsoByWindowFn
   public Iterable>> call(
   WindowedValue>> windowedValue) 
throws Exception {
 K key = windowedValue.getValue().getKey();
-Iterable inputs = 
windowedValue.getValue().getValue();
+Iterable values = 
windowedValue.getValue().getValue();
 
 //-- based on GroupAlsoByWindowsViaOutputBufferDoFn --//
 
@@ -131,24 +130,8 @@ public class SparkGroupAlsoByWindowFn
 reduceFn,
 runtimeContext.getPipelineOptions());
 
-Iterable> chunks = Iterables.partition(inputs, 
1000);
-for (Iterable chunk : chunks) {
-  // Process the chunk of elements.
-  reduceFnRunner.processElements(chunk);
-
-  // Then, since elements are sorted by their timestamp, advance the input 
watermark
-  // to the first element.
-  
timerInternals.advanceInputWatermark(chunk.iterator().next().getTimestamp());
-  // Advance the processing times.
-  timerInternals.advanceProcessingTime(Instant.now());
-  timerInternals.advanceSynchronizedProcessingTime(Instant.now());
-
-  // Fire all the eligible timers.
-  fireEligibleTimers(timerInternals, reduceFnRunner);
-
-  // Leave the output watermark undefined. Since there's no late data in 
batch mode
-  // there's really no need to track it as we do for streaming.
-}
+// Process the grouped values.
+reduceFnRunner.processElements(values);
 
 // Finish any pending windows by advancing the input watermark to infinity.
 timerInternals.advanceInputWatermark(BoundedWindow.TIMESTAMP_MAX_VALUE);
@@ -168,21 +151,21 @@ public class SparkGroupAlsoByWindowFn
   ReduceFnRunner reduceFnRunner) throws 
Exception {
 List timers = new ArrayList<>();
 while (true) {
-TimerInternals.TimerData timer;
-while ((timer = timerInternals.removeNextEventTimer()) != null) {
-  timers.add(timer);
-}
-while ((timer = timerInternals.removeNextProcessingTimer()) != null) {
-  timers.add(timer);
-}
-while ((timer = 
timerInternals.removeNextSynchronizedProcessingTimer()) != null) {
-  timers.add(timer);
-}
-if (timers.isEmpty()) {
-  break;
-}
-reduceFnRunner.onTimers(timers);
-timers.clear();
+  TimerInternals.TimerData timer;
+  while ((timer = timerInternals.removeNextEventTimer()) != null) {
+timers.add(timer);
+  }
+  while ((timer = timerInternals.removeNextProcessingTimer()) != null) {
+timers.add(timer);
+  }
+  while ((timer = timerInternals.removeNextSynchronizedProcessingTimer()) 
!= null) {
+timers.add(timer);
+  }
+  if (timers.isEmpty()) {
+break;
+  }
+  reduceFnRunner.onTimers(timers);
+  timers.clear();
 }
   }
 

[jira] [Commented] (BEAM-1388) Update default configuration of retry decorator so that wait times are more practical

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854549#comment-15854549
 ] 

ASF GitHub Bot commented on BEAM-1388:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1916


> Update default configuration of retry decorator so that wait times are more 
> practical
> -
>
> Key: BEAM-1388
> URL: https://issues.apache.org/jira/browse/BEAM-1388
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> Currently retry decorator [1] has following default configuration.
> initial delay: 5 sec
> number of retries: 16
> factor: 2
> max delay: 4 hours
> I propose updating these values to following so that default values are more 
> practical.
> initial delay: 5 sec
> number of retries: 7
> factor: 2
> max delay: 1 hour
> With this update, for maximum number of retries, system will wait for 635 sec 
> while wait before last retry being 320 sec.
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/retry.py#L117



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1916: [BEAM-1388] Updates default values used by retry de...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1916


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Updates default values used by retry decorator.

[altay]

Repository: beam
Updated Branches:
  refs/heads/master d18db9344 -> 1e49ee8f2


Updates default values used by retry decorator.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/50d00cb4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/50d00cb4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/50d00cb4

Branch: refs/heads/master
Commit: 50d00cb4620c6ed22f51bf6161fa0799396d08c1
Parents: d18db93
Author: Chamikara Jayalath 
Authored: Fri Feb 3 17:32:49 2017 -0800
Committer: Ahmet Altay 
Committed: Mon Feb 6 10:48:29 2017 -0800

--
 sdks/python/apache_beam/utils/retry.py  |  8 
 sdks/python/apache_beam/utils/retry_test.py | 15 +++
 2 files changed, 11 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/50d00cb4/sdks/python/apache_beam/utils/retry.py
--
diff --git a/sdks/python/apache_beam/utils/retry.py 
b/sdks/python/apache_beam/utils/retry.py
index 3874c7a..648d726 100644
--- a/sdks/python/apache_beam/utils/retry.py
+++ b/sdks/python/apache_beam/utils/retry.py
@@ -54,11 +54,11 @@ class FuzzedExponentialIntervals(object):
   [(1 - fuzz) * d, d].
 max_delay_sec: Maximum delay (in seconds). After this limit is reached,
   further tries use max_delay_sec instead of exponentially increasing
-  the time. Defaults to 4 hours.
+  the time. Defaults to 1 hour.
   """
 
   def __init__(self, initial_delay_secs, num_retries, factor=2, fuzz=0.5,
-   max_delay_secs=60 * 60 * 4):
+   max_delay_secs=60 * 60 * 1):
 self._initial_delay_secs = initial_delay_secs
 self._num_retries = num_retries
 self._factor = factor
@@ -115,9 +115,9 @@ def no_retries(fun):
 
 
 def with_exponential_backoff(
-num_retries=16, initial_delay_secs=5.0, logger=logging.warning,
+num_retries=7, initial_delay_secs=5.0, logger=logging.warning,
 retry_filter=retry_on_server_errors_filter,
-clock=Clock(), fuzz=True, factor=2, max_delay_secs=60 * 60 * 4):
+clock=Clock(), fuzz=True, factor=2, max_delay_secs=60 * 60):
   """Decorator with arguments that control the retry logic.
 
   Args:

http://git-wip-us.apache.org/repos/asf/beam/blob/50d00cb4/sdks/python/apache_beam/utils/retry_test.py
--
diff --git a/sdks/python/apache_beam/utils/retry_test.py 
b/sdks/python/apache_beam/utils/retry_test.py
index 46f6e54..dd1741c 100644
--- a/sdks/python/apache_beam/utils/retry_test.py
+++ b/sdks/python/apache_beam/utils/retry_test.py
@@ -75,7 +75,7 @@ class RetryTest(unittest.TestCase):
 
   def transient_failure(self, a, b):
 self.calls += 1
-if self.calls > 8:
+if self.calls > 4:
   return a + b
 raise NotImplementedError
 
@@ -99,7 +99,7 @@ class RetryTest(unittest.TestCase):
   retry.with_exponential_backoff(clock=self.clock)(
   self.permanent_failure),
   10, b=20)
-self.assertEqual(len(self.clock.calls), 16)
+self.assertEqual(len(self.clock.calls), 7)
 
   def test_with_explicit_number_of_retries(self):
 self.assertRaises(NotImplementedError,
@@ -133,7 +133,7 @@ class RetryTest(unittest.TestCase):
   fuzz=False)(
   self.permanent_failure),
   10, b=20)
-self.assertEqual(len(self.clock.calls), 16)
+self.assertEqual(len(self.clock.calls), 7)
 self.assertEqual(self.clock.calls[0], 10.0)
 
   def test_log_calls_for_permanent_failure(self):
@@ -142,7 +142,7 @@ class RetryTest(unittest.TestCase):
   clock=self.clock, logger=self.logger.log)(
   self.permanent_failure),
   10, b=20)
-self.assertEqual(len(self.logger.calls), 16)
+self.assertEqual(len(self.logger.calls), 7)
 for message, func_name, exn_name  in self.logger.calls:
   self.assertTrue(message.startswith('Retry with exponential backoff:'))
   self.assertEqual(exn_name, 'NotImplementedError\n')
@@ -153,11 +153,10 @@ class RetryTest(unittest.TestCase):
 clock=self.clock, logger=self.logger.log, fuzz=False)(
 self.transient_failure)(10, b=20)
 self.assertEqual(result, 30)
-self.assertEqual(len(self.clock.calls), 8)
+self.assertEqual(len(self.clock.calls), 4)
 self.assertEqual(self.clock.calls,
- [5.0 * 1, 5.0 * 2, 5.0 * 4, 5.0 * 8,
-  5.0 * 16, 5.0 * 32, 5.0 * 64, 5.0 * 128])
-self.assertEqual(len(self.logger.calls), 8)
+ [5.0 * 1, 5.0 * 2, 5.0 * 4, 5.0 * 8,])
+self.assertEqual(len(self.logger.calls), 4)
 for message, 

[2/2] beam git commit: This closes #1916

[altay]

This closes #1916


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1e49ee8f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1e49ee8f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1e49ee8f

Branch: refs/heads/master
Commit: 1e49ee8f2d7d7044bb99e1bcd06a9fca06e06c8b
Parents: d18db93 50d00cb
Author: Ahmet Altay 
Authored: Mon Feb 6 10:48:41 2017 -0800
Committer: Ahmet Altay 
Committed: Mon Feb 6 10:48:41 2017 -0800

--
 sdks/python/apache_beam/utils/retry.py  |  8 
 sdks/python/apache_beam/utils/retry_test.py | 15 +++
 2 files changed, 11 insertions(+), 12 deletions(-)
--

[jira] [Commented] (BEAM-1369) some unit tests in python take longer than 10s to run

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854497#comment-15854497
 ] 

ASF GitHub Bot commented on BEAM-1369:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1923


> some unit tests in python take longer than 10s to run
> -
>
> Key: BEAM-1369
> URL: https://issues.apache.org/jira/browse/BEAM-1369
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
>
> 4 or 5 take up 50% of the time to run the unit tests.
> apache_beam.transforms.combiners_test.CombineTest.test_sample: 17.1545s
> apache_beam.io.avroio_test.TestAvro.test_dynamic_work_rebalancing_exhaustive: 
> 16.s
> apache_beam.examples.complete.estimate_pi_test.EstimatePiTest.test_basics: 
> 8.3667s
> apache_beam.io.textio_test.TextSourceTest.test_dynamic_work_rebalancing: 
> 1.9172s
> apache_beam.io.avroio_test.TestAvro.test_read_with_splitting_multiple_blocks: 
> 1.3939s
> apache_beam.io.textio_test.TextSourceTest.test_read_gzip_large: 1.2832s
> apache_beam.io.source_test_utils_test.SourceTestUtilsTest.test_split_at_fraction_exhaustive:
>  1.2787s
> apache_beam.io.bigquery_test.TestBigQueryReader.test_read_from_table_and_job_complete_retry:
>  1.0082s



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1923: [BEAM-1369] Reduce test times for two retry based t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1923


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1923

[altay]

This closes #1923


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d18db934
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d18db934
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d18db934

Branch: refs/heads/master
Commit: d18db93440e8308fe79f2e26875178ad4d61e7cf
Parents: 7564486 ade9c33
Author: Ahmet Altay 
Authored: Mon Feb 6 10:18:27 2017 -0800
Committer: Ahmet Altay 
Committed: Mon Feb 6 10:18:27 2017 -0800

--
 sdks/python/apache_beam/io/gcsio_test.py| 2 +-
 sdks/python/apache_beam/utils/retry_test.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--

[1/2] beam git commit: Reduce test times for two retry based tests

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 7564486f5 -> d18db9344


Reduce test times for two retry based tests


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ade9c33d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ade9c33d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ade9c33d

Branch: refs/heads/master
Commit: ade9c33d49997515cac8edf16cb9a5122c3a71b5
Parents: 7564486
Author: Sourabh Bajaj 
Authored: Sun Feb 5 20:18:52 2017 -0800
Committer: Ahmet Altay 
Committed: Mon Feb 6 10:17:52 2017 -0800

--
 sdks/python/apache_beam/io/gcsio_test.py| 2 +-
 sdks/python/apache_beam/utils/retry_test.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ade9c33d/sdks/python/apache_beam/io/gcsio_test.py
--
diff --git a/sdks/python/apache_beam/io/gcsio_test.py 
b/sdks/python/apache_beam/io/gcsio_test.py
index 6532707..06371aa 100644
--- a/sdks/python/apache_beam/io/gcsio_test.py
+++ b/sdks/python/apache_beam/io/gcsio_test.py
@@ -435,7 +435,7 @@ class TestGCSIO(unittest.TestCase):
 f = self.gcs.open(file_name)
 random.seed(0)
 f.buffer_size = 1024 * 1024
-f.segment_timeout = 0.1
+f.segment_timeout = 0.01
 self.assertEqual(f.mode, 'r')
 f._real_get_segment = f._get_segment
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ade9c33d/sdks/python/apache_beam/utils/retry_test.py
--
diff --git a/sdks/python/apache_beam/utils/retry_test.py 
b/sdks/python/apache_beam/utils/retry_test.py
index 7570ca0..46f6e54 100644
--- a/sdks/python/apache_beam/utils/retry_test.py
+++ b/sdks/python/apache_beam/utils/retry_test.py
@@ -51,7 +51,7 @@ def test_function(a, b):
   raise NotImplementedError
 
 
-@retry.with_exponential_backoff(initial_delay_secs=1.0, num_retries=1)
+@retry.with_exponential_backoff(initial_delay_secs=0.1, num_retries=1)
 def test_function_with_real_clock(a, b):
   _ = a, b
   raise NotImplementedError

[jira] [Updated] (BEAM-440) Create.values() returns a type-unsafe Coder

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-440:
-
Labels: backward-incompatible newbie starter  (was: newbie starter)

> Create.values() returns a type-unsafe Coder
> ---
>
> Key: BEAM-440
> URL: https://issues.apache.org/jira/browse/BEAM-440
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Jason White
>  Labels: backward-incompatible, newbie, starter
>
> {{Create.values()}} with no arguments will default to a {{VoidCoder}}, unless 
> one is set later with {{setCoder(Coder)}}.
> Although it will encode its input correctly, this seems like a bad choice in 
> many cases. E.g., with {{Flatten}}:
> {code}
> PCollection> initial = p.apply("First", 
> Create.>of());
> PCollection> second =
> p.apply("Second", Create.of("a", "b")).apply(ParDo.of(new 
> MyAvroDoFn()));
> PCollectionList
> .of(initial).and(second)
> .apply(Flatten.>pCollections());
> {code}
> This crashes trying to cast a KV from "Second" to a Void.class.
> 1. Suggest throwing a warning in {{getDefaultOutputCoder}} when defaulting to 
> {{VoidCoder}} for an empty elements list. Should this be an error?
> 2. Suggest adding something like {{Create.empty(TypeDescriptor)}} to handle 
> this case properly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #2211

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-298) Make TestPipeline implement the TestRule interface

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-298:
-
Issue Type: Improvement  (was: Bug)

> Make TestPipeline implement the TestRule interface
> --
>
> Key: BEAM-298
> URL: https://issues.apache.org/jira/browse/BEAM-298
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Stas Levin
>Priority: Minor
> Fix For: 0.5.0
>
>
> https://github.com/junit-team/junit4/wiki/Rules
> A JUnit Rule allows a test to use a field annotated with @Rule to wrap 
> executing tests. Doing so allows the TestPipeline to, at the time the test 
> completes, assert that all applied transforms have been executed. This 
> ensures that all unit tests that utilize a TestPipeline rule either are 
> declared to explicitly not expect to execute or have executed the pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #132: Add 0.5.0 release

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/132


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[08/52] [abbrv] [partial] beam-site git commit: Regenerate website

[davor]

http://git-wip-us.apache.org/repos/asf/beam-site/blob/7b1f1576/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/triggers/DefaultTriggerStateMachine.html
--
diff --git 
a/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/triggers/DefaultTriggerStateMachine.html
 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/triggers/DefaultTriggerStateMachine.html
new file mode 100644
index 000..d5308e6
--- /dev/null
+++ 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/triggers/DefaultTriggerStateMachine.html
@@ -0,0 +1,458 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+DefaultTriggerStateMachine (Apache Beam SDK for Java, version 
0.5.0)
+
+
+
+
+
+
+var methods = {"i0":10,"i1":10,"i2":9,"i3":10,"i4":10,"i5":10,"i6":10};
+var tabs = {65535:["t0","All Methods"],1:["t1","Static 
Methods"],2:["t2","Instance Methods"],8:["t4","Concrete Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.core.triggers
+Class 
DefaultTriggerStateMachine
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">java.lang.Object
+
+
+org.apache.beam.runners.core.triggers.TriggerStateMachine
+
+
+org.apache.beam.runners.core.triggers.DefaultTriggerStateMachine
+
+
+
+
+
+
+
+
+
+All Implemented Interfaces:
+http://docs.oracle.com/javase/7/docs/api/java/io/Serializable.html?is-external=true;
 title="class or interface in java.io">Serializable
+
+
+
+@Experimental(value=TRIGGER)
+public class DefaultTriggerStateMachine
+extends TriggerStateMachine
+A trigger that is equivalent to 
Repeatedly.forever(AfterWatermark.pastEndOfWindow()). See
+ RepeatedlyStateMachine.forever(org.apache.beam.runners.core.triggers.TriggerStateMachine)
 and AfterWatermarkStateMachine.pastEndOfWindow()
 for
+ more details.
+
+See Also:
+Serialized
 Form
+
+
+
+
+
+
+
+
+
+
+
+
+Nested Class Summary
+
+
+
+
+Nested classes/interfaces inherited from 
classorg.apache.beam.runners.core.triggers.TriggerStateMachine
+TriggerStateMachine.MergingTriggerInfo,
 TriggerStateMachine.OnceTriggerStateMachine,
 TriggerStateMachine.OnElementContext,
 TriggerStateMachine.OnMergeContext, 
TriggerStateMachine.TriggerContext, TriggerStateMachine.TriggerInfo
+
+
+
+
+
+
+
+
+Field Summary
+
+
+
+
+Fields inherited from classorg.apache.beam.runners.core.triggers.TriggerStateMachine
+subTriggers
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsStatic MethodsInstance MethodsConcrete Methods
+
+Modifier and Type
+Method and Description
+
+
+void
+clear(TriggerStateMachine.TriggerContextc)
+Clear any state associated with this trigger in the given 
window.
+
+
+
+boolean
+isCompatible(TriggerStateMachineother)
+Returns whether this performs the same triggering as the 
given Trigger.
+
+
+
+static DefaultTriggerStateMachine
+of()
+Returns the default trigger.
+
+
+
+void
+onElement(TriggerStateMachine.OnElementContextc)
+Called every time an element is incorporated into a 
window.
+
+
+
+void
+onFire(TriggerStateMachine.TriggerContextcontext)
+Adjusts the state of the trigger to be ready for the next 
pane.
+
+
+
+void
+onMerge(TriggerStateMachine.OnMergeContextc)
+Called immediately after windows have been merged.
+
+
+
+boolean
+shouldFire(TriggerStateMachine.TriggerContextcontext)
+Returns true if the current state of the 
trigger indicates that its condition
+ is satisfied and it is ready to fire.
+
+
+
+
+
+
+
+Methods inherited from classorg.apache.beam.runners.core.triggers.TriggerStateMachine
+equals,
 hashCode,
 orFinally,
 prefetchOnElement,
 prefetchOnFire,
 prefetchOnMerge,
 prefetchShouldFire,
 subTriggers,
 toString
+
+
+
+
+
+Methods inherited from classjava.lang.http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">Object
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#clone--;
 title="class or interface in java.lang">clone, 

[06/52] [abbrv] [partial] beam-site git commit: Regenerate website

[davor]

http://git-wip-us.apache.org/repos/asf/beam-site/blob/7b1f1576/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/triggers/RepeatedlyStateMachine.html
--
diff --git 
a/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/triggers/RepeatedlyStateMachine.html
 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/triggers/RepeatedlyStateMachine.html
new file mode 100644
index 000..8f0ba69
--- /dev/null
+++ 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/triggers/RepeatedlyStateMachine.html
@@ -0,0 +1,440 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+RepeatedlyStateMachine (Apache Beam SDK for Java, version 0.5.0)
+
+
+
+
+
+
+var methods = {"i0":9,"i1":10,"i2":10,"i3":10,"i4":10,"i5":10};
+var tabs = {65535:["t0","All Methods"],1:["t1","Static 
Methods"],2:["t2","Instance Methods"],8:["t4","Concrete Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.core.triggers
+Class 
RepeatedlyStateMachine
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">java.lang.Object
+
+
+org.apache.beam.runners.core.triggers.TriggerStateMachine
+
+
+org.apache.beam.runners.core.triggers.RepeatedlyStateMachine
+
+
+
+
+
+
+
+
+
+All Implemented Interfaces:
+http://docs.oracle.com/javase/7/docs/api/java/io/Serializable.html?is-external=true;
 title="class or interface in java.io">Serializable
+
+
+
+public class RepeatedlyStateMachine
+extends TriggerStateMachine
+Repeat a trigger, either until some condition is met or 
forever.
+
+ For example, to fire after the end of the window, and every time late data 
arrives:
+  
+ Repeatedly.forever(AfterWatermark.isPastEndOfWindow());
+  
+
+ Repeatedly.forever(someTrigger) behaves like an infinite
+ AfterEach.inOrder(someTrigger, someTrigger, someTrigger, 
...).
+
+See Also:
+Serialized
 Form
+
+
+
+
+
+
+
+
+
+
+
+
+Nested Class Summary
+
+
+
+
+Nested classes/interfaces inherited from 
classorg.apache.beam.runners.core.triggers.TriggerStateMachine
+TriggerStateMachine.MergingTriggerInfo,
 TriggerStateMachine.OnceTriggerStateMachine,
 TriggerStateMachine.OnElementContext,
 TriggerStateMachine.OnMergeContext, 
TriggerStateMachine.TriggerContext, TriggerStateMachine.TriggerInfo
+
+
+
+
+
+
+
+
+Field Summary
+
+
+
+
+Fields inherited from classorg.apache.beam.runners.core.triggers.TriggerStateMachine
+subTriggers
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsStatic MethodsInstance MethodsConcrete Methods
+
+Modifier and Type
+Method and Description
+
+
+static RepeatedlyStateMachine
+forever(TriggerStateMachinerepeated)
+Create a composite trigger that repeatedly executes the 
trigger repeated, firing each
+ time it fires and ignoring any indications to finish.
+
+
+
+void
+onElement(TriggerStateMachine.OnElementContextc)
+Called every time an element is incorporated into a 
window.
+
+
+
+void
+onFire(TriggerStateMachine.TriggerContextcontext)
+Adjusts the state of the trigger to be ready for the next 
pane.
+
+
+
+void
+onMerge(TriggerStateMachine.OnMergeContextc)
+Called immediately after windows have been merged.
+
+
+
+boolean
+shouldFire(TriggerStateMachine.TriggerContextcontext)
+Returns true if the current state of the 
trigger indicates that its condition
+ is satisfied and it is ready to fire.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">String
+toString()
+
+
+
+
+
+
+Methods inherited from classorg.apache.beam.runners.core.triggers.TriggerStateMachine
+clear,
 equals,
 hashCode,
 isCompatible,
 orFinally,
 prefetchOnElement,
 prefetchOnFire,
 prefetchOnMerge,
 prefetchShouldFire,
 subTriggers
+
+
+
+
+
+Methods inherited from classjava.lang.http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">Object
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#clone--;
 title="class or interface in 

[35/52] [abbrv] [partial] beam-site git commit: Regenerate website

[davor]

http://git-wip-us.apache.org/repos/asf/beam-site/blob/7b1f1576/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/GroupByKeyViaGroupByKeyOnly.GroupAlsoByWindow.html
--
diff --git 
a/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/GroupByKeyViaGroupByKeyOnly.GroupAlsoByWindow.html
 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/GroupByKeyViaGroupByKeyOnly.GroupAlsoByWindow.html
new file mode 100644
index 000..2782a24
--- /dev/null
+++ 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/GroupByKeyViaGroupByKeyOnly.GroupAlsoByWindow.html
@@ -0,0 +1,371 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+GroupByKeyViaGroupByKeyOnly.GroupAlsoByWindow (Apache Beam SDK for 
Java, version 0.5.0)
+
+
+
+
+
+
+var methods = {"i0":10,"i1":10,"i2":10,"i3":10};
+var tabs = {65535:["t0","All Methods"],2:["t2","Instance 
Methods"],8:["t4","Concrete Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.core
+Class 
GroupByKeyViaGroupByKeyOnly.GroupAlsoByWindowK,V
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">java.lang.Object
+
+
+org.apache.beam.sdk.transforms.PTransformPCollectionKVK,http://docs.oracle.com/javase/7/docs/api/java/lang/Iterable.html?is-external=true;
 title="class or interface in java.lang">IterableWindowedValueV,PCollectionKVK,http://docs.oracle.co
 m/javase/7/docs/api/java/lang/Iterable.html?is-external=true" title="class or 
interface in java.lang">IterableV
+
+
+org.apache.beam.runners.core.GroupByKeyViaGroupByKeyOnly.GroupAlsoByWindowK,V
+
+
+
+
+
+
+
+
+
+All Implemented Interfaces:
+http://docs.oracle.com/javase/7/docs/api/java/io/Serializable.html?is-external=true;
 title="class or interface in java.io">Serializable, HasDisplayData
+
+
+Enclosing class:
+GroupByKeyViaGroupByKeyOnlyK,V
+
+
+
+public static class GroupByKeyViaGroupByKeyOnly.GroupAlsoByWindowK,V
+extends PTransformPCollectionKVK,http://docs.oracle.com/javase/7/docs/api/java/lang/Iterable.html?is-external=true;
 title="class or interface in java.lang">IterableWindowedValueV,PCollectionKVK,http://docs.oracle.com/javase/7/docs/api/java/la
 ng/Iterable.html?is-external=true" title="class or interface in 
java.lang">IterableV
+Runner-specific primitive that takes a collection of 
timestamp-ordered values associated with
+ each key, groups the values by window, merges windows as needed, and for each 
window in each
+ key, outputs a collection of key/value-list pairs implicitly assigned to the 
window and with
+ the timestamp derived from that window.
+
+See Also:
+Serialized
 Form
+
+
+
+
+
+
+
+
+
+
+
+
+Field Summary
+
+
+
+
+Fields inherited from classorg.apache.beam.sdk.transforms.PTransform
+name
+
+
+
+
+
+
+
+
+Constructor Summary
+
+Constructors
+
+Constructor and Description
+
+
+GroupAlsoByWindow(WindowingStrategy?,?windowingStrategy)
+
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsInstance MethodsConcrete Methods
+
+Modifier and Type
+Method and Description
+
+
+PCollectionKVK,http://docs.oracle.com/javase/7/docs/api/java/lang/Iterable.html?is-external=true;
 title="class or interface in java.lang">IterableV
+expand(PCollectionKVK,http://docs.oracle.com/javase/7/docs/api/java/lang/Iterable.html?is-external=true;
 title="class or interface in java.lang">IterableWindowedValueVinput)
+Applies this PTransform on the given 
InputT, and returns its
+ Output.
+
+
+
+CoderK
+getKeyCoder(CoderKVK,http://docs.oracle.com/javase/7/docs/api/java/lang/Iterable.html?is-external=true;
 title="class or interface in java.lang">IterableWindowedValueVinputCoder)
+
+
+CoderV
+getValueCoder(CoderKVK,http://docs.oracle.com/javase/7/docs/api/java/lang/Iterable.html?is-external=true;
 title="class or interface in java.lang">IterableWindowedValueVinputCoder)
+
+
+WindowingStrategy?,?
+getWindowingStrategy()
+
+
+
+
+
+
+Methods inherited from classorg.apache.beam.sdk.transforms.PTransform
+getDefaultOutputCoder,
 

[45/52] [abbrv] [partial] beam-site git commit: Regenerate website

[davor]

http://git-wip-us.apache.org/repos/asf/beam-site/blob/7b1f1576/content/documentation/sdks/javadoc/0.5.0/index-all.html
--
diff --git a/content/documentation/sdks/javadoc/0.5.0/index-all.html 
b/content/documentation/sdks/javadoc/0.5.0/index-all.html
new file mode 100644
index 000..6b00365
--- /dev/null
+++ b/content/documentation/sdks/javadoc/0.5.0/index-all.html
@@ -0,0 +1,21764 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Index (Apache Beam SDK for Java, version 0.5.0)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+ABCDEFGHIJKLMNOPQRSTUVWXYZ
+
+
+A
+
+absolute(String,
 String...) - Static method in class 
org.apache.beam.sdk.transforms.display.DisplayData.Path
+
+Construct a path from an absolute component path 
hierarchy.
+
+access(StateTag?
 super K, StateT) - Method in interface 
org.apache.beam.sdk.util.state.StateAccessor
+
+Access the storage for the given address in 
the current window.
+
+accessInEachMergingWindow(StateTag?
 super K, StateT) - Method in interface 
org.apache.beam.sdk.util.state.MergingStateAccessor
+
+Analogous to StateAccessor.access(org.apache.beam.sdk.util.state.StateTag?
 super K, StateT), but returned as a map from each window which 
is
+ about to be merged to the corresponding state.
+
+AccumulatingCombineFn()
 - Constructor for class org.apache.beam.sdk.transforms.Combine.AccumulatingCombineFn
+
+accumulatingFiredPanes()
 - Static method in class org.apache.beam.sdk.transforms.windowing.Window
+
+Returns a new Window PTransform 
that uses the registered WindowFn and
+ Triggering behavior, and that accumulates elements in a pane after they are 
triggered.
+
+accumulatingFiredPanes()
 - Method in class org.apache.beam.sdk.transforms.windowing.Window.Bound
+
+Returns a new Window PTransform 
that uses the registered WindowFn and
+ Triggering behavior, and that accumulates elements in a pane after they are 
triggered.
+
+AccumulatorCombiningStateInputT,AccumT,OutputT - Interface 
in org.apache.beam.sdk.util.state
+
+State for a single value that is managed by a Combine.CombineFn.
+
+ackDeadlineSeconds(PubsubClient.SubscriptionPath)
 - Method in class org.apache.beam.sdk.util.PubsubClient
+
+Return the ack deadline, in seconds, for 
subscription.
+
+ackDeadlineSeconds(PubsubClient.SubscriptionPath)
 - Method in class org.apache.beam.sdk.util.PubsubGrpcClient
+
+ackDeadlineSeconds(PubsubClient.SubscriptionPath)
 - Method in class org.apache.beam.sdk.util.PubsubJsonClient
+
+ackDeadlineSeconds(PubsubClient.SubscriptionPath)
 - Method in class org.apache.beam.sdk.util.PubsubTestClient
+
+ackId
 - Variable in class org.apache.beam.sdk.util.PubsubClient.IncomingMessage
+
+Id to pass back to Pubsub to acknowledge receipt of this 
message.
+
+acknowledge(PubsubClient.SubscriptionPath,
 ListString) - Method in class org.apache.beam.sdk.util.PubsubClient
+
+Acknowldege messages from subscription with 
ackIds.
+
+acknowledge(PubsubClient.SubscriptionPath,
 ListString) - Method in class org.apache.beam.sdk.util.PubsubGrpcClient
+
+acknowledge(PubsubClient.SubscriptionPath,
 ListString) - Method in class org.apache.beam.sdk.util.PubsubJsonClient
+
+acknowledge(PubsubClient.SubscriptionPath,
 ListString) - Method in class org.apache.beam.sdk.util.PubsubTestClient
+
+ActiveWindowSetW extends BoundedWindow - Interface 
in org.apache.beam.runners.core
+
+Track which windows are active, and the state 
address window(s) under which their
+ state is stored.
+
+ActiveWindowSet.MergeCallbackW extends BoundedWindow - Interface 
in org.apache.beam.runners.core
+
+Callback for ActiveWindowSet.merge(org.apache.beam.runners.core.ActiveWindowSet.MergeCallbackW).
+
+adaptTagBinder(StateTag.StateBinderK)
 - Static method in class org.apache.beam.sdk.util.state.StateSpecs
+
+Deprecated.
+for migration purposes 
only
+
+
+add(KVbyte[],
 byte[]) - Method in class 
org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter
+
+add(Long)
 - Method in class org.apache.beam.sdk.transforms.ApproximateUnique.ApproximateUniqueCombineFn.LargestUnique
+
+Adds a value to the heap, returning whether the value is 
(large enough
+ to be) in the heap.
+
+add(DisplayData.ItemSpec?)
 - Method in interface org.apache.beam.sdk.transforms.display.DisplayData.Builder
+
+Register the given display item.
+
+add(long, 
long) - Method in class 

[30/52] [abbrv] [partial] beam-site git commit: Regenerate website

[davor]

http://git-wip-us.apache.org/repos/asf/beam-site/blob/7b1f1576/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/OldDoFn.Context.html
--
diff --git 
a/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/OldDoFn.Context.html
 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/OldDoFn.Context.html
new file mode 100644
index 000..73d3879
--- /dev/null
+++ 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/OldDoFn.Context.html
@@ -0,0 +1,506 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+OldDoFn.Context (Apache Beam SDK for Java, version 0.5.0)
+
+
+
+
+
+
+var methods = {"i0":6,"i1":6,"i2":6,"i3":6,"i4":10,"i5":6,"i6":6};
+var tabs = {65535:["t0","All Methods"],2:["t2","Instance 
Methods"],4:["t3","Abstract Methods"],8:["t4","Concrete Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.core
+Class OldDoFn.Context
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">java.lang.Object
+
+
+org.apache.beam.runners.core.OldDoFn.Context
+
+
+
+
+
+
+
+Direct Known Subclasses:
+OldDoFn.ProcessContext
+
+
+Enclosing class:
+OldDoFnInputT,OutputT
+
+
+
+public abstract class OldDoFn.Context
+extends http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">Object
+Information accessible to all methods in this 
OldDoFn.
+ Used primarily to output elements.
+
+
+
+
+
+
+
+
+
+
+
+Constructor Summary
+
+Constructors
+
+Constructor and Description
+
+
+Context()
+
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsInstance MethodsAbstract MethodsConcrete Methods
+
+Modifier and Type
+Method and Description
+
+
+abstract AggInputT,AggOutputTAggregatorAggInputT,AggOutputT
+createAggregatorInternal(http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">Stringname,
+Combine.CombineFnAggInputT,?,AggOutputTcombiner)
+Creates an Aggregator 
in the OldDoFn context 
with the
+ specified name and aggregation logic specified by Combine.CombineFn.
+
+
+
+abstract PipelineOptions
+getPipelineOptions()
+Returns the PipelineOptions specified with the
+ PipelineRunner
+ invoking this OldDoFn.
+
+
+
+abstract void
+output(OutputToutput)
+Adds the given element to the main output 
PCollection.
+
+
+
+abstract void
+outputWithTimestamp(OutputToutput,
+   org.joda.time.Instanttimestamp)
+Adds the given element to the main output 
PCollection,
+ with the given timestamp.
+
+
+
+protected void
+setupDelegateAggregators()
+Sets up Aggregators created by the OldDoFn so they are
+ usable within this context.
+
+
+
+abstract Tvoid
+sideOutput(TupleTagTtag,
+  Toutput)
+Adds the given element to the side output 
PCollection with the
+ given tag.
+
+
+
+abstract Tvoid
+sideOutputWithTimestamp(TupleTagTtag,
+   Toutput,
+   org.joda.time.Instanttimestamp)
+Adds the given element to the specified side output 
PCollection,
+ with the given timestamp.
+
+
+
+
+
+
+
+Methods inherited from classjava.lang.http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">Object
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#clone--;
 title="class or interface in java.lang">clone, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#equals-java.lang.Object-;
 title="class or interface in java.lang">equals, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#finalize--;
 title="class or interface in java.lang">finalize, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#getClass--;
 title="class or interface in java.lang">getClass, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#hashCode--;
 title="class or interface in java.lang">hashCode, 

[27/52] [abbrv] [partial] beam-site git commit: Regenerate website

[davor]

http://git-wip-us.apache.org/repos/asf/beam-site/blob/7b1f1576/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.html
--
diff --git 
a/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.html
 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.html
new file mode 100644
index 000..a3d807d
--- /dev/null
+++ 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/PushbackSideInputDoFnRunner.html
@@ -0,0 +1,368 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+PushbackSideInputDoFnRunner (Apache Beam SDK for Java, version 
0.5.0)
+
+
+
+
+
+
+var methods = {"i0":9,"i1":10,"i2":10,"i3":10,"i4":10,"i5":10};
+var tabs = {65535:["t0","All Methods"],1:["t1","Static 
Methods"],2:["t2","Instance Methods"],8:["t4","Concrete Methods"]};
+var altColor = "altColor";
+var rowColor = "rowColor";
+var tableTab = "tableTab";
+var activeTableTab = "activeTableTab";
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+PrevClass
+NextClass
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+Summary:
+Nested|
+Field|
+Constr|
+Method
+
+
+Detail:
+Field|
+Constr|
+Method
+
+
+
+
+
+
+
+
+org.apache.beam.runners.core
+Class 
PushbackSideInputDoFnRunnerInputT,OutputT
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">java.lang.Object
+
+
+org.apache.beam.runners.core.PushbackSideInputDoFnRunnerInputT,OutputT
+
+
+
+
+
+
+
+All Implemented Interfaces:
+DoFnRunnerInputT,OutputT
+
+
+
+public class PushbackSideInputDoFnRunnerInputT,OutputT
+extends http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">Object
+implements DoFnRunnerInputT,OutputT
+A DoFnRunner 
that can refuse to process elements that are not ready, instead returning
+ them via the processElementInReadyWindows(WindowedValue).
+
+
+
+
+
+
+
+
+
+
+
+Method Summary
+
+All MethodsStatic MethodsInstance MethodsConcrete Methods
+
+Modifier and Type
+Method and Description
+
+
+static InputT,OutputTPushbackSideInputDoFnRunnerInputT,OutputT
+create(DoFnRunnerInputT,OutputTunderlying,
+  http://docs.oracle.com/javase/7/docs/api/java/util/Collection.html?is-external=true;
 title="class or interface in java.util">CollectionPCollectionView?views,
+  ReadyCheckingSideInputReadersideInputReader)
+
+
+void
+finishBundle()
+Call the underlying DoFnRunner.finishBundle().
+
+
+
+void
+onTimer(http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true;
 title="class or interface in java.lang">StringtimerId,
+   BoundedWindowwindow,
+   org.joda.time.Instanttimestamp,
+   TimeDomaintimeDomain)
+Calls a DoFn's @OnTimer 
method for the given timer
+ in the given window.
+
+
+
+void
+processElement(WindowedValueInputTelem)
+Calls a DoFn's 
@ProcessElement method with a
+ DoFn.ProcessContext containing the provided element.
+
+
+
+http://docs.oracle.com/javase/7/docs/api/java/lang/Iterable.html?is-external=true;
 title="class or interface in java.lang">IterableWindowedValueInputT
+processElementInReadyWindows(WindowedValueInputTelem)
+Call the underlying DoFnRunner.processElement(WindowedValue)
 for the provided element
+ for each window the element is in that is ready.
+
+
+
+void
+startBundle()
+Prepares and calls a DoFn's 
@StartBundle method.
+
+
+
+
+
+
+
+Methods inherited from classjava.lang.http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true;
 title="class or interface in java.lang">Object
+http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#clone--;
 title="class or interface in java.lang">clone, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#equals-java.lang.Object-;
 title="class or interface in java.lang">equals, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#finalize--;
 title="class or interface in java.lang">finalize, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#getClass--;
 title="class or interface in java.lang">getClass, http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true#hashCode--;
 title="class or interface in 

[16/52] [abbrv] [partial] beam-site git commit: Regenerate website

[davor]

http://git-wip-us.apache.org/repos/asf/beam-site/blob/7b1f1576/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/class-use/PushbackSideInputDoFnRunner.html
--
diff --git 
a/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/class-use/PushbackSideInputDoFnRunner.html
 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/class-use/PushbackSideInputDoFnRunner.html
new file mode 100644
index 000..1b44e55
--- /dev/null
+++ 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/class-use/PushbackSideInputDoFnRunner.html
@@ -0,0 +1,170 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Class org.apache.beam.runners.core.PushbackSideInputDoFnRunner 
(Apache Beam SDK for Java, version 0.5.0)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+
+Uses of 
Classorg.apache.beam.runners.core.PushbackSideInputDoFnRunner
+
+
+
+
+
+Packages that use PushbackSideInputDoFnRunner
+
+Package
+Description
+
+
+
+org.apache.beam.runners.core
+
+Provides utilities for Beam runner authors.
+
+
+
+
+
+
+
+
+
+
+Uses of PushbackSideInputDoFnRunner 
in org.apache.beam.runners.core
+
+Methods in org.apache.beam.runners.core
 that return PushbackSideInputDoFnRunner
+
+Modifier and Type
+Method and Description
+
+
+
+static InputT,OutputTPushbackSideInputDoFnRunnerInputT,OutputT
+PushbackSideInputDoFnRunner.create(DoFnRunnerInputT,OutputTunderlying,
+  http://docs.oracle.com/javase/7/docs/api/java/util/Collection.html?is-external=true;
 title="class or interface in java.util">CollectionPCollectionView?views,
+  ReadyCheckingSideInputReadersideInputReader)
+
+
+
+
+
+
+
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+Copyright  20162017 https://www.apache.org/;>The Apache Software Foundation. All rights 
reserved.
+
+

http://git-wip-us.apache.org/repos/asf/beam-site/blob/7b1f1576/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/class-use/ReduceFn.Context.html
--
diff --git 
a/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/class-use/ReduceFn.Context.html
 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/class-use/ReduceFn.Context.html
new file mode 100644
index 000..3ff272a
--- /dev/null
+++ 
b/content/documentation/sdks/javadoc/0.5.0/org/apache/beam/runners/core/class-use/ReduceFn.Context.html
@@ -0,0 +1,217 @@
+http://www.w3.org/TR/html4/loose.dtd;>
+
+
+
+
+
+Uses of Class org.apache.beam.runners.core.ReduceFn.Context (Apache 
Beam SDK for Java, version 0.5.0)
+
+
+
+
+
+
+
+
+JavaScript is disabled on your browser.
+
+
+
+
+
+Skip navigation links
+
+
+
+
+Overview
+Package
+Class
+Use
+Tree
+Deprecated
+Index
+Help
+
+
+
+
+Prev
+Next
+
+
+Frames
+NoFrames
+
+
+AllClasses
+
+
+
+
+
+
+
+
+
+
+Uses of 
Classorg.apache.beam.runners.core.ReduceFn.Context
+
+
+
+
+
+Packages that use ReduceFn.Context
+
+Package
+Description
+
+
+
+org.apache.beam.runners.core
+
+Provides utilities for Beam runner authors.
+
+
+
+
+
+
+
+
+
+
+Uses of ReduceFn.Context in org.apache.beam.runners.core
+
+Subclasses of ReduceFn.Context in org.apache.beam.runners.core
+
+Modifier and Type
+Class and Description
+
+
+
+class
+ReduceFn.OnMergeContext
+Information accessible within ReduceFn.onMerge(org.apache.beam.runners.core.ReduceFnK,
 InputT, OutputT, W.OnMergeContext).