date:20171026

Build failed in Jenkins: beam_PerformanceTests_Python #491

2017-10-26 Thread Apache Jenkins Server

See 


Changes:

[klk] Stage the pipeline in Python DataflowRunner

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 1f680307889e03fae3746585c6baf963b2bdf9ba (origin/master)
Commit message: "This closes #4010: [BEAM-3074] Stage the pipeline in Python 
DataflowRunner"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1f680307889e03fae3746585c6baf963b2bdf9ba
 > git rev-list 0c2211375d1fc4abebda7aa7cf7960e7b09fdc1b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5536837657517006361.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2572978783176830323.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7347003087878554748.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #3367

2017-10-26 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init 
 > 
 >  # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 1f680307889e03fae3746585c6baf963b2bdf9ba (origin/master)
java.lang.NoClassDefFoundError: Could not initialize class 
jenkins.model.Jenkins$MasterComputer
at 
org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:71)
at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:896)
at 
hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:870)
at 
hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:829)
at hudson.remoting.UserRequest.perform(UserRequest.java:181)
at hudson.remoting.UserRequest.perform(UserRequest.java:52)
at hudson.remoting.Request$2.run(Request.java:336)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
at ..remote call to beam8(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1554)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:281)
at hudson.remoting.Channel.call(Channel.java:839)
Caused: java.io.IOException: Remote call on beam8 failed
at hudson.remoting.Channel.call(Channel.java:847)
at 
hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
at com.sun.proxy.$Proxy108.withRepository(Unknown Source)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl.withRepository(RemoteGitImpl.java:235)
at hudson.plugins.git.GitSCM.printCommitMessageToLog(GitSCM.java:1195)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1159)
at hudson.scm.SCM.checkout(SCM.java:495)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1212)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:566)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:491)
at hudson.model.Run.execute(Run.java:1737)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:419)
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 1f680307889e03fae3746585c6baf963b2bdf9ba (origin/master)
java.lang.NoClassDefFoundError: Could not initialize class 
jenkins.model.Jenkins$MasterComputer
at 
org.jenkinsci.plugins.gitclient.

[jira] [Commented] (BEAM-2430) Java FnApiDoFnRunner to share across runners

2017-10-26 Thread Kenneth Knowles (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221705#comment-16221705
 ] 

Kenneth Knowles commented on BEAM-2430:
---

I think Thomas's current work on working out the harness dependency issues is 
leading to this.

> Java FnApiDoFnRunner to share across runners
> 
>
> Key: BEAM-2430
> URL: https://issues.apache.org/jira/browse/BEAM-2430
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>  Labels: portability
>
> As the portability framework comes into focus, let's fill out the support 
> code for making it easy to onboard a new runner.
> There is some amount of using the Fn API that has to do only with the fact 
> that a runner is implemented in Java, and is not specific to that runner. 
> This should be part of the runners-core library, and designed so that a 
> runner can set it up however it likes, and just pass elements without having 
> to explicitly manage things like requests, responses, protos, and coders.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2430) Java FnApiDoFnRunner to share across runners

2017-10-26 Thread Kenneth Knowles (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-2430:
-

Assignee: Thomas Groh  (was: Kenneth Knowles)

> Java FnApiDoFnRunner to share across runners
> 
>
> Key: BEAM-2430
> URL: https://issues.apache.org/jira/browse/BEAM-2430
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>  Labels: portability
>
> As the portability framework comes into focus, let's fill out the support 
> code for making it easy to onboard a new runner.
> There is some amount of using the Fn API that has to do only with the fact 
> that a runner is implemented in Java, and is not specific to that runner. 
> This should be part of the runners-core library, and designed so that a 
> runner can set it up however it likes, and just pass elements without having 
> to explicitly manage things like requests, responses, protos, and coders.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2878) Release process for container images

2017-10-26 Thread Kenneth Knowles (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221672#comment-16221672
 ] 

Kenneth Knowles commented on BEAM-2878:
---

We've got a docker repo for releases; I can write up a process for that.

We need a docker repo for release candidates and nightlies. That is INFRA-15382.

> Release process for container images
> 
>
> Key: BEAM-2878
> URL: https://issues.apache.org/jira/browse/BEAM-2878
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Kenneth Knowles
>  Labels: portability
>
> We need to push container images -- notably SDK harness container images -- 
> to a public place with each release, so that they can be referenced and used. 
> For example, https://hub.docker.com/r/apache/ might be an option, although 
> Docker hub seems more geared towards containerizing container-unware content 
> post-release.
> For users who wish to build dev (or custom) container images any location is 
> fine (personal account on docker hub, GCR, etc), but we do not want it to be 
> a requirement for normal use of official Beam releases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3431

2017-10-26 Thread Apache Jenkins Server

See 


--
[...truncated 931.78 KB...]
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apach

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5115

2017-10-26 Thread Apache Jenkins Server

See 


Changes:

[klk] Stage the pipeline in Python DataflowRunner

--
[...truncated 1.36 MB...]
2017-10-27T03:28:14.603 [INFO] Excluding com.twitter:chill-java:jar:0.5.0 from 
the shaded jar.
2017-10-27T03:28:14.603 [INFO] Excluding 
org.apache.xbean:xbean-asm5-shaded:jar:4.4 from the shaded jar.
2017-10-27T03:28:14.603 [INFO] Excluding 
org.apache.hadoop:hadoop-client:jar:2.7.3 from the shaded jar.
2017-10-27T03:28:14.603 [INFO] Excluding 
org.apache.hadoop:hadoop-common:jar:2.7.3 from the shaded jar.
2017-10-27T03:28:14.603 [INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from 
the shaded jar.
2017-10-27T03:28:14.603 [INFO] Excluding 
org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
2017-10-27T03:28:14.603 [INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.directory.api:api-util:jar:1.0.0-M20 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.curator:curator-client:jar:2.7.1 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.3 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.3 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.3 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.3 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-api:jar:2.7.3 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.7.3 from the shaded 
jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.hadoop:hadoop-annotations:jar:2.7.3 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.spark:spark-launcher_2.10:jar:1.6.3 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.spark:spark-network-common_2.10:jar:1.6.3 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.spark:spark-network-shuffle_2.10:jar:1.6.3 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.spark:spark-unsafe_2.10:jar:1.6.3 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding net.java.dev.jets3t:jets3t:jar:0.7.1 
from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
commons-httpclient:commons-httpclient:jar:3.1 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.curator:curator-recipes:jar:2.4.0 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.curator:curator-framework:jar:2.4.0 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016 from the shaded 
jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.apache.commons:commons-math3:jar:3.4.1 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding org.slf4j:jul-to-slf4j:jar:1.7.10 from 
the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.10 
from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded 
jar.
2017-10-27T03:28:14.604 [INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.10 
from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding com.ning:compress-lzf:jar:1.0.3 from 
the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding net.jpountz.lz4:lz4:jar:1.3.0 from the 
shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.roaringbitmap:RoaringBitmap:jar:0.5.11 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding commons-net:commons-net:jar:2.2 from 
the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
com.typesafe.akka:akka-remote_2.10:jar:2.3.11 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
com.typesafe.akka:akka-actor_2.10:jar:2.3.11 from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
org.uncommons.maths:uncommons-maths:jar:1.2.2a from the shaded jar.
2017-10-27T03:28:14.604 [INFO] Excluding 
com.typesafe.akka:akka-slf4j_2.10:jar:2.3.11 from the shaded jar.
2017-10-27T03:28:14.605 [INFO] Excluding 
org.scala-lang:scala-library:jar:2.10.5 from the shaded jar.
2017-10-27T03:28:14.605 [INFO] Excluding 
org.json4s:json4s-jackson_2.10:jar:3.2.10 from the sh

[GitHub] beam pull request #4049: [BEAM-3081] NonNull by default for sdk/transforms a...

2017-10-26 Thread kennknowles

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4049

[BEAM-3081] NonNull by default for sdk/transforms and below

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @jkff 

Continuing the saga of workcrastination / "what I do while other things are 
building"

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam NonNull-up-to-transforms

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4049.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4049






---

[jira] [Commented] (BEAM-3081) Our findbugs config does not actually use Nullable annotations effectively

2017-10-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221656#comment-16221656
 ] 

ASF GitHub Bot commented on BEAM-3081:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4049

[BEAM-3081] NonNull by default for sdk/transforms and below

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @jkff 

Continuing the saga of workcrastination / "what I do while other things are 
building"

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam NonNull-up-to-transforms

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4049.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4049






> Our findbugs config does not actually use Nullable annotations effectively
> --
>
> Key: BEAM-3081
> URL: https://issues.apache.org/jira/browse/BEAM-3081
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> We use {{@Nullable}} annotations mostly appropriately, but in fact our 
> findbugs config was not delivering value based on these annotations, because 
> it does not default to {{@NonNull}}. We can and should set this default.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4048: Miscellaneous touch ups

2017-10-26 Thread kennknowles

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4048

Miscellaneous touch ups

R: @iemejia (randomly chosen victim)

Very trivial PR for warnings that I kept seeing and addressed just because 
they were there.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam misc-touch-ups

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4048.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4048


commit e127c4874ddb1455973afa5867826a9da9224a74
Author: Kenneth Knowles 
Date:   2017-10-26T19:40:18Z

Remove extraneous type arguments in PAssert

commit 50de2938ea91db0f38c6a1595dacb54c66ee71f8
Author: Kenneth Knowles 
Date:   2017-10-26T19:41:06Z

Remove extraneous type arguments in Latest.java

commit 26a3fc55596139f96084ff6310defb0609791d6d
Author: Kenneth Knowles 
Date:   2017-10-26T19:43:08Z

Suppress AutoValue warnings in TextIO




---

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #2683

2017-10-26 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PostCommit_Python_Verify #3430

2017-10-26 Thread Apache Jenkins Server

See 


Changes:

[klk] Stage the pipeline in Python DataflowRunner

--
[...truncated 934.49 KB...]
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/execu

[jira] [Commented] (BEAM-3074) Propagate pipeline protos through Dataflow API from Python

2017-10-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221592#comment-16221592
 ] 

ASF GitHub Bot commented on BEAM-3074:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4010


> Propagate pipeline protos through Dataflow API from Python
> --
>
> Key: BEAM-3074
> URL: https://issues.apache.org/jira/browse/BEAM-3074
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: portability
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-3074) Propagate pipeline protos through Dataflow API from Python

2017-10-26 Thread Kenneth Knowles (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-3074.
---
   Resolution: Fixed
Fix Version/s: 2.3.0

> Propagate pipeline protos through Dataflow API from Python
> --
>
> Key: BEAM-3074
> URL: https://issues.apache.org/jira/browse/BEAM-3074
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: portability
> Fix For: 2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4010: [BEAM-3074] Stage the pipeline in Python DataflowRu...

2017-10-26 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4010


---

[1/2] beam git commit: Stage the pipeline in Python DataflowRunner

2017-10-26 Thread kenn

Repository: beam
Updated Branches:
  refs/heads/master 0c2211375 -> 1f6803078


Stage the pipeline in Python DataflowRunner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7d59c96e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7d59c96e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7d59c96e

Branch: refs/heads/master
Commit: 7d59c96e8031c39fdf43e022291ae25ac50b39e6
Parents: d5aff5d
Author: Kenneth Knowles 
Authored: Wed Oct 18 13:56:28 2017 -0700
Committer: Kenneth Knowles 
Committed: Wed Oct 25 20:17:09 2017 -0700

--
 .../runners/dataflow/dataflow_runner.py |  5 ++-
 .../runners/dataflow/internal/apiclient.py  | 23 +++--
 .../runners/dataflow/internal/apiclient_test.py | 36 ++--
 .../runners/dataflow/internal/names.py  |  2 ++
 .../runners/dataflow/template_runner_test.py|  3 +-
 sdks/python/apache_beam/transforms/core.py  |  3 +-
 6 files changed, 57 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7d59c96e/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index b7d89f1..6253c80 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -272,6 +272,9 @@ class DataflowRunner(PipelineRunner):
   'Google Cloud Dataflow runner not available, '
   'please install apache_beam[gcp]')
 
+# Snapshot the pipeline in a portable proto before mutating it
+proto_pipeline = pipeline.to_runner_api()
+
 # Performing configured PTransform overrides.
 pipeline.replace_all(DataflowRunner._PTRANSFORM_OVERRIDES)
 
@@ -282,7 +285,7 @@ class DataflowRunner(PipelineRunner):
   plugins = list(set(plugins + setup_options.beam_plugins))
 setup_options.beam_plugins = plugins
 
-self.job = apiclient.Job(pipeline._options)
+self.job = apiclient.Job(pipeline._options, proto_pipeline)
 
 # Dataflow runner requires a KV type for GBK inputs, hence we enforce that
 # here.

http://git-wip-us.apache.org/repos/asf/beam/blob/7d59c96e/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py 
b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
index eec598a..d225503 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
@@ -41,6 +41,7 @@ from apache_beam.options.pipeline_options import 
GoogleCloudOptions
 from apache_beam.options.pipeline_options import StandardOptions
 from apache_beam.options.pipeline_options import WorkerOptions
 from apache_beam.runners.dataflow.internal import dependency
+from apache_beam.runners.dataflow.internal import names
 from apache_beam.runners.dataflow.internal.clients import dataflow
 from apache_beam.runners.dataflow.internal.dependency import 
get_sdk_name_and_version
 from apache_beam.runners.dataflow.internal.names import PropertyNames
@@ -118,11 +119,12 @@ class Step(object):
 class Environment(object):
   """Wrapper for a dataflow Environment protobuf."""
 
-  def __init__(self, packages, options, environment_version):
+  def __init__(self, packages, options, environment_version, pipeline_url):
 self.standard_options = options.view_as(StandardOptions)
 self.google_cloud_options = options.view_as(GoogleCloudOptions)
 self.worker_options = options.view_as(WorkerOptions)
 self.debug_options = options.view_as(DebugOptions)
+self.pipeline_url = pipeline_url
 self.proto = dataflow.Environment()
 self.proto.clusterManagerApiService = 
GoogleCloudOptions.COMPUTE_API_SERVICE
 self.proto.dataset = '{}/cloud_dataflow'.format(
@@ -188,10 +190,16 @@ class Environment(object):
 pool = dataflow.WorkerPool(
 kind='local' if self.local else 'harness',
 packages=package_descriptors,
+metadata=dataflow.WorkerPool.MetadataValue(),
 taskrunnerSettings=dataflow.TaskRunnerSettings(
 parallelWorkerSettings=dataflow.WorkerSettings(
 baseUrl=GoogleCloudOptions.DATAFLOW_ENDPOINT,
 servicePath=self.google_cloud_options.dataflow_endpoint)))
+
+pool.metadata.additionalProperties.append(
+dataflow.WorkerPool.MetadataValue.AdditionalProperty(
+key=names.STAGED_PIPELINE_URL_METADATA_FIELD, value=pipeline_url))
+
 pool.autoscalingSettings = dataflow.AutoscalingSettings()
 # Set worker pool o

[2/2] beam git commit: This closes #4010: [BEAM-3074] Stage the pipeline in Python DataflowRunner

2017-10-26 Thread kenn

This closes #4010: [BEAM-3074] Stage the pipeline in Python DataflowRunner

  Stage the pipeline in Python DataflowRunner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1f680307
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1f680307
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1f680307

Branch: refs/heads/master
Commit: 1f680307889e03fae3746585c6baf963b2bdf9ba
Parents: 0c22113 7d59c96
Author: Kenneth Knowles 
Authored: Thu Oct 26 18:43:14 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Oct 26 18:43:14 2017 -0700

--
 .../runners/dataflow/dataflow_runner.py |  5 ++-
 .../runners/dataflow/internal/apiclient.py  | 23 +++--
 .../runners/dataflow/internal/apiclient_test.py | 36 ++--
 .../runners/dataflow/internal/names.py  |  2 ++
 .../runners/dataflow/template_runner_test.py|  3 +-
 sdks/python/apache_beam/transforms/core.py  |  3 +-
 6 files changed, 57 insertions(+), 15 deletions(-)
--

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5114

2017-10-26 Thread Apache Jenkins Server

See 


--
[...truncated 1.36 MB...]
2017-10-27T01:10:25.626 [INFO] Excluding com.twitter:chill_2.10:jar:0.5.0 from 
the shaded jar.
2017-10-27T01:10:25.626 [INFO] Excluding com.twitter:chill-java:jar:0.5.0 from 
the shaded jar.
2017-10-27T01:10:25.627 [INFO] Excluding 
org.apache.xbean:xbean-asm5-shaded:jar:4.4 from the shaded jar.
2017-10-27T01:10:25.627 [INFO] Excluding 
org.apache.hadoop:hadoop-client:jar:2.7.3 from the shaded jar.
2017-10-27T01:10:25.627 [INFO] Excluding 
org.apache.hadoop:hadoop-common:jar:2.7.3 from the shaded jar.
2017-10-27T01:10:25.627 [INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from 
the shaded jar.
2017-10-27T01:10:25.627 [INFO] Excluding 
org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
2017-10-27T01:10:25.627 [INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
2017-10-27T01:10:25.627 [INFO] Excluding 
org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from the shaded jar.
2017-10-27T01:10:25.627 [INFO] Excluding 
org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the shaded jar.
2017-10-27T01:10:25.628 [INFO] Excluding 
org.apache.directory.api:api-util:jar:1.0.0-M20 from the shaded jar.
2017-10-27T01:10:25.628 [INFO] Excluding 
org.apache.curator:curator-client:jar:2.7.1 from the shaded jar.
2017-10-27T01:10:25.628 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.3 from the shaded jar.
2017-10-27T01:10:25.628 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.3 from the shaded jar.
2017-10-27T01:10:25.628 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.3 from the shaded jar.
2017-10-27T01:10:25.628 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.3 from the shaded jar.
2017-10-27T01:10:25.628 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-api:jar:2.7.3 from the shaded jar.
2017-10-27T01:10:25.628 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from the shaded jar.
2017-10-27T01:10:25.628 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.7.3 from the shaded 
jar.
2017-10-27T01:10:25.629 [INFO] Excluding 
org.apache.hadoop:hadoop-annotations:jar:2.7.3 from the shaded jar.
2017-10-27T01:10:25.629 [INFO] Excluding 
org.apache.spark:spark-launcher_2.10:jar:1.6.3 from the shaded jar.
2017-10-27T01:10:25.629 [INFO] Excluding 
org.apache.spark:spark-network-common_2.10:jar:1.6.3 from the shaded jar.
2017-10-27T01:10:25.629 [INFO] Excluding 
org.apache.spark:spark-network-shuffle_2.10:jar:1.6.3 from the shaded jar.
2017-10-27T01:10:25.629 [INFO] Excluding 
org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the shaded jar.
2017-10-27T01:10:25.629 [INFO] Excluding 
org.apache.spark:spark-unsafe_2.10:jar:1.6.3 from the shaded jar.
2017-10-27T01:10:25.629 [INFO] Excluding net.java.dev.jets3t:jets3t:jar:0.7.1 
from the shaded jar.
2017-10-27T01:10:25.629 [INFO] Excluding 
commons-httpclient:commons-httpclient:jar:3.1 from the shaded jar.
2017-10-27T01:10:25.629 [INFO] Excluding 
org.apache.curator:curator-recipes:jar:2.4.0 from the shaded jar.
2017-10-27T01:10:25.630 [INFO] Excluding 
org.apache.curator:curator-framework:jar:2.4.0 from the shaded jar.
2017-10-27T01:10:25.630 [INFO] Excluding 
org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016 from the shaded 
jar.
2017-10-27T01:10:25.630 [INFO] Excluding 
org.apache.commons:commons-math3:jar:3.4.1 from the shaded jar.
2017-10-27T01:10:25.630 [INFO] Excluding org.slf4j:jul-to-slf4j:jar:1.7.10 from 
the shaded jar.
2017-10-27T01:10:25.630 [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.10 
from the shaded jar.
2017-10-27T01:10:25.630 [INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded 
jar.
2017-10-27T01:10:25.630 [INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.10 
from the shaded jar.
2017-10-27T01:10:25.630 [INFO] Excluding com.ning:compress-lzf:jar:1.0.3 from 
the shaded jar.
2017-10-27T01:10:25.630 [INFO] Excluding net.jpountz.lz4:lz4:jar:1.3.0 from the 
shaded jar.
2017-10-27T01:10:25.631 [INFO] Excluding 
org.roaringbitmap:RoaringBitmap:jar:0.5.11 from the shaded jar.
2017-10-27T01:10:25.631 [INFO] Excluding commons-net:commons-net:jar:2.2 from 
the shaded jar.
2017-10-27T01:10:25.631 [INFO] Excluding 
com.typesafe.akka:akka-remote_2.10:jar:2.3.11 from the shaded jar.
2017-10-27T01:10:25.631 [INFO] Excluding 
com.typesafe.akka:akka-actor_2.10:jar:2.3.11 from the shaded jar.
2017-10-27T01:10:25.631 [INFO] Excluding 
org.uncommons.maths:uncommons-maths:jar:1.2.2a from the shaded jar.
2017-10-27T01:10:25.631 [INFO] Excluding 
com.typesafe.akka:akka-slf4j_2.10:jar:2.3.11 from the shaded jar.
2017-10-27T01:10:25.631 [INFO] Excluding 
org.scala-lang:scala-library:jar:2.10.5 from the shaded jar.
2017-10-27T01:10:25.631 [INFO] Excluding 
org.json4s:json4s-jackson_2.10:

[beam] Git Push Summary

2017-10-26 Thread reuvenlax

Repository: beam
Updated Tags:  refs/tags/v2.2.0-RC1 [created] 4f90bd3b0

beam git commit: [maven-release-plugin] rollback changes from release preparation of v2.2.0-RC1

2017-10-26 Thread reuvenlax

Repository: beam
Updated Branches:
  refs/heads/release-2.2.0 9b709e794 -> 576d22a67


[maven-release-plugin] rollback changes from release preparation of v2.2.0-RC1


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/576d22a6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/576d22a6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/576d22a6

Branch: refs/heads/release-2.2.0
Commit: 576d22a67ffeb29adf26f8adc09aaa6078099cf6
Parents: 9b709e7
Author: Reuven Lax 
Authored: Thu Oct 26 17:36:09 2017 -0700
Committer: Reuven Lax 
Committed: Thu Oct 26 17:36:09 2017 -0700

--
 examples/java/pom.xml| 2 +-
 examples/java8/pom.xml   | 2 +-
 examples/pom.xml | 2 +-
 pom.xml  | 4 ++--
 runners/apex/pom.xml | 2 +-
 runners/core-construction-java/pom.xml   | 2 +-
 runners/core-java/pom.xml| 2 +-
 runners/direct-java/pom.xml  | 2 +-
 runners/flink/pom.xml| 2 +-
 runners/gcp/gcemd/pom.xml| 2 +-
 runners/gcp/gcsproxy/pom.xml | 2 +-
 runners/gcp/pom.xml  | 2 +-
 runners/gearpump/pom.xml | 2 +-
 runners/google-cloud-dataflow-java/pom.xml   | 2 +-
 runners/local-artifact-service-java/pom.xml  | 2 +-
 runners/pom.xml  | 2 +-
 runners/reference/pom.xml| 2 +-
 runners/spark/pom.xml| 2 +-
 sdks/common/fn-api/pom.xml   | 2 +-
 sdks/common/pom.xml  | 2 +-
 sdks/common/runner-api/pom.xml   | 2 +-
 sdks/go/pom.xml  | 2 +-
 sdks/java/build-tools/pom.xml| 2 +-
 sdks/java/container/pom.xml  | 2 +-
 sdks/java/core/pom.xml   | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml  | 2 +-
 sdks/java/extensions/jackson/pom.xml | 2 +-
 sdks/java/extensions/join-library/pom.xml| 2 +-
 sdks/java/extensions/pom.xml | 2 +-
 sdks/java/extensions/protobuf/pom.xml| 2 +-
 sdks/java/extensions/sorter/pom.xml  | 2 +-
 sdks/java/extensions/sql/pom.xml | 2 +-
 sdks/java/harness/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml| 2 +-
 sdks/java/io/cassandra/pom.xml   | 2 +-
 sdks/java/io/common/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml   | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml   | 2 +-
 .../io/elasticsearch-tests/elasticsearch-tests-common/pom.xml| 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml   | 2 +-
 sdks/java/io/hadoop-common/pom.xml   | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml  | 2 +-
 sdks/java/io/hadoop/input-format/pom.xml | 2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml | 2 +-
 sdks/java/io/hadoop/pom.xml  | 2 +-
 sdks/java/io/hbase/pom.xml   | 2 +-
 sdks/java/io/hcatalog/pom.xml| 2 +-
 sdks/java/io/jdbc/pom.xml| 2 +-
 sdks/java/io/jms/pom.xml | 2 +-
 sdks/java/io/kafka/pom.xml   | 2 +-
 sdks/java/io/kinesis/pom.xml | 2 +-
 sdks/java/io/mongodb/pom.xml | 2 +-
 sdks/java/io/mqtt/pom.xml| 2 +-
 sdks/java/io/pom.xml | 2 +-
 sdks/java/io/redis/pom.xml   | 2 +-
 sdks/java/io/solr/pom.xml| 2 +-
 sdks/java/io/tika/pom.xml

beam git commit: [maven-release-plugin] prepare release v2.2.0-RC1

2017-10-26 Thread reuvenlax

Repository: beam
Updated Branches:
  refs/heads/release-2.2.0 e47b477b9 -> 9b709e794


[maven-release-plugin] prepare release v2.2.0-RC1


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9b709e79
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9b709e79
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9b709e79

Branch: refs/heads/release-2.2.0
Commit: 9b709e794f77fe44ba2884d78fcf243c55d1c88a
Parents: e47b477
Author: Reuven Lax 
Authored: Thu Oct 26 17:32:28 2017 -0700
Committer: Reuven Lax 
Committed: Thu Oct 26 17:32:28 2017 -0700

--
 examples/java/pom.xml| 2 +-
 examples/java8/pom.xml   | 2 +-
 examples/pom.xml | 2 +-
 pom.xml  | 4 ++--
 runners/apex/pom.xml | 2 +-
 runners/core-construction-java/pom.xml   | 2 +-
 runners/core-java/pom.xml| 2 +-
 runners/direct-java/pom.xml  | 2 +-
 runners/flink/pom.xml| 2 +-
 runners/gcp/gcemd/pom.xml| 2 +-
 runners/gcp/gcsproxy/pom.xml | 2 +-
 runners/gcp/pom.xml  | 2 +-
 runners/gearpump/pom.xml | 2 +-
 runners/google-cloud-dataflow-java/pom.xml   | 2 +-
 runners/local-artifact-service-java/pom.xml  | 2 +-
 runners/pom.xml  | 2 +-
 runners/reference/pom.xml| 2 +-
 runners/spark/pom.xml| 2 +-
 sdks/common/fn-api/pom.xml   | 2 +-
 sdks/common/pom.xml  | 2 +-
 sdks/common/runner-api/pom.xml   | 2 +-
 sdks/go/pom.xml  | 2 +-
 sdks/java/build-tools/pom.xml| 2 +-
 sdks/java/container/pom.xml  | 2 +-
 sdks/java/core/pom.xml   | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml  | 2 +-
 sdks/java/extensions/jackson/pom.xml | 2 +-
 sdks/java/extensions/join-library/pom.xml| 2 +-
 sdks/java/extensions/pom.xml | 2 +-
 sdks/java/extensions/protobuf/pom.xml| 2 +-
 sdks/java/extensions/sorter/pom.xml  | 2 +-
 sdks/java/extensions/sql/pom.xml | 2 +-
 sdks/java/harness/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml| 2 +-
 sdks/java/io/cassandra/pom.xml   | 2 +-
 sdks/java/io/common/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml   | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml   | 2 +-
 .../io/elasticsearch-tests/elasticsearch-tests-common/pom.xml| 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml   | 2 +-
 sdks/java/io/hadoop-common/pom.xml   | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml  | 2 +-
 sdks/java/io/hadoop/input-format/pom.xml | 2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml | 2 +-
 sdks/java/io/hadoop/pom.xml  | 2 +-
 sdks/java/io/hbase/pom.xml   | 2 +-
 sdks/java/io/hcatalog/pom.xml| 2 +-
 sdks/java/io/jdbc/pom.xml| 2 +-
 sdks/java/io/jms/pom.xml | 2 +-
 sdks/java/io/kafka/pom.xml   | 2 +-
 sdks/java/io/kinesis/pom.xml | 2 +-
 sdks/java/io/mongodb/pom.xml | 2 +-
 sdks/java/io/mqtt/pom.xml| 2 +-
 sdks/java/io/pom.xml | 2 +-
 sdks/java/io/redis/pom.xml   | 2 +-
 sdks/java/io/solr/pom.xml| 2 +-
 sdks/java/io/tika/pom.xml| 2 +-

Build failed in Jenkins: beam_PerformanceTests_Python #490

2017-10-26 Thread Apache Jenkins Server

See 


Changes:

[lcwik] [BEAM-2566] Decouple SDK harness from Dataflow runner by elevating

[lcwik] Add missing @RunWith to test.

[kirpichov] Many improvements to TikaIO

[kirpichov] Adds ParseResult.failure()

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0c2211375d1fc4abebda7aa7cf7960e7b09fdc1b (origin/master)
Commit message: "This closes #3835: [BEAM-2994] Many improvements to TikaIO"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0c2211375d1fc4abebda7aa7cf7960e7b09fdc1b
 > git rev-list cde3b7b2d64a93d34c31c0959e7a1b77a0f0d807 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1544325615304898008.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1897812347755321388.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins678612637251450012.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8624265307172044844.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: grpc

[jira] [Commented] (BEAM-3083) BigQueryIO.write() with DynamicDestinations should not call getSchema() on every element

2017-10-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221495#comment-16221495
 ] 

ASF GitHub Bot commented on BEAM-3083:
--

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/4047

Pushes format function to lowest level in BigQueryIO

This likely shuffles a lot fewer data, and also will at some point allow 
using a format function that formats to an Avro GenericRecord, rather than to 
the inefficient JSON TableRow. (though that will be usable only with 
BATCH_LOADS method, but that's fine)

Also makes sure that getSchema() is called once per destination rather than 
once per element (https://issues.apache.org/jira/browse/BEAM-3083).

R: @steveniemitz 
CC: @reuvenlax 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam bq-format-fn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4047.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4047


commit 8b34098e351a22b919aecff8953a634708d3a978
Author: Eugene Kirpichov 
Date:   2017-10-26T23:17:42Z

Pushes format function to lowest level in BigQueryIO

This likely shuffles a lot fewer data, and also will at some point allow
using a format function that formats to an Avro GenericRecord, rather
than to the inefficient JSON TableRow.

Also makes sure that getSchema() is called once per destination
rather than once per element.




> BigQueryIO.write() with DynamicDestinations should not call getSchema() on 
> every element
> 
>
> Key: BEAM-3083
> URL: https://issues.apache.org/jira/browse/BEAM-3083
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Eugene Kirpichov
>Assignee: Reuven Lax
>
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/CreateTables.java#L103
>  calls getSchema() on every element. This might be very expensive if the user 
> expects that this will be called about once per destination.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4047: Pushes format function to lowest level in BigQueryI...

2017-10-26 Thread jkff

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/4047

Pushes format function to lowest level in BigQueryIO

This likely shuffles a lot fewer data, and also will at some point allow 
using a format function that formats to an Avro GenericRecord, rather than to 
the inefficient JSON TableRow. (though that will be usable only with 
BATCH_LOADS method, but that's fine)

Also makes sure that getSchema() is called once per destination rather than 
once per element (https://issues.apache.org/jira/browse/BEAM-3083).

R: @steveniemitz 
CC: @reuvenlax 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam bq-format-fn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4047.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4047


commit 8b34098e351a22b919aecff8953a634708d3a978
Author: Eugene Kirpichov 
Date:   2017-10-26T23:17:42Z

Pushes format function to lowest level in BigQueryIO

This likely shuffles a lot fewer data, and also will at some point allow
using a format function that formats to an Avro GenericRecord, rather
than to the inefficient JSON TableRow.

Also makes sure that getSchema() is called once per destination
rather than once per element.




---

[beam-site] 03/03: This closes #336

2017-10-26 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 0cf69dd7a039e7b76e6a4cf901e27d525119bf09
Merge: aa074e7 961cc7c
Author: Ahmet Altay 
AuthorDate: Thu Oct 26 16:58:45 2017 -0700

This closes #336

 content/documentation/io/built-in/index.html |  6 +---
 content/get-started/wordcount-example/index.html | 32 +
 src/get-started/wordcount-example.md | 36 
 3 files changed, 2 insertions(+), 72 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[jira] [Commented] (BEAM-664) Port Dataflow SDK WordCount walkthrough to Beam site

2017-10-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221485#comment-16221485
 ] 

ASF GitHub Bot commented on BEAM-664:
-

asfgit closed pull request #336: [BEAM-664] Update Java MinimalWordCount 
instructions
URL: https://github.com/apache/beam-site/pull/336
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/get-started/wordcount-example.md 
b/src/get-started/wordcount-example.md
index c40a2c706..8b379377a 100644
--- a/src/get-started/wordcount-example.md
+++ b/src/get-started/wordcount-example.md
@@ -50,46 +50,10 @@ input and output sources and show other best practices.
 
 **To run this example in Java:**
 
-{:.runner-direct}
 ```
 $ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount
 ```
 
-{:.runner-apex}
-```
-$ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
- -Dexec.args="--inputFile=pom.xml --output=counts --runner=ApexRunner" 
-Papex-runner
-```
-
-{:.runner-flink-local}
-```
-$ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
- -Dexec.args="--runner=FlinkRunner --inputFile=pom.xml --output=counts" 
-Pflink-runner
-```
-
-{:.runner-flink-cluster}
-```
-$ mvn package exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
- -Dexec.args="--runner=FlinkRunner --flinkMaster= 
--filesToStage=target/word-count-beam-bundled-0.1.jar \
-  --inputFile=/path/to/quickstart/pom.xml 
--output=/tmp/counts" -Pflink-runner
-
-You can monitor the running job by visiting the Flink dashboard at 
http://:8081
-```
-
-{:.runner-spark}
-```
-$ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
- -Dexec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" 
-Pspark-runner
-```
-
-{:.runner-dataflow}
-```
-$ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
-   -Dexec.args="--runner=DataflowRunner 
--gcpTempLocation=gs:///tmp \
---inputFile=gs://apache-beam-samples/shakespeare/* 
--output=gs:///counts" \
- -Pdataflow-runner
-```
-
 To view the full code in Java, see
 
**[MinimalWordCount](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java).**
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Port Dataflow SDK WordCount walkthrough to Beam site
> 
>
> Key: BEAM-664
> URL: https://issues.apache.org/jira/browse/BEAM-664
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
> Fix For: Not applicable
>
>
> Port the WordCount walkthrough from Dataflow docs to Beam website. 
> * Copy prose (translate from html to md, remove Dataflow references, etc)
> * Add accurate "How to Run" instructions for each of the WC examples
> * Include code snippets from real examples



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[beam-site] 01/03: [BEAM-664] Update docs: MinimalWordCount in Java is intentionally hardcoded to run only on DirectRunner

2017-10-26 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit fa66528c45235d3ef73566e46ac8baa98cda8dae
Author: Alex Filatov 
AuthorDate: Thu Oct 19 17:44:18 2017 +0300

[BEAM-664] Update docs: MinimalWordCount in Java is intentionally hardcoded 
to run only on DirectRunner
---
 src/get-started/wordcount-example.md | 36 
 1 file changed, 36 deletions(-)

diff --git a/src/get-started/wordcount-example.md 
b/src/get-started/wordcount-example.md
index c40a2c7..8b37937 100644
--- a/src/get-started/wordcount-example.md
+++ b/src/get-started/wordcount-example.md
@@ -50,46 +50,10 @@ input and output sources and show other best practices.
 
 **To run this example in Java:**
 
-{:.runner-direct}
 ```
 $ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount
 ```
 
-{:.runner-apex}
-```
-$ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
- -Dexec.args="--inputFile=pom.xml --output=counts --runner=ApexRunner" 
-Papex-runner
-```
-
-{:.runner-flink-local}
-```
-$ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
- -Dexec.args="--runner=FlinkRunner --inputFile=pom.xml --output=counts" 
-Pflink-runner
-```
-
-{:.runner-flink-cluster}
-```
-$ mvn package exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
- -Dexec.args="--runner=FlinkRunner --flinkMaster= 
--filesToStage=target/word-count-beam-bundled-0.1.jar \
-  --inputFile=/path/to/quickstart/pom.xml 
--output=/tmp/counts" -Pflink-runner
-
-You can monitor the running job by visiting the Flink dashboard at 
http://:8081
-```
-
-{:.runner-spark}
-```
-$ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
- -Dexec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" 
-Pspark-runner
-```
-
-{:.runner-dataflow}
-```
-$ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
-   -Dexec.args="--runner=DataflowRunner 
--gcpTempLocation=gs:///tmp \
---inputFile=gs://apache-beam-samples/shakespeare/* 
--output=gs:///counts" \
- -Pdataflow-runner
-```
-
 To view the full code in Java, see
 
**[MinimalWordCount](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java).**
 

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 02/03: Regenerate website

2017-10-26 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 961cc7c50d4bb02735ec5a1c1567e0d5a10c0bec
Author: Ahmet Altay 
AuthorDate: Thu Oct 26 16:58:45 2017 -0700

Regenerate website
---
 content/documentation/io/built-in/index.html |  6 +
 content/get-started/wordcount-example/index.html | 32 +---
 2 files changed, 2 insertions(+), 36 deletions(-)

diff --git a/content/documentation/io/built-in/index.html 
b/content/documentation/io/built-in/index.html
index 5bd86c4..312e09c 100644
--- a/content/documentation/io/built-in/index.html
+++ b/content/documentation/io/built-in/index.html
@@ -182,7 +182,7 @@
 https://github.com/apache/beam/tree/master/sdks/java/io/hbase";>Apache 
HBase
 https://github.com/apache/beam/tree/master/sdks/java/io/hcatalog";>Apache 
Hive (HCatalog)
 https://github.com/apache/beam/tree/master/sdks/java/io/solr";>Apache 
Solr
-https://github.com/apache/beam/tree/master/sdks/java/io/elasticsearch";>Elasticsearch
 (v2.x)
+https://github.com/apache/beam/tree/master/sdks/java/io/elasticsearch";>Elasticsearch
 (v2.x and v5.x)
 https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery";>Google
 BigQuery
 https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable";>Google
 Cloud Bigtable
 https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore";>Google
 Cloud Datastore
@@ -237,10 +237,6 @@
 https://issues.apache.org/jira/browse/BEAM-1893";>BEAM-1893
   
   
-Elasticsearch (v5.x)Java
-https://issues.apache.org/jira/browse/BEAM-1637";>BEAM-1637
-  
-  
 Google Cloud SpannerJava
 https://issues.apache.org/jira/browse/BEAM-1542";>BEAM-1542
   
diff --git a/content/get-started/wordcount-example/index.html 
b/content/get-started/wordcount-example/index.html
index 0bc72e3..76148be 100644
--- a/content/get-started/wordcount-example/index.html
+++ b/content/get-started/wordcount-example/index.html
@@ -223,37 +223,7 @@ input and output sources and show other best practices.
 
 To run this example in Java:
 
-$ 
mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.MinimalWordCount
-
-
-
-$ mvn 
compile exec:java -Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
- -Dexec.args="--inputFile=pom.xml --output=counts --runner=ApexRunner" 
-Papex-runner
-
-
-
-$ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
- -Dexec.args="--runner=FlinkRunner --inputFile=pom.xml --output=counts" 
-Pflink-runner
-
-
-
-$ mvn package exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
- -Dexec.args="--runner=FlinkRunner --flinkMaster= 
--filesToStage=target/word-count-beam-bundled-0.1.jar \
-  --inputFile=/path/to/quickstart/pom.xml 
--output=/tmp/counts" -Pflink-runner
-
-You can monitor the running job by visiting the Flink dashboard at 
http://:8081
-
-
-
-$ mvn 
compile exec:java -Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
- -Dexec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" 
-Pspark-runner
-
-
-
-$ 
mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.MinimalWordCount \
-   -Dexec.args="--runner=DataflowRunner 
--gcpTempLocation=gs:///tmp \
---inputFile=gs://apache-beam-samples/shakespeare/* 
--output=gs:///counts" \
- -Pdataflow-runner
+$ mvn compile 
exec:java -Dexec.mainClass=org.apache.beam.examples.MinimalWordCount
 
 
 

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch asf-site updated (aa074e7 -> 0cf69dd)

2017-10-26 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from aa074e7  This closes #322
 new fa66528  [BEAM-664] Update docs: MinimalWordCount in Java is 
intentionally hardcoded to run only on DirectRunner
 new 961cc7c  Regenerate website
 new 0cf69dd  This closes #336

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/io/built-in/index.html |  6 +---
 content/get-started/wordcount-example/index.html | 32 +
 src/get-started/wordcount-example.md | 36 
 3 files changed, 2 insertions(+), 72 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[2/2] beam git commit: Revert "[maven-release-plugin] prepare release v-RC1-DupdateWorkingCopyVersions=false"

2017-10-26 Thread reuvenlax

Revert "[maven-release-plugin] prepare release 
v-RC1-DupdateWorkingCopyVersions=false"

This reverts commit 2c5f85a8e262f6d55fcc06faead69b38c349b42c.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e47b477b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e47b477b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e47b477b

Branch: refs/heads/release-2.2.0
Commit: e47b477b983e318f0cc16356ce8b9086e2af5191
Parents: b6a3528
Author: Reuven Lax 
Authored: Thu Oct 26 17:03:52 2017 -0700
Committer: Reuven Lax 
Committed: Thu Oct 26 17:03:52 2017 -0700

--
 examples/java/pom.xml| 2 +-
 examples/java8/pom.xml   | 2 +-
 examples/pom.xml | 2 +-
 pom.xml  | 4 ++--
 runners/apex/pom.xml | 2 +-
 runners/core-construction-java/pom.xml   | 2 +-
 runners/core-java/pom.xml| 2 +-
 runners/direct-java/pom.xml  | 2 +-
 runners/flink/pom.xml| 2 +-
 runners/gcp/gcemd/pom.xml| 2 +-
 runners/gcp/gcsproxy/pom.xml | 2 +-
 runners/gcp/pom.xml  | 2 +-
 runners/gearpump/pom.xml | 2 +-
 runners/google-cloud-dataflow-java/pom.xml   | 2 +-
 runners/local-artifact-service-java/pom.xml  | 2 +-
 runners/pom.xml  | 2 +-
 runners/reference/pom.xml| 2 +-
 runners/spark/pom.xml| 2 +-
 sdks/common/fn-api/pom.xml   | 2 +-
 sdks/common/pom.xml  | 2 +-
 sdks/common/runner-api/pom.xml   | 2 +-
 sdks/go/pom.xml  | 2 +-
 sdks/java/build-tools/pom.xml| 2 +-
 sdks/java/container/pom.xml  | 2 +-
 sdks/java/core/pom.xml   | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml  | 2 +-
 sdks/java/extensions/jackson/pom.xml | 2 +-
 sdks/java/extensions/join-library/pom.xml| 2 +-
 sdks/java/extensions/pom.xml | 2 +-
 sdks/java/extensions/protobuf/pom.xml| 2 +-
 sdks/java/extensions/sorter/pom.xml  | 2 +-
 sdks/java/extensions/sql/pom.xml | 2 +-
 sdks/java/harness/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml| 2 +-
 sdks/java/io/cassandra/pom.xml   | 2 +-
 sdks/java/io/common/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml   | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml   | 2 +-
 .../io/elasticsearch-tests/elasticsearch-tests-common/pom.xml| 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml   | 2 +-
 sdks/java/io/hadoop-common/pom.xml   | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml  | 2 +-
 sdks/java/io/hadoop/input-format/pom.xml | 2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml | 2 +-
 sdks/java/io/hadoop/pom.xml  | 2 +-
 sdks/java/io/hbase/pom.xml   | 2 +-
 sdks/java/io/hcatalog/pom.xml| 2 +-
 sdks/java/io/jdbc/pom.xml| 2 +-
 sdks/java/io/jms/pom.xml | 2 +-
 sdks/java/io/kafka/pom.xml   | 2 +-
 sdks/java/io/kinesis/pom.xml | 2 +-
 sdks/java/io/mongodb/pom.xml | 2 +-
 sdks/java/io/mqtt/pom.xml| 2 +-
 sdks/java/io/pom.xml | 2 +-
 sdks/java/io/redis/pom.xml   | 2 +-
 sdks/java/io/solr/pom.xml| 2 +-
 sdks/java/io/tika/pom.xml

[1/2] beam git commit: Revert "[maven-release-plugin] prepare for next development iteration"

2017-10-26 Thread reuvenlax

Repository: beam
Updated Branches:
  refs/heads/release-2.2.0 b6d15dc98 -> e47b477b9


Revert "[maven-release-plugin] prepare for next development iteration"

This reverts commit b6d15dc98f7dc13ca0cb013980b8239149fed6c8.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b6a35282
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b6a35282
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b6a35282

Branch: refs/heads/release-2.2.0
Commit: b6a35282f531e7148c609cb28a16572680e0daee
Parents: b6d15dc
Author: Reuven Lax 
Authored: Thu Oct 26 17:03:31 2017 -0700
Committer: Reuven Lax 
Committed: Thu Oct 26 17:03:31 2017 -0700

--
 examples/java/pom.xml| 2 +-
 examples/java8/pom.xml   | 2 +-
 examples/pom.xml | 2 +-
 pom.xml  | 4 ++--
 runners/apex/pom.xml | 2 +-
 runners/core-construction-java/pom.xml   | 2 +-
 runners/core-java/pom.xml| 2 +-
 runners/direct-java/pom.xml  | 2 +-
 runners/flink/pom.xml| 2 +-
 runners/gcp/gcemd/pom.xml| 2 +-
 runners/gcp/gcsproxy/pom.xml | 2 +-
 runners/gcp/pom.xml  | 2 +-
 runners/gearpump/pom.xml | 2 +-
 runners/google-cloud-dataflow-java/pom.xml   | 2 +-
 runners/local-artifact-service-java/pom.xml  | 2 +-
 runners/pom.xml  | 2 +-
 runners/reference/pom.xml| 2 +-
 runners/spark/pom.xml| 2 +-
 sdks/common/fn-api/pom.xml   | 2 +-
 sdks/common/pom.xml  | 2 +-
 sdks/common/runner-api/pom.xml   | 2 +-
 sdks/go/pom.xml  | 2 +-
 sdks/java/build-tools/pom.xml| 2 +-
 sdks/java/container/pom.xml  | 2 +-
 sdks/java/core/pom.xml   | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml  | 2 +-
 sdks/java/extensions/jackson/pom.xml | 2 +-
 sdks/java/extensions/join-library/pom.xml| 2 +-
 sdks/java/extensions/pom.xml | 2 +-
 sdks/java/extensions/protobuf/pom.xml| 2 +-
 sdks/java/extensions/sorter/pom.xml  | 2 +-
 sdks/java/extensions/sql/pom.xml | 2 +-
 sdks/java/harness/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml| 2 +-
 sdks/java/io/cassandra/pom.xml   | 2 +-
 sdks/java/io/common/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml   | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml   | 2 +-
 .../io/elasticsearch-tests/elasticsearch-tests-common/pom.xml| 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml   | 2 +-
 sdks/java/io/hadoop-common/pom.xml   | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml  | 2 +-
 sdks/java/io/hadoop/input-format/pom.xml | 2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml | 2 +-
 sdks/java/io/hadoop/pom.xml  | 2 +-
 sdks/java/io/hbase/pom.xml   | 2 +-
 sdks/java/io/hcatalog/pom.xml| 2 +-
 sdks/java/io/jdbc/pom.xml| 2 +-
 sdks/java/io/jms/pom.xml | 2 +-
 sdks/java/io/kafka/pom.xml   | 2 +-
 sdks/java/io/kinesis/pom.xml | 2 +-
 sdks/java/io/mongodb/pom.xml | 2 +-
 sdks/java/io/mqtt/pom.xml| 2 +-
 sdks/java/io/pom.xml | 2 +-
 sdks/java/io/redis/pom.xml   | 2 +-
 sdks/java/io/solr/pom.xml

[GitHub] beam pull request #4046: Reset Jenkins config

2017-10-26 Thread lukecwik

Github user lukecwik closed the pull request at:

https://github.com/apache/beam/pull/4046


---

[GitHub] beam pull request #4046: Reset Jenkins config

2017-10-26 Thread lukecwik

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/4046

Reset Jenkins config

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam jenkins2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4046.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4046


commit b11b30bd36d028f3cf659b0cb33bdc43eabba4f3
Author: Luke Cwik 
Date:   2017-10-26T23:51:29Z

Reset Jenkins config




---

[jira] [Assigned] (BEAM-3099) Implement HDFS FileSystem for Python SDK

2017-10-26 Thread Chamikara Jayalath (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath reassigned BEAM-3099:


Assignee: Udi Meiri  (was: Chamikara Jayalath)

> Implement HDFS FileSystem for Python SDK
> 
>
> Key: BEAM-3099
> URL: https://issues.apache.org/jira/browse/BEAM-3099
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chamikara Jayalath
>Assignee: Udi Meiri
>
> Currently Java SDK has HDFS support but Python SDK does not. With current 
> portability efforts other runners may soon be able to use Python SDK. Having 
> HDFS support will allow these runners to execute large scale jobs without 
> using GCS. 
> Following suggests some libraries that can be used to connect to HDFS from 
> Python.
> http://wesmckinney.com/blog/python-hdfs-interfaces/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3105) Cancel Jenkins builds when an update to a PR makes a prior build irrelevant

2017-10-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221478#comment-16221478
 ] 

ASF GitHub Bot commented on BEAM-3105:
--

Github user lukecwik closed the pull request at:

https://github.com/apache/beam/pull/4042


> Cancel Jenkins builds when an update to a PR makes a prior build irrelevant
> ---
>
> Key: BEAM-3105
> URL: https://issues.apache.org/jira/browse/BEAM-3105
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Luke Cwik
>Priority: Minor
>
> It seems as though we can cancel builds using the *ghprbCancelBuildsOnUpdate* 
> extension:
> See:
> https://builds.apache.org/plugin/job-dsl/api-viewer/index.html#method/org.jenkinsci.plugins.ghprb.extensions.GhprbExtension$$List.ghprbCancelBuildsOnUpdate



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4045: Reset Jenkins config

2017-10-26 Thread lukecwik

Github user lukecwik closed the pull request at:

https://github.com/apache/beam/pull/4045


---

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #2682

2017-10-26 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0c2211375d1fc4abebda7aa7cf7960e7b09fdc1b (origin/master)
java.lang.NoClassDefFoundError: Could not initialize class 
jenkins.model.Jenkins$MasterComputer
at 
org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:71)
at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:896)
at 
hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:870)
at 
hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:829)
at hudson.remoting.UserRequest.perform(UserRequest.java:181)
at hudson.remoting.UserRequest.perform(UserRequest.java:52)
at hudson.remoting.Request$2.run(Request.java:336)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
at ..remote call to beam8(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1554)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:281)
at hudson.remoting.Channel.call(Channel.java:839)
Caused: java.io.IOException: Remote call on beam8 failed
at hudson.remoting.Channel.call(Channel.java:847)
at 
hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
at com.sun.proxy.$Proxy108.withRepository(Unknown Source)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl.withRepository(RemoteGitImpl.java:235)
at hudson.plugins.git.GitSCM.printCommitMessageToLog(GitSCM.java:1195)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1159)
at hudson.scm.SCM.checkout(SCM.java:495)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1212)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:566)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:491)
at hudson.model.Run.execute(Run.java:1737)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:419)
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0c2211375d1fc4abebda7aa7cf7960e7b09fdc1b (origin/master)
java.lang.NoClassDefFoundError: Could not initialize class 
jenkins.model.Jenkins$MasterComputer
at 
org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:71)
at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
hudson.remoting.RemoteInvocationHandler$RPCRequest

[GitHub] beam pull request #4042: [BEAM-3105] Cancel Jenkins builds when an update to...

2017-10-26 Thread lukecwik

Github user lukecwik closed the pull request at:

https://github.com/apache/beam/pull/4042


---

[jira] [Assigned] (BEAM-1872) implement Reshuffle transform in python, make it experimental in Java

2017-10-26 Thread Udi Meiri (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri reassigned BEAM-1872:
---

Assignee: Udi Meiri

> implement Reshuffle transform in python, make it experimental in Java
> -
>
> Key: BEAM-1872
> URL: https://issues.apache.org/jira/browse/BEAM-1872
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>  Labels: sdk-consistency
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3105) Cancel Jenkins builds when an update to a PR makes a prior build irrelevant

2017-10-26 Thread Luke Cwik (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221475#comment-16221475
 ] 

Luke Cwik commented on BEAM-3105:
-

Turns out the GHBRP plugin doesn't expose the ability to cancel through the 
extensions context.

https://github.com/jenkinsci/ghprb-plugin/blob/5cbe08938401bb04e65f9a6b1bbe4d2a37e1f036/src/main/java/org/jenkinsci/plugins/ghprb/jobdsl/GhprbExtensionContext.java

After adding support for that and the GHPRB plugin being released, Apache INFRA 
would need to update the verison of the GHBRP plugin available.

> Cancel Jenkins builds when an update to a PR makes a prior build irrelevant
> ---
>
> Key: BEAM-3105
> URL: https://issues.apache.org/jira/browse/BEAM-3105
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Luke Cwik
>Priority: Minor
>
> It seems as though we can cancel builds using the *ghprbCancelBuildsOnUpdate* 
> extension:
> See:
> https://builds.apache.org/plugin/job-dsl/api-viewer/index.html#method/org.jenkinsci.plugins.ghprb.extensions.GhprbExtension$$List.ghprbCancelBuildsOnUpdate



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-3105) Cancel Jenkins builds when an update to a PR makes a prior build irrelevant

2017-10-26 Thread Luke Cwik (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-3105:
---

Assignee: (was: Luke Cwik)

> Cancel Jenkins builds when an update to a PR makes a prior build irrelevant
> ---
>
> Key: BEAM-3105
> URL: https://issues.apache.org/jira/browse/BEAM-3105
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Luke Cwik
>Priority: Minor
>
> It seems as though we can cancel builds using the *ghprbCancelBuildsOnUpdate* 
> extension:
> See:
> https://builds.apache.org/plugin/job-dsl/api-viewer/index.html#method/org.jenkinsci.plugins.ghprb.extensions.GhprbExtension$$List.ghprbCancelBuildsOnUpdate



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-1872) implement Reshuffle transform in python, make it experimental in Java

2017-10-26 Thread Chamikara Jayalath (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath reassigned BEAM-1872:


Assignee: (was: Kenneth Knowles)

> implement Reshuffle transform in python, make it experimental in Java
> -
>
> Key: BEAM-1872
> URL: https://issues.apache.org/jira/browse/BEAM-1872
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-py-core
>Reporter: Ahmet Altay
>  Labels: sdk-consistency
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-1872) implement Reshuffle transform in python, make it experimental in Java

2017-10-26 Thread Chamikara Jayalath (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath reassigned BEAM-1872:


Assignee: Chamikara Jayalath  (was: Kenneth Knowles)

> implement Reshuffle transform in python, make it experimental in Java
> -
>
> Key: BEAM-1872
> URL: https://issues.apache.org/jira/browse/BEAM-1872
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Chamikara Jayalath
>  Labels: sdk-consistency
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-1872) implement Reshuffle transform in python, make it experimental in Java

2017-10-26 Thread Chamikara Jayalath (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath reassigned BEAM-1872:


Assignee: Kenneth Knowles

> implement Reshuffle transform in python, make it experimental in Java
> -
>
> Key: BEAM-1872
> URL: https://issues.apache.org/jira/browse/BEAM-1872
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Kenneth Knowles
>  Labels: sdk-consistency
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4045: Reset Jenkins config

2017-10-26 Thread lukecwik

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/4045

Reset Jenkins config

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam jenkins2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4045.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4045


commit b11b30bd36d028f3cf659b0cb33bdc43eabba4f3
Author: Luke Cwik 
Date:   2017-10-26T23:51:29Z

Reset Jenkins config




---

[jira] [Created] (BEAM-3110) The transform Read(UnboundedKafkaSource) is currently not supported

2017-10-26 Thread Xu Mingmin (JIRA)

Xu Mingmin created BEAM-3110:


 Summary: The transform Read(UnboundedKafkaSource) is currently not 
supported
 Key: BEAM-3110
 URL: https://issues.apache.org/jira/browse/BEAM-3110
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Affects Versions: 2.2.0, 2.3.0
Reporter: Xu Mingmin
Assignee: Aljoscha Krettek


I see this issue when submitting a job to Flink cluster. It appears after build 
{{2.2.0-20170912.083349-51}}.

{code}
org.apache.flink.client.program.ProgramInvocationException: The main method 
caused an error.
at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381)
at 
org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259)
at 
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130)
at 
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1129)
Caused by: java.lang.UnsupportedOperationException: The transform 
Read(UnboundedKafkaSource) is currently not supported.
at 
org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.visitPrimitiveTransform(FlinkStreamingPipelineTranslator.java:113)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:666)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:658)
at 
org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at 
org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:451)
at 
org.apache.beam.runners.flink.FlinkPipelineTranslator.translate(FlinkPipelineTranslator.java:38)
at 
org.apache.beam.runners.flink.FlinkStreamingPipelineTranslator.translate(FlinkStreamingPipelineTranslator.java:69)
at 
org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:104)
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:113)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:304)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:290)
{code} 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-664) Port Dataflow SDK WordCount walkthrough to Beam site

2017-10-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221449#comment-16221449
 ] 

ASF GitHub Bot commented on BEAM-664:
-

melap commented on issue #336: [BEAM-664] Update Java MinimalWordCount 
instructions
URL: https://github.com/apache/beam-site/pull/336#issuecomment-339829871
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Port Dataflow SDK WordCount walkthrough to Beam site
> 
>
> Key: BEAM-664
> URL: https://issues.apache.org/jira/browse/BEAM-664
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Hadar Hod
> Fix For: Not applicable
>
>
> Port the WordCount walkthrough from Dataflow docs to Beam website. 
> * Copy prose (translate from html to md, remove Dataflow references, etc)
> * Add accurate "How to Run" instructions for each of the WC examples
> * Include code snippets from real examples



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

svn commit: r22691 - /dev/beam/2.2.0/

2017-10-26 Thread reuvenlax

Author: reuvenlax
Date: Thu Oct 26 22:53:17 2017
New Revision: 22691

Log:
2.2.0 RC1 release

Added:
dev/beam/2.2.0/
dev/beam/2.2.0/apache-beam-2.2.0-python.zip   (with props)
dev/beam/2.2.0/apache-beam-2.2.0-python.zip.asc
dev/beam/2.2.0/apache-beam-2.2.0-python.zip.md5
dev/beam/2.2.0/apache-beam-2.2.0-python.zip.sha1
dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip   (with props)
dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.asc
dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.md5
dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.sha1

Added: dev/beam/2.2.0/apache-beam-2.2.0-python.zip
==
Binary file - no diff available.

Propchange: dev/beam/2.2.0/apache-beam-2.2.0-python.zip
--
svn:mime-type = application/octet-stream

Added: dev/beam/2.2.0/apache-beam-2.2.0-python.zip.asc
==
--- dev/beam/2.2.0/apache-beam-2.2.0-python.zip.asc (added)
+++ dev/beam/2.2.0/apache-beam-2.2.0-python.zip.asc Thu Oct 26 22:53:17 2017
@@ -0,0 +1,16 @@
+-BEGIN PGP SIGNATURE-
+
+iQIzBAABCAAdFiEEuyC2JJPY3kqDVd0KO8cuG7mLdwgFAlnyX/MACgkQO8cuG7mL
+dwiNCQ/+N2fSrOoD5yt6dQf0I4OgJQs+fwpNHLasUgfjEmjyH0/i/8kS2LGutE1h
+Xu2a4gPRcMchTo3jCZMiFyc8u5kTICm6Vn/K1VITa/E25dRBr7qWGspbi9aFMZME
+kOSNdkeu0EF7leWfwG322AFb8sITcEAYcjZDvATU/SRARenx/3yVpxcuGHQsbwE4
+1+S/Swh46Wkde7GLf3PhZmZB1zYIC99OBkDEmf7P6+VAuy0PCEtLtL41NOUOt0HY
+xKUsi6PRFzM1515Ng/m4Nxy5ri/Lrqw3E1Wm+Uyy6tW+FyxAgImEVNVgPeH8OeCs
+X+fmBDcRnz/a2TrW9AgH3Lkol+Iz+3b1TLXQCjK01CaZc41XoGgEmmWYr9iijqRa
+ux8EVodw86Z0MA4whYzmvZ+oDBjSBHMBhMV+ZDgZvgQhMqIV1+svOzZFBmTD3lHz
+3zRUFzy9z3mW+no03epSehMtGdR1yKCGGqWvAcYVLD/+9Nq1JraIbw1suleR6JmT
+G+gzn6En3bnrWST7TQD8MWZMqSBH3ZpF9pQI/fmXlX7yL2NqHXxanjMc8popnm2Y
+Rwtw1yvho8PhrmKcoyg/wrCH94EypDNPb5rTefJWQLKbDBkBx2ABV33240MeKHkF
+O6F+XY8BeEsHblEMjByix5gTuwG//FGHtTeiQvG1Sm1mx9qh20E=
+=o/lX
+-END PGP SIGNATURE-

Added: dev/beam/2.2.0/apache-beam-2.2.0-python.zip.md5
==
--- dev/beam/2.2.0/apache-beam-2.2.0-python.zip.md5 (added)
+++ dev/beam/2.2.0/apache-beam-2.2.0-python.zip.md5 Thu Oct 26 22:53:17 2017
@@ -0,0 +1 @@
+9af76a98b10fb95804e9a262cd1e3ddb  apache-beam-2.2.0-python.zip

Added: dev/beam/2.2.0/apache-beam-2.2.0-python.zip.sha1
==
--- dev/beam/2.2.0/apache-beam-2.2.0-python.zip.sha1 (added)
+++ dev/beam/2.2.0/apache-beam-2.2.0-python.zip.sha1 Thu Oct 26 22:53:17 2017
@@ -0,0 +1 @@
+55906fd8c728017d1829db3cbcd7eaf52a6fd4f0  apache-beam-2.2.0-python.zip

Added: dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip
==
Binary file - no diff available.

Propchange: dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip
--
svn:mime-type = application/octet-stream

Added: dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.asc
==
--- dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.asc (added)
+++ dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.asc Thu Oct 26 22:53:17 
2017
@@ -0,0 +1,16 @@
+-BEGIN PGP SIGNATURE-
+
+iQIzBAABCAAdFiEEuyC2JJPY3kqDVd0KO8cuG7mLdwgFAlnyNuoACgkQO8cuG7mL
+dwhfxA//UCwCWqFf282CpJ/o5Hm3xEmjU/P9y33R9nGTaR66eObAaAHlbnjr696Q
+DTh9AxQ9vs0yZLtScCbHav7rIMjaSM3S5raDaDN4Ggis18FBljggM2DQ4GC84dVN
+i0GF22c3L0J03h+TufFFyWw4KEiRIoNpPX6P0joV/usn3P6XfpzW2g6QSqlSjeIh
+AjzwLpFSNQWbBFPRY9fvp/zR4+SkiY6ZGAfJKB3Z++lctjSF16ShpYN9ykEw2+lo
+s+CTVM3wFSNQt3QHyB6M30HhUGkzlA9Cjo6BSUcT7Y05FSQNotsGul3nE7bDZFYb
+TlaMbEiYSlt0ZBXTPLQ6kUUz2dWHQjLdIsrKzOAOY3NnLsIB9sOKve5sRUwPW4q2
+3J0u6UeVbC6tLBi5qv3rw45T1BPqTZlwuIfKKLqTXtL/kSDJx171NHf0QiIQfXYc
+49Wy8mt12utLyPUBzv4iPpj82jroEeBWxNQWyhb2RYMx34O6egOZrXevCATq6CJ3
+mvt4vuplpR/0NcDpIMOTUrj6YGgsXqmhhluyfzqML/WE0JcPuNb8LamGR3Wngr71
+g19fnmvt3+0sNdlUmIUQONKkAiP8X9N90q00fuLuHPLVnc02u8S55+7MIpSLAf3R
+sdT5ku5JKxTXBNdGRBnFMdNw1IlmdcUu0QXFt56K0UUbvHCedf0=
+=vEsg
+-END PGP SIGNATURE-

Added: dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.md5
==
--- dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.md5 (added)
+++ dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.md5 Thu Oct 26 22:53:17 
2017
@@ -0,0 +1 @@
+f54ca8b4f7a31df877abb64b5f179f8c  apache-beam-2.2.0-source-release.zip

Added: dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.sha1
==
--- dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.sha1 (added)
+++ dev/beam/2.2.0/apache-beam-2.2.0-source-release.zip.sha1 Thu Oct 26 
22:53:17 2017
@@ -0

[jira] [Created] (BEAM-3109) Add an element batching transform

2017-10-26 Thread Ahmet Altay (JIRA)

Ahmet Altay created BEAM-3109:
-

 Summary: Add an element batching transform
 Key: BEAM-3109
 URL: https://issues.apache.org/jira/browse/BEAM-3109
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Ahmet Altay
Assignee: Robert Bradshaw
 Fix For: 2.2.0


Merge https://github.com/apache/beam/pull/3971 to the release branch



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3108) Align names with those produced by the dataflow runner harness

2017-10-26 Thread Ahmet Altay (JIRA)

Ahmet Altay created BEAM-3108:
-

 Summary: Align names with those produced by the dataflow runner 
harness
 Key: BEAM-3108
 URL: https://issues.apache.org/jira/browse/BEAM-3108
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Ahmet Altay
Assignee: Robert Bradshaw
 Fix For: 2.2.0


Merge https://github.com/apache/beam/pull/3941 to the release branch



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (BEAM-2829) Add ability to set job labels in DataflowPipelineOptions

2017-10-26 Thread Ahmet Altay (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reopened BEAM-2829:
---
  Assignee: Ahmet Altay  (was: Zongwei Zhou)

Re-opening to track merging https://github.com/apache/beam/pull/3993 to the 
release branch.

> Add ability to set job labels in DataflowPipelineOptions
> 
>
> Key: BEAM-2829
> URL: https://issues.apache.org/jira/browse/BEAM-2829
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Zongwei Zhou
>Assignee: Ahmet Altay
>Priority: Minor
> Fix For: 2.2.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Enable setting job labels --labels in DataflowPipelineOptions (earlier 
> Dataflow SDK 1.x supports this)
> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.java



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3107) Python Fnapi based workloads failing

2017-10-26 Thread Ahmet Altay (JIRA)

Ahmet Altay created BEAM-3107:
-

 Summary: Python Fnapi based workloads failing
 Key: BEAM-3107
 URL: https://issues.apache.org/jira/browse/BEAM-3107
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-harness
Affects Versions: 2.2.0
Reporter: Ahmet Altay
Assignee: Valentyn Tymofieiev
 Fix For: 2.2.0


Python post commits are failing because the runner harness is not compatible 
with the sdk harness.

We need a new runner harness compatible with: 
https://github.com/apache/beam/commit/80c6f4ec0c2a3cc3a441289a9cc8ff53cb70f863



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3064) Update dataflow runner containers for the release branch

2017-10-26 Thread Ahmet Altay (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-3064:
--
Summary: Update dataflow runner containers for the release branch  (was: 
Update dataflow runner containers for the release bracn)

> Update dataflow runner containers for the release branch
> 
>
> Key: BEAM-3064
> URL: https://issues.apache.org/jira/browse/BEAM-3064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Critical
> Fix For: 2.2.0
>
>
> Blocked by:
> https://github.com/apache/beam/pull/3970
> https://github.com/apache/beam/pull/3941 - cp into release branch.
> cc: [~reuvenlax] [~robertwb] [~tvalentyn]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3064) Update dataflow runner containers for the release branch

2017-10-26 Thread Ahmet Altay (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-3064:
--
Fix Version/s: 2.2.0

> Update dataflow runner containers for the release branch
> 
>
> Key: BEAM-3064
> URL: https://issues.apache.org/jira/browse/BEAM-3064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Blocker
> Fix For: 2.2.0
>
>
> Blocked by:
> https://github.com/apache/beam/pull/3970
> https://github.com/apache/beam/pull/3941 - cp into release branch.
> cc: [~reuvenlax] [~robertwb] [~tvalentyn]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-3064) Update dataflow runner containers for the release branch

2017-10-26 Thread Ahmet Altay (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-3064:
--
Priority: Blocker  (was: Critical)

> Update dataflow runner containers for the release branch
> 
>
> Key: BEAM-3064
> URL: https://issues.apache.org/jira/browse/BEAM-3064
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Blocker
> Fix For: 2.2.0
>
>
> Blocked by:
> https://github.com/apache/beam/pull/3970
> https://github.com/apache/beam/pull/3941 - cp into release branch.
> cc: [~reuvenlax] [~robertwb] [~tvalentyn]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-366) Support Display Data on Composite Transforms

2017-10-26 Thread Kenneth Knowles (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221306#comment-16221306
 ] 

Kenneth Knowles commented on BEAM-366:
--

The full pipeline is staged now. DisplayData protos need to be fleshed out a 
bit more.

> Support Display Data on Composite Transforms
> 
>
> Key: BEAM-366
> URL: https://issues.apache.org/jira/browse/BEAM-366
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Ben Chambers
>
> Today, the Dataflow runner doesn't represent composites all the way to the UI 
> (it reconstructs them from the name). This means it doesn't support attaching 
> Display Data to composites.
> With the runner API refactoring, Dataflow runner should start supporting 
> composites, at which point we should make sure that Display Data is plumbed 
> through properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4044: Do not relocate generated Model Classes

2017-10-26 Thread tgroh

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/4044

Do not relocate generated Model Classes

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
Doing so with this pattern breaks the clients, as the shade plugin will
rewrite the gRPC method name as well, which servers do not implement.
All calls will then be broken.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam do_not_shade_model_in_harness

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4044.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4044


commit 2c1e4e1480c47ffbfd45688edc1213cd9ce961d5
Author: Thomas Groh 
Date:   2017-10-26T22:00:12Z

Do not relocate generated Model Classes

Doing so with this pattern breaks the clients, as the shade plugin will
rewrite the gRPC method name as well, which servers do not implement.
All calls will then be broken.




---

Build failed in Jenkins: beam_PostCommit_Python_Verify #3429

2017-10-26 Thread Apache Jenkins Server

See 


--
[...truncated 934.06 KB...]
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apach

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5113

2017-10-26 Thread Apache Jenkins Server

See 


Changes:

[kirpichov] Many improvements to TikaIO

[kirpichov] Adds ParseResult.failure()

--
[...truncated 1.35 MB...]
2017-10-26T21:36:24.249 [INFO] Excluding com.twitter:chill_2.10:jar:0.5.0 from 
the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding com.twitter:chill-java:jar:0.5.0 from 
the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.xbean:xbean-asm5-shaded:jar:4.4 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.hadoop:hadoop-client:jar:2.7.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.hadoop:hadoop-common:jar:2.7.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from 
the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.directory.api:api-util:jar:1.0.0-M20 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.curator:curator-client:jar:2.7.1 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-api:jar:2.7.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.7.3 from the shaded 
jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.hadoop:hadoop-annotations:jar:2.7.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.spark:spark-launcher_2.10:jar:1.6.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.spark:spark-network-common_2.10:jar:1.6.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.spark:spark-network-shuffle_2.10:jar:1.6.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.spark:spark-unsafe_2.10:jar:1.6.3 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding net.java.dev.jets3t:jets3t:jar:0.7.1 
from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
commons-httpclient:commons-httpclient:jar:3.1 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.curator:curator-recipes:jar:2.4.0 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.curator:curator-framework:jar:2.4.0 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016 from the shaded 
jar.
2017-10-26T21:36:24.249 [INFO] Excluding 
org.apache.commons:commons-math3:jar:3.4.1 from the shaded jar.
2017-10-26T21:36:24.249 [INFO] Excluding org.slf4j:jul-to-slf4j:jar:1.7.10 from 
the shaded jar.
2017-10-26T21:36:24.250 [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.10 
from the shaded jar.
2017-10-26T21:36:24.250 [INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded 
jar.
2017-10-26T21:36:24.250 [INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.10 
from the shaded jar.
2017-10-26T21:36:24.250 [INFO] Excluding com.ning:compress-lzf:jar:1.0.3 from 
the shaded jar.
2017-10-26T21:36:24.250 [INFO] Excluding net.jpountz.lz4:lz4:jar:1.3.0 from the 
shaded jar.
2017-10-26T21:36:24.250 [INFO] Excluding 
org.roaringbitmap:RoaringBitmap:jar:0.5.11 from the shaded jar.
2017-10-26T21:36:24.250 [INFO] Excluding commons-net:commons-net:jar:2.2 from 
the shaded jar.
2017-10-26T21:36:24.250 [INFO] Excluding 
com.typesafe.akka:akka-remote_2.10:jar:2.3.11 from the shaded jar.
2017-10-26T21:36:24.250 [INFO] Excluding 
com.typesafe.akka:akka-actor_2.10:jar:2.3.11 from the shaded jar.
2017-10-26T21:36:24.250 [INFO] Excluding 
org.uncommons.maths:uncommons-maths:jar:1.2.2a from the shaded jar.
2017-10-26T21:36:24.250 [INFO] Excluding 
com.typesafe.akka:akka-slf4j_2.10:jar:2.3.11 from the shaded jar.
2017-10-26T21:36:24.250 [INFO] Excluding 
org.scala-lang:scala-library:j

[jira] [Closed] (BEAM-2994) Refactor TikaIO

2017-10-26 Thread Eugene Kirpichov (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-2994.
--
Resolution: Fixed

> Refactor TikaIO
> ---
>
> Key: BEAM-2994
> URL: https://issues.apache.org/jira/browse/BEAM-2994
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-extensions
>Affects Versions: 2.2.0
>Reporter: Sergey Beryozkin
>Assignee: Sergey Beryozkin
> Fix For: 2.3.0
>
>
> TikaIO is currently implemented as a BoundedSource and asynchronous 
> BoundedReader returning individual document's text chunks as Strings, 
> eventually passed unordered (and not linked to the original documents) to the 
> pipeline functions.
> It was decided in the recent beam-dev thread that initially TikaIO should 
> support the cases where only a single composite bean per file, capturing the 
> file content, location (or name) and metadata, should flow to the pipeline, 
> and thus avoiding the need to implement TikaIO as a BoundedSource/Reader.
> Enhancing  TikaIO to support the streaming of the content into the pipelines 
> may be considered in the next phase, based on the specific use-cases... 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5110

2017-10-26 Thread Apache Jenkins Server

See 


Changes:

[altay] Add python_requires to setup.py

--
[...truncated 1.97 MB...]
2017-10-26T21:07:51.601 [INFO] Excluding 
org.tachyonproject:tachyon-underfs-hdfs:jar:0.8.2 from the shaded jar.
2017-10-26T21:07:51.601 [INFO] Excluding 
org.tachyonproject:tachyon-underfs-s3:jar:0.8.2 from the shaded jar.
2017-10-26T21:07:51.601 [INFO] Excluding 
org.tachyonproject:tachyon-underfs-local:jar:0.8.2 from the shaded jar.
2017-10-26T21:07:51.601 [INFO] Excluding net.razorvine:pyrolite:jar:4.9 from 
the shaded jar.
2017-10-26T21:07:51.601 [INFO] Excluding net.sf.py4j:py4j:jar:0.9 from the 
shaded jar.
2017-10-26T21:07:51.601 [INFO] Excluding 
org.spark-project.spark:unused:jar:1.0.0 from the shaded jar.
2017-10-26T21:07:51.601 [INFO] Excluding 
org.apache.spark:spark-streaming_2.10:jar:1.6.3 from the shaded jar.
2017-10-26T21:08:11.610 [INFO] Replacing original artifact with shaded artifact.
2017-10-26T21:08:11.820 [INFO] 
2017-10-26T21:08:11.820 [INFO] --- maven-javadoc-plugin:3.0.0-M1:jar 
(attach-javadocs) @ beam-sdks-java-javadoc ---
2017-10-26T21:08:11.825 [INFO] Not executing Javadoc as the project is not a 
Java classpath-capable package
2017-10-26T21:08:11.948 [INFO] 
2017-10-26T21:08:11.948 [INFO] --- maven-source-plugin:3.0.1:jar-no-fork 
(attach-sources) @ beam-sdks-java-javadoc ---
2017-10-26T21:08:12.060 [INFO] 
2017-10-26T21:08:12.060 [INFO] --- maven-source-plugin:3.0.1:test-jar-no-fork 
(attach-test-sources) @ beam-sdks-java-javadoc ---
2017-10-26T21:08:12.183 [INFO] 
2017-10-26T21:08:12.183 [INFO] --- 
reproducible-build-maven-plugin:0.3:strip-jar (default) @ 
beam-sdks-java-javadoc ---
2017-10-26T21:08:12.184 [INFO] Stripping 

2017-10-26T21:08:12.468 [INFO] 
2017-10-26T21:08:12.468 [INFO] --- maven-dependency-plugin:3.0.1:analyze-only 
(default) @ beam-sdks-java-javadoc ---
2017-10-26T21:08:12.469 [INFO] Skipping plugin execution
2017-10-26T21:08:12.579 [INFO] 
2017-10-26T21:08:12.579 [INFO] --- maven-install-plugin:2.5.2:install 
(default-install) @ beam-sdks-java-javadoc ---
2017-10-26T21:08:12.581 [INFO] Installing 

 to 

2017-10-26T21:08:12.583 [INFO] Installing 

 to 

[JENKINS] Archiving disabled
2017-10-26T21:08:14.150 [INFO]  
   
2017-10-26T21:08:14.150 [INFO] 

2017-10-26T21:08:14.150 [INFO] Building Apache Beam :: Parent 2.3.0-SNAPSHOT
2017-10-26T21:08:14.150 [INFO] 

2017-10-26T21:08:14.175 [WARNING] The POM for 
org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, no dependency 
information available
2017-10-26T21:08:14.198 [WARNING] Failed to retrieve plugin descriptor for 
org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be 
resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
https://repo.maven.apache.org/maven2 was cached in the local repository, 
resolution will not be reattempted until the update interval of central has 
elapsed or updates are forced
2017-10-26T21:08:14.199 [INFO] 
2017-10-26T21:08:14.199 [INFO] --- coveralls-maven-plugin:4.3.0:report 
(default-cli) @ beam-parent ---
2017-10-26T21:08:14.214 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.5.2/httpmime-4.5.2.pom
2017-10-26T21:08:14.259 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.5.2/httpmime-4.5.2.pom
 (5 KB at 103.8 KB/sec)
2017-10-26T21:08:14.264 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.8.3/jackson-core-2.8.3.pom
2017-10-26T21:08:14.306 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.8.3/jackson-core-2.8.3.pom
 (6 KB at 125.9 KB/sec)
2017-10-26T21:08:14.311 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.8.3/jackson-annotations-2.8.3.pom
2017-10-26T21:08:14.355 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core

[GitHub] beam pull request #4043: Remove coveralls invocations from all Jenkins jobs

2017-10-26 Thread kennknowles

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4043

Remove coveralls invocations from all Jenkins jobs

It is flaking, and I don't think it has added such value that we need to 
put effort into it.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam coveralls

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4043.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4043


commit 7eb2446fa77e63e9c05d21599452ff7d4ff68105
Author: Kenneth Knowles 
Date:   2017-10-26T20:57:53Z

Remove coveralls invocations from all jobs




---

Build failed in Jenkins: beam_PostCommit_Python_Verify #3428

2017-10-26 Thread Apache Jenkins Server

See 


Changes:

[kirpichov] Many improvements to TikaIO

[kirpichov] Adds ParseResult.failure()

--
[...truncated 931.80 KB...]
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying ap

[jira] [Created] (BEAM-3106) Consider not pinning all python dependencies, or moving them to requirements.txt

2017-10-26 Thread Maximilian Roos (JIRA)

Maximilian Roos created BEAM-3106:
-

 Summary: Consider not pinning all python dependencies, or moving 
them to requirements.txt
 Key: BEAM-3106
 URL: https://issues.apache.org/jira/browse/BEAM-3106
 Project: Beam
  Issue Type: Wish
  Components: build-system
Affects Versions: 2.1.0
 Environment: python
Reporter: Maximilian Roos
Assignee: Davor Bonaci


Currently all python dependencies are [pinned or 
capped|https://github.com/apache/beam/blob/master/sdks/python/setup.py#L97]

While there's a good argument for supplying a `requirements.txt` with well 
tested dependencies, having them specified in `setup.py` forces them to an 
exact state on each install of Beam. This makes using Beam in any environment 
with other libraries nigh on impossible. 

This is particularly severe for the `gcp` dependencies, where we have libraries 
that won't work with an older version (but Beam _does_ work with an newer 
version). We have to do a bunch of gymnastics to get the correct versions 
installed because of this. Unfortunately, airflow repeats this practice and 
conflicts on a number of dependencies, adding further complication (but, again 
there is no real conflict).

I haven't seen this practice outside of the Apache & Google ecosystem - for 
example no libraries in numerical python do this. Here's a [discussion on 
SO|https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #4152

2017-10-26 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-2994) Refactor TikaIO

2017-10-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221150#comment-16221150
 ] 

ASF GitHub Bot commented on BEAM-2994:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3835


> Refactor TikaIO
> ---
>
> Key: BEAM-2994
> URL: https://issues.apache.org/jira/browse/BEAM-2994
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-extensions
>Affects Versions: 2.2.0
>Reporter: Sergey Beryozkin
>Assignee: Sergey Beryozkin
> Fix For: 2.3.0
>
>
> TikaIO is currently implemented as a BoundedSource and asynchronous 
> BoundedReader returning individual document's text chunks as Strings, 
> eventually passed unordered (and not linked to the original documents) to the 
> pipeline functions.
> It was decided in the recent beam-dev thread that initially TikaIO should 
> support the cases where only a single composite bean per file, capturing the 
> file content, location (or name) and metadata, should flow to the pipeline, 
> and thus avoiding the need to implement TikaIO as a BoundedSource/Reader.
> Enhancing  TikaIO to support the streaming of the content into the pipelines 
> may be considered in the next phase, based on the specific use-cases... 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3835: [BEAM-2994] Prepare for a switch to ParDo based imp...

2017-10-26 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3835


---

[2/3] beam git commit: Adds ParseResult.failure()

2017-10-26 Thread jkff

Adds ParseResult.failure()

Also groups the tests into fewer methods providing the same coverage.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c76c518b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c76c518b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c76c518b

Branch: refs/heads/master
Commit: c76c518b62eee22334ff77fd15ae4eeae089e828
Parents: ba93dd3
Author: Eugene Kirpichov 
Authored: Wed Oct 25 16:57:36 2017 -0700
Committer: Eugene Kirpichov 
Committed: Thu Oct 26 13:20:39 2017 -0700

--
 .../beam/sdk/testing/SuccessOrFailure.java  |  21 +-
 .../beam/sdk/util/SerializableThrowable.java|  49 +
 sdks/java/io/tika/pom.xml   |   6 +
 .../apache/beam/sdk/io/tika/ParseResult.java| 104 +++---
 .../org/apache/beam/sdk/io/tika/TikaIO.java | 192 +
 .../beam/sdk/io/tika/ParseResultTest.java   |  54 -
 .../org/apache/beam/sdk/io/tika/TikaIOTest.java | 208 +++
 .../src/test/resources/apache-beam-tika-pdf.zip | Bin 11685 -> 0 bytes
 .../src/test/resources/apache-beam-tika.pdf | Bin 12392 -> 0 bytes
 .../src/test/resources/apache-beam-tika1.odt| Bin 12540 -> 0 bytes
 .../src/test/resources/apache-beam-tika2.odt| Bin 11412 -> 0 bytes
 .../resources/valid/apache-beam-tika-pdf.zip| Bin 0 -> 11685 bytes
 .../test/resources/valid/apache-beam-tika.odt   | Bin 0 -> 12540 bytes
 13 files changed, 351 insertions(+), 283 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c76c518b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SuccessOrFailure.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SuccessOrFailure.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SuccessOrFailure.java
index a63bbcc..bac4eb3 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SuccessOrFailure.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/SuccessOrFailure.java
@@ -18,34 +18,17 @@
 package org.apache.beam.sdk.testing;
 
 import com.google.common.base.MoreObjects;
-import java.io.IOException;
-import java.io.ObjectInputStream;
 import java.io.Serializable;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.coders.DefaultCoder;
 import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.util.SerializableThrowable;
 
 /**
  * Output of {@link PAssert}. Passed to a conclude function to act upon.
  */
 @DefaultCoder(SerializableCoder.class)
 public final class SuccessOrFailure implements Serializable {
-  private static final class SerializableThrowable implements Serializable {
-@Nullable private final Throwable throwable;
-@Nullable private final StackTraceElement[] stackTrace;
-
-private SerializableThrowable(@Nullable Throwable t) {
-  this.throwable = t;
-  this.stackTrace = (t == null) ? null : t.getStackTrace();
-}
-
-private void readObject(ObjectInputStream is) throws IOException, 
ClassNotFoundException {
-  is.defaultReadObject();
-  if (throwable != null) {
-throwable.setStackTrace(stackTrace);
-  }
-}
-  }
 
   private final boolean isSuccess;
   @Nullable
@@ -68,7 +51,7 @@ public final class SuccessOrFailure implements Serializable {
 
   @Nullable
   public AssertionError assertionError() {
-return site == null ? null : site.wrap(throwable.throwable);
+return site == null ? null : site.wrap(throwable.getThrowable());
   }
 
   public static SuccessOrFailure success() {

http://git-wip-us.apache.org/repos/asf/beam/blob/c76c518b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SerializableThrowable.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SerializableThrowable.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SerializableThrowable.java
new file mode 100644
index 000..4951958
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/SerializableThrowable.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRA

[1/3] beam git commit: Many improvements to TikaIO

2017-10-26 Thread jkff

Repository: beam
Updated Branches:
  refs/heads/master e01c78da7 -> 0c2211375


Many improvements to TikaIO

This addresses most of the comments in #3378.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ba93dd39
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ba93dd39
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ba93dd39

Branch: refs/heads/master
Commit: ba93dd39111ab2b13f811d0abeb76a49a4a4f035
Parents: e01c78d
Author: Sergey Beryozkin 
Authored: Mon Sep 11 16:11:10 2017 +0100
Committer: Eugene Kirpichov 
Committed: Thu Oct 26 12:45:19 2017 -0700

--
 sdks/java/io/tika/pom.xml   |  10 -
 .../apache/beam/sdk/io/tika/ParseResult.java|  98 
 .../org/apache/beam/sdk/io/tika/TikaIO.java | 334 ++---
 .../apache/beam/sdk/io/tika/TikaOptions.java|  78 
 .../org/apache/beam/sdk/io/tika/TikaSource.java | 466 ---
 .../beam/sdk/io/tika/ParseResultTest.java   |  43 ++
 .../org/apache/beam/sdk/io/tika/TikaIOTest.java | 252 --
 .../apache/beam/sdk/io/tika/TikaReaderTest.java |  82 
 .../apache/beam/sdk/io/tika/TikaSourceTest.java |  73 ---
 9 files changed, 392 insertions(+), 1044 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ba93dd39/sdks/java/io/tika/pom.xml
--
diff --git a/sdks/java/io/tika/pom.xml b/sdks/java/io/tika/pom.xml
index b8f7ece..d7f7e42 100644
--- a/sdks/java/io/tika/pom.xml
+++ b/sdks/java/io/tika/pom.xml
@@ -54,16 +54,6 @@
 
 
 
-joda-time
-joda-time
-
-
-
-org.slf4j
-slf4j-api
-
-
-
 org.apache.tika
 tika-core
 ${tika.version}

http://git-wip-us.apache.org/repos/asf/beam/blob/ba93dd39/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/ParseResult.java
--
diff --git 
a/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/ParseResult.java 
b/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/ParseResult.java
new file mode 100644
index 000..0a77491
--- /dev/null
+++ 
b/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/ParseResult.java
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.tika;
+
+import java.io.Serializable;
+import java.util.Arrays;
+
+import org.apache.tika.metadata.Metadata;
+
+/**
+ * Tika parse result containing the file location, metadata
+ * and content converted to String.
+ */
+@SuppressWarnings("serial")
+public class ParseResult implements Serializable {
+  private final String fileLocation;
+  private final String content;
+  private final Metadata metadata;
+  private final String[] metadataNames;
+
+  public ParseResult(String fileLocation, String content) {
+this(fileLocation, content, new Metadata());
+  }
+
+  public ParseResult(String fileLocation, String content, Metadata metadata) {
+this.fileLocation = fileLocation;
+this.content = content;
+this.metadata = metadata;
+this.metadataNames = metadata.names();
+  }
+
+  /**
+   * Gets a file content.
+   */
+  public String getContent() {
+return content;
+  }
+
+  /**
+   * Gets a file metadata.
+   */
+  public Metadata getMetadata() {
+return metadata;
+  }
+
+  /**
+   * Gets a file location.
+   */
+  public String getFileLocation() {
+return fileLocation;
+  }
+
+  @Override
+  public int hashCode() {
+int hashCode = 1;
+hashCode = 31 * hashCode + fileLocation.hashCode();
+hashCode = 31 * hashCode + content.hashCode();
+hashCode = 31 * hashCode + getMetadataHashCode();
+return hashCode;
+  }
+
+  @Override
+  public boolean equals(Object obj) {
+if (!(obj instanceof ParseResult)) {
+  return false;
+}
+
+ParseResult pr = (ParseResult) obj;
+return this.fileLocation.equals(pr.fileLocation)
+  && this.content.equals(pr.co

[3/3] beam git commit: This closes #3835: [BEAM-2994] Many improvements to TikaIO

2017-10-26 Thread jkff

This closes #3835: [BEAM-2994] Many improvements to TikaIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0c221137
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0c221137
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0c221137

Branch: refs/heads/master
Commit: 0c2211375d1fc4abebda7aa7cf7960e7b09fdc1b
Parents: e01c78d c76c518
Author: Eugene Kirpichov 
Authored: Thu Oct 26 13:21:07 2017 -0700
Committer: Eugene Kirpichov 
Committed: Thu Oct 26 13:21:07 2017 -0700

--
 .../beam/sdk/testing/SuccessOrFailure.java  |  21 +-
 .../beam/sdk/util/SerializableThrowable.java|  49 ++
 sdks/java/io/tika/pom.xml   |  16 +-
 .../apache/beam/sdk/io/tika/ParseResult.java| 144 ++
 .../org/apache/beam/sdk/io/tika/TikaIO.java | 394 
 .../apache/beam/sdk/io/tika/TikaOptions.java|  78 
 .../org/apache/beam/sdk/io/tika/TikaSource.java | 466 ---
 .../beam/sdk/io/tika/ParseResultTest.java   |  83 
 .../org/apache/beam/sdk/io/tika/TikaIOTest.java | 290 
 .../apache/beam/sdk/io/tika/TikaReaderTest.java |  82 
 .../apache/beam/sdk/io/tika/TikaSourceTest.java |  73 ---
 .../src/test/resources/apache-beam-tika-pdf.zip | Bin 11685 -> 0 bytes
 .../src/test/resources/apache-beam-tika.pdf | Bin 12392 -> 0 bytes
 .../src/test/resources/apache-beam-tika1.odt| Bin 12540 -> 0 bytes
 .../src/test/resources/apache-beam-tika2.odt| Bin 11412 -> 0 bytes
 .../resources/valid/apache-beam-tika-pdf.zip| Bin 0 -> 11685 bytes
 .../test/resources/valid/apache-beam-tika.odt   | Bin 0 -> 12540 bytes
 17 files changed, 556 insertions(+), 1140 deletions(-)
--

[beam] Git Push Summary

2017-10-26 Thread reuvenlax

Repository: beam
Updated Tags:  refs/tags/v-RC1-DupdateWorkingCopyVersions=false [created] 
ff8c2289c

beam git commit: [maven-release-plugin] prepare for next development iteration

2017-10-26 Thread reuvenlax

Repository: beam
Updated Branches:
  refs/heads/release-2.2.0 2c5f85a8e -> b6d15dc98


[maven-release-plugin] prepare for next development iteration


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b6d15dc9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b6d15dc9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b6d15dc9

Branch: refs/heads/release-2.2.0
Commit: b6d15dc98f7dc13ca0cb013980b8239149fed6c8
Parents: 2c5f85a
Author: Reuven Lax 
Authored: Thu Oct 26 13:14:34 2017 -0700
Committer: Reuven Lax 
Committed: Thu Oct 26 13:14:34 2017 -0700

--
 examples/java/pom.xml| 2 +-
 examples/java8/pom.xml   | 2 +-
 examples/pom.xml | 2 +-
 pom.xml  | 4 ++--
 runners/apex/pom.xml | 2 +-
 runners/core-construction-java/pom.xml   | 2 +-
 runners/core-java/pom.xml| 2 +-
 runners/direct-java/pom.xml  | 2 +-
 runners/flink/pom.xml| 2 +-
 runners/gcp/gcemd/pom.xml| 2 +-
 runners/gcp/gcsproxy/pom.xml | 2 +-
 runners/gcp/pom.xml  | 2 +-
 runners/gearpump/pom.xml | 2 +-
 runners/google-cloud-dataflow-java/pom.xml   | 2 +-
 runners/local-artifact-service-java/pom.xml  | 2 +-
 runners/pom.xml  | 2 +-
 runners/reference/pom.xml| 2 +-
 runners/spark/pom.xml| 2 +-
 sdks/common/fn-api/pom.xml   | 2 +-
 sdks/common/pom.xml  | 2 +-
 sdks/common/runner-api/pom.xml   | 2 +-
 sdks/go/pom.xml  | 2 +-
 sdks/java/build-tools/pom.xml| 2 +-
 sdks/java/container/pom.xml  | 2 +-
 sdks/java/core/pom.xml   | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml  | 2 +-
 sdks/java/extensions/jackson/pom.xml | 2 +-
 sdks/java/extensions/join-library/pom.xml| 2 +-
 sdks/java/extensions/pom.xml | 2 +-
 sdks/java/extensions/protobuf/pom.xml| 2 +-
 sdks/java/extensions/sorter/pom.xml  | 2 +-
 sdks/java/extensions/sql/pom.xml | 2 +-
 sdks/java/harness/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml| 2 +-
 sdks/java/io/cassandra/pom.xml   | 2 +-
 sdks/java/io/common/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml   | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml   | 2 +-
 .../io/elasticsearch-tests/elasticsearch-tests-common/pom.xml| 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml   | 2 +-
 sdks/java/io/hadoop-common/pom.xml   | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml  | 2 +-
 sdks/java/io/hadoop/input-format/pom.xml | 2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml | 2 +-
 sdks/java/io/hadoop/pom.xml  | 2 +-
 sdks/java/io/hbase/pom.xml   | 2 +-
 sdks/java/io/hcatalog/pom.xml| 2 +-
 sdks/java/io/jdbc/pom.xml| 2 +-
 sdks/java/io/jms/pom.xml | 2 +-
 sdks/java/io/kafka/pom.xml   | 2 +-
 sdks/java/io/kinesis/pom.xml | 2 +-
 sdks/java/io/mongodb/pom.xml | 2 +-
 sdks/java/io/mqtt/pom.xml| 2 +-
 sdks/java/io/pom.xml | 2 +-
 sdks/java/io/redis/pom.xml   | 2 +-
 sdks/java/io/solr/pom.xml| 2 +-
 sdks/java/io/tika/pom.xml

beam git commit: [maven-release-plugin] prepare release v-RC1-DupdateWorkingCopyVersions=false

2017-10-26 Thread reuvenlax

Repository: beam
Updated Branches:
  refs/heads/release-2.2.0 c2cb1dd19 -> 2c5f85a8e


[maven-release-plugin] prepare release v-RC1-DupdateWorkingCopyVersions=false


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2c5f85a8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2c5f85a8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2c5f85a8

Branch: refs/heads/release-2.2.0
Commit: 2c5f85a8e262f6d55fcc06faead69b38c349b42c
Parents: c2cb1dd
Author: Reuven Lax 
Authored: Thu Oct 26 12:50:46 2017 -0700
Committer: Reuven Lax 
Committed: Thu Oct 26 12:50:46 2017 -0700

--
 examples/java/pom.xml| 2 +-
 examples/java8/pom.xml   | 2 +-
 examples/pom.xml | 2 +-
 pom.xml  | 4 ++--
 runners/apex/pom.xml | 2 +-
 runners/core-construction-java/pom.xml   | 2 +-
 runners/core-java/pom.xml| 2 +-
 runners/direct-java/pom.xml  | 2 +-
 runners/flink/pom.xml| 2 +-
 runners/gcp/gcemd/pom.xml| 2 +-
 runners/gcp/gcsproxy/pom.xml | 2 +-
 runners/gcp/pom.xml  | 2 +-
 runners/gearpump/pom.xml | 2 +-
 runners/google-cloud-dataflow-java/pom.xml   | 2 +-
 runners/local-artifact-service-java/pom.xml  | 2 +-
 runners/pom.xml  | 2 +-
 runners/reference/pom.xml| 2 +-
 runners/spark/pom.xml| 2 +-
 sdks/common/fn-api/pom.xml   | 2 +-
 sdks/common/pom.xml  | 2 +-
 sdks/common/runner-api/pom.xml   | 2 +-
 sdks/go/pom.xml  | 2 +-
 sdks/java/build-tools/pom.xml| 2 +-
 sdks/java/container/pom.xml  | 2 +-
 sdks/java/core/pom.xml   | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml  | 2 +-
 sdks/java/extensions/jackson/pom.xml | 2 +-
 sdks/java/extensions/join-library/pom.xml| 2 +-
 sdks/java/extensions/pom.xml | 2 +-
 sdks/java/extensions/protobuf/pom.xml| 2 +-
 sdks/java/extensions/sorter/pom.xml  | 2 +-
 sdks/java/extensions/sql/pom.xml | 2 +-
 sdks/java/harness/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml| 2 +-
 sdks/java/io/cassandra/pom.xml   | 2 +-
 sdks/java/io/common/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml   | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml   | 2 +-
 .../io/elasticsearch-tests/elasticsearch-tests-common/pom.xml| 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml | 2 +-
 sdks/java/io/elasticsearch/pom.xml   | 2 +-
 sdks/java/io/google-cloud-platform/pom.xml   | 2 +-
 sdks/java/io/hadoop-common/pom.xml   | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml  | 2 +-
 sdks/java/io/hadoop/input-format/pom.xml | 2 +-
 sdks/java/io/hadoop/jdk1.8-tests/pom.xml | 2 +-
 sdks/java/io/hadoop/pom.xml  | 2 +-
 sdks/java/io/hbase/pom.xml   | 2 +-
 sdks/java/io/hcatalog/pom.xml| 2 +-
 sdks/java/io/jdbc/pom.xml| 2 +-
 sdks/java/io/jms/pom.xml | 2 +-
 sdks/java/io/kafka/pom.xml   | 2 +-
 sdks/java/io/kinesis/pom.xml | 2 +-
 sdks/java/io/mongodb/pom.xml | 2 +-
 sdks/java/io/mqtt/pom.xml| 2 +-
 sdks/java/io/pom.xml | 2 +-
 sdks/java/io/redis/pom.xml   | 2 +-
 sdks/java/io/solr/pom.xml| 2 +-
 sdks/java/io/tika/pom.xml

Build failed in Jenkins: beam_PostCommit_Python_Verify #3427

2017-10-26 Thread Apache Jenkins Server

See 


Changes:

[lcwik] [BEAM-2566] Decouple SDK harness from Dataflow runner by elevating

[lcwik] Add missing @RunWith to test.

--
[...truncated 928.71 KB...]
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
co

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #2680

2017-10-26 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-1664) Support Kafka0.8.x client in KafkaIO

2017-10-26 Thread Raghu Angadi (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221039#comment-16221039
 ] 

Raghu Angadi commented on BEAM-1664:


[~r7raul], there are no plans to support 0.8x. It is a very old version and 
client works entirely different. We are in fact considering deprecating 0.9.x 
support. I don't think it is worth the effort to restructure KafkaIO to support 
0.8x.


> Support  Kafka0.8.x client in KafkaIO
> -
>
> Key: BEAM-1664
> URL: https://issues.apache.org/jira/browse/BEAM-1664
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: JiJun Tang
>Assignee: Reuven Lax
>
> Kafka-0.8 is not supported yet, these's a big change from 0.8 to 0.9. So we 
> need to create a specific KafkaIO moudle for 0.8. After complete this 
> moudle,we will consider to extract common code to kafkaio-common  moudle.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-1664) Support Kafka0.8.x client in KafkaIO

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi reassigned BEAM-1664:
--

Assignee: Reuven Lax  (was: Raghu Angadi)

> Support  Kafka0.8.x client in KafkaIO
> -
>
> Key: BEAM-1664
> URL: https://issues.apache.org/jira/browse/BEAM-1664
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: JiJun Tang
>Assignee: Reuven Lax
>
> Kafka-0.8 is not supported yet, these's a big change from 0.8 to 0.9. So we 
> need to create a specific KafkaIO moudle for 0.8. After complete this 
> moudle,we will consider to extract common code to kafkaio-common  moudle.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2704) KafkaIO: NPE without key serializer set

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2704:
---
Priority: Major  (was: Blocker)

> KafkaIO: NPE without key serializer set
> ---
>
> Key: BEAM-2704
> URL: https://issues.apache.org/jira/browse/BEAM-2704
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Chris Pettitt
>Assignee: Raghu Angadi
>
> The KafkaIO javadoc implies that you do not need to set a Serializer if you 
> only want to emit values:
> {code}
>  * Often you might want to write just values without any keys to Kafka. 
> Use {@code values()} to
>  * write records with default empty(null) key:
>  *
>  * {@code
>  *  PCollection strings = ...;
>  *  strings.apply(KafkaIO.write()
>  *  .withBootstrapServers("broker_1:9092,broker_2:9092")
>  *  .withTopic("results")
>  *  .withValueSerializer(new StringSerializer()) // just need serializer 
> for value
>  *  .values()
>  *);
>  * }
> {code}
> However, if you don't set the key serializer then Kafka blows up when trying 
> to instantiate the key serializer (in Kafka 0.10.1, at least). It would be 
> more convenient if KafkaIO worked as documented and assigned a null 
> serializer if values() is used.  
> Relevant stack trace:
> {code}
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:230)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:163)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.setup(KafkaIO.java:1582)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter$DoFnInvoker.invokeSetup(Unknown
>  Source)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2704) KafkaIO: NPE without key serializer set

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2704:
---
Fix Version/s: (was: 2.2.0)

> KafkaIO: NPE without key serializer set
> ---
>
> Key: BEAM-2704
> URL: https://issues.apache.org/jira/browse/BEAM-2704
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Chris Pettitt
>Assignee: Raghu Angadi
>Priority: Blocker
>
> The KafkaIO javadoc implies that you do not need to set a Serializer if you 
> only want to emit values:
> {code}
>  * Often you might want to write just values without any keys to Kafka. 
> Use {@code values()} to
>  * write records with default empty(null) key:
>  *
>  * {@code
>  *  PCollection strings = ...;
>  *  strings.apply(KafkaIO.write()
>  *  .withBootstrapServers("broker_1:9092,broker_2:9092")
>  *  .withTopic("results")
>  *  .withValueSerializer(new StringSerializer()) // just need serializer 
> for value
>  *  .values()
>  *);
>  * }
> {code}
> However, if you don't set the key serializer then Kafka blows up when trying 
> to instantiate the key serializer (in Kafka 0.10.1, at least). It would be 
> more convenient if KafkaIO worked as documented and assigned a null 
> serializer if values() is used.  
> Relevant stack trace:
> {code}
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:230)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:163)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.setup(KafkaIO.java:1582)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter$DoFnInvoker.invokeSetup(Unknown
>  Source)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2257) KafkaIO write without key requires a producer fn

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2257:
---
Priority: Major  (was: Blocker)

> KafkaIO write without key requires a producer fn
> 
>
> Key: BEAM-2257
> URL: https://issues.apache.org/jira/browse/BEAM-2257
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> The {{KafkaIO}} javadoc says that it's possible to write directly {{String}} 
> to the topic without key:
> {code}
>PCollection strings = ...;
>strings.apply(KafkaIO.write()
>.withBootstrapServers("broker_1:9092,broker_2:9092")
>.withTopic("results")
>.withValueSerializer(new StringSerializer()) // just need serializer 
> for value
>.values()
>  );
> {code}
> This is not fully correct:
> 1. {{withValueSerializer()}} requires a class of serializer, not an instance. 
> So, it should be {{withValueSerializer(StringSerializer.class)}}.
> 2. As the key serializer is not provider, a kafka producer fn is required, 
> else, the user will get:
> {code}
> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka 
> producer
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:321)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:156)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.setup(KafkaIO.java:1494)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:300)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:156)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.setup(KafkaIO.java:1494)
> {code}
> A possible workaround is to create a {{VoidSerializer}} and pass it via 
> {{withKeySerializer()}} or provide the producer fn.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2257) KafkaIO write without key requires a producer fn

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2257:
---
Fix Version/s: (was: 2.2.0)

> KafkaIO write without key requires a producer fn
> 
>
> Key: BEAM-2257
> URL: https://issues.apache.org/jira/browse/BEAM-2257
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> The {{KafkaIO}} javadoc says that it's possible to write directly {{String}} 
> to the topic without key:
> {code}
>PCollection strings = ...;
>strings.apply(KafkaIO.write()
>.withBootstrapServers("broker_1:9092,broker_2:9092")
>.withTopic("results")
>.withValueSerializer(new StringSerializer()) // just need serializer 
> for value
>.values()
>  );
> {code}
> This is not fully correct:
> 1. {{withValueSerializer()}} requires a class of serializer, not an instance. 
> So, it should be {{withValueSerializer(StringSerializer.class)}}.
> 2. As the key serializer is not provider, a kafka producer fn is required, 
> else, the user will get:
> {code}
> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka 
> producer
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:321)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:156)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.setup(KafkaIO.java:1494)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:300)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:156)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.setup(KafkaIO.java:1494)
> {code}
> A possible workaround is to create a {{VoidSerializer}} and pass it via 
> {{withKeySerializer()}} or provide the producer fn.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2257) KafkaIO write without key requires a producer fn

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2257:
---
Fix Version/s: 2.2.0

> KafkaIO write without key requires a producer fn
> 
>
> Key: BEAM-2257
> URL: https://issues.apache.org/jira/browse/BEAM-2257
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.2.0
>
>
> The {{KafkaIO}} javadoc says that it's possible to write directly {{String}} 
> to the topic without key:
> {code}
>PCollection strings = ...;
>strings.apply(KafkaIO.write()
>.withBootstrapServers("broker_1:9092,broker_2:9092")
>.withTopic("results")
>.withValueSerializer(new StringSerializer()) // just need serializer 
> for value
>.values()
>  );
> {code}
> This is not fully correct:
> 1. {{withValueSerializer()}} requires a class of serializer, not an instance. 
> So, it should be {{withValueSerializer(StringSerializer.class)}}.
> 2. As the key serializer is not provider, a kafka producer fn is required, 
> else, the user will get:
> {code}
> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka 
> producer
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:321)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:156)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.setup(KafkaIO.java:1494)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:300)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:156)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.setup(KafkaIO.java:1494)
> {code}
> A possible workaround is to create a {{VoidSerializer}} and pass it via 
> {{withKeySerializer()}} or provide the producer fn.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2257) KafkaIO write without key requires a producer fn

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2257:
---
Priority: Blocker  (was: Major)

> KafkaIO write without key requires a producer fn
> 
>
> Key: BEAM-2257
> URL: https://issues.apache.org/jira/browse/BEAM-2257
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.2.0
>
>
> The {{KafkaIO}} javadoc says that it's possible to write directly {{String}} 
> to the topic without key:
> {code}
>PCollection strings = ...;
>strings.apply(KafkaIO.write()
>.withBootstrapServers("broker_1:9092,broker_2:9092")
>.withTopic("results")
>.withValueSerializer(new StringSerializer()) // just need serializer 
> for value
>.values()
>  );
> {code}
> This is not fully correct:
> 1. {{withValueSerializer()}} requires a class of serializer, not an instance. 
> So, it should be {{withValueSerializer(StringSerializer.class)}}.
> 2. As the key serializer is not provider, a kafka producer fn is required, 
> else, the user will get:
> {code}
> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka 
> producer
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:321)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:156)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.setup(KafkaIO.java:1494)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:300)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:156)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.setup(KafkaIO.java:1494)
> {code}
> A possible workaround is to create a {{VoidSerializer}} and pass it via 
> {{withKeySerializer()}} or provide the producer fn.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2703) KafkaIO: watermark outside the bounds of BoundedWindow

2017-10-26 Thread Chamikara Jayalath (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-2703.
--
   Resolution: Fixed
Fix Version/s: 2.1.0

> KafkaIO: watermark outside the bounds of BoundedWindow
> --
>
> Key: BEAM-2703
> URL: https://issues.apache.org/jira/browse/BEAM-2703
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Chris Pettitt
>Assignee: Raghu Angadi
> Fix For: 2.1.0
>
>
> KafkaIO appears to use an incorrect lower bound for it's initial watermark 
> with respect to BoundedWindow.TIMESTAMP_MIN_VALUE.
> KafkaIO's initial watermark:
> new Instant(Long.MIN_VALUE) -> -9223372036854775808
> BoundedWindow.TIMESTAMP_MIN_VALUE:
> new Instant(TimeUnit.MICROSECONDS.toMillis(Long.MIN_VALUE)) -> 
> -9223372036854775
> The difference is that the last three digits have been truncated due to the 
> micro to millis conversion.
> This difference can cause errors in runners that assert that the input 
> watermark can never regress as KafkaIO gives a value below the lower bound 
> when no messages have been received yet. For consistency it would probably be 
> best for it to use BoundedWindow.TIMESTAMP_MIN_VALUE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2704) KafkaIO: NPE without key serializer set

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2704:
---
Priority: Blocker  (was: Major)

> KafkaIO: NPE without key serializer set
> ---
>
> Key: BEAM-2704
> URL: https://issues.apache.org/jira/browse/BEAM-2704
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Chris Pettitt
>Assignee: Raghu Angadi
>Priority: Blocker
> Fix For: 2.2.0
>
>
> The KafkaIO javadoc implies that you do not need to set a Serializer if you 
> only want to emit values:
> {code}
>  * Often you might want to write just values without any keys to Kafka. 
> Use {@code values()} to
>  * write records with default empty(null) key:
>  *
>  * {@code
>  *  PCollection strings = ...;
>  *  strings.apply(KafkaIO.write()
>  *  .withBootstrapServers("broker_1:9092,broker_2:9092")
>  *  .withTopic("results")
>  *  .withValueSerializer(new StringSerializer()) // just need serializer 
> for value
>  *  .values()
>  *);
>  * }
> {code}
> However, if you don't set the key serializer then Kafka blows up when trying 
> to instantiate the key serializer (in Kafka 0.10.1, at least). It would be 
> more convenient if KafkaIO worked as documented and assigned a null 
> serializer if values() is used.  
> Relevant stack trace:
> {code}
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:230)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:163)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.setup(KafkaIO.java:1582)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter$DoFnInvoker.invokeSetup(Unknown
>  Source)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2704) KafkaIO: NPE without key serializer set

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2704:
---
Fix Version/s: 2.2.0

> KafkaIO: NPE without key serializer set
> ---
>
> Key: BEAM-2704
> URL: https://issues.apache.org/jira/browse/BEAM-2704
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Chris Pettitt
>Assignee: Raghu Angadi
>Priority: Blocker
> Fix For: 2.2.0
>
>
> The KafkaIO javadoc implies that you do not need to set a Serializer if you 
> only want to emit values:
> {code}
>  * Often you might want to write just values without any keys to Kafka. 
> Use {@code values()} to
>  * write records with default empty(null) key:
>  *
>  * {@code
>  *  PCollection strings = ...;
>  *  strings.apply(KafkaIO.write()
>  *  .withBootstrapServers("broker_1:9092,broker_2:9092")
>  *  .withTopic("results")
>  *  .withValueSerializer(new StringSerializer()) // just need serializer 
> for value
>  *  .values()
>  *);
>  * }
> {code}
> However, if you don't set the key serializer then Kafka blows up when trying 
> to instantiate the key serializer (in Kafka 0.10.1, at least). It would be 
> more convenient if KafkaIO worked as documented and assigned a null 
> serializer if values() is used.  
> Relevant stack trace:
> {code}
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:230)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:163)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.setup(KafkaIO.java:1582)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter$DoFnInvoker.invokeSetup(Unknown
>  Source)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2703) KafkaIO: watermark outside the bounds of BoundedWindow

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2703:
---
Fix Version/s: (was: 2.2.0)

> KafkaIO: watermark outside the bounds of BoundedWindow
> --
>
> Key: BEAM-2703
> URL: https://issues.apache.org/jira/browse/BEAM-2703
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Chris Pettitt
>Assignee: Raghu Angadi
>
> KafkaIO appears to use an incorrect lower bound for it's initial watermark 
> with respect to BoundedWindow.TIMESTAMP_MIN_VALUE.
> KafkaIO's initial watermark:
> new Instant(Long.MIN_VALUE) -> -9223372036854775808
> BoundedWindow.TIMESTAMP_MIN_VALUE:
> new Instant(TimeUnit.MICROSECONDS.toMillis(Long.MIN_VALUE)) -> 
> -9223372036854775
> The difference is that the last three digits have been truncated due to the 
> micro to millis conversion.
> This difference can cause errors in runners that assert that the input 
> watermark can never regress as KafkaIO gives a value below the lower bound 
> when no messages have been received yet. For consistency it would probably be 
> best for it to use BoundedWindow.TIMESTAMP_MIN_VALUE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2703) KafkaIO: watermark outside the bounds of BoundedWindow

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2703:
---
Priority: Blocker  (was: Major)

> KafkaIO: watermark outside the bounds of BoundedWindow
> --
>
> Key: BEAM-2703
> URL: https://issues.apache.org/jira/browse/BEAM-2703
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Chris Pettitt
>Assignee: Raghu Angadi
>Priority: Blocker
>
> KafkaIO appears to use an incorrect lower bound for it's initial watermark 
> with respect to BoundedWindow.TIMESTAMP_MIN_VALUE.
> KafkaIO's initial watermark:
> new Instant(Long.MIN_VALUE) -> -9223372036854775808
> BoundedWindow.TIMESTAMP_MIN_VALUE:
> new Instant(TimeUnit.MICROSECONDS.toMillis(Long.MIN_VALUE)) -> 
> -9223372036854775
> The difference is that the last three digits have been truncated due to the 
> micro to millis conversion.
> This difference can cause errors in runners that assert that the input 
> watermark can never regress as KafkaIO gives a value below the lower bound 
> when no messages have been received yet. For consistency it would probably be 
> best for it to use BoundedWindow.TIMESTAMP_MIN_VALUE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2703) KafkaIO: watermark outside the bounds of BoundedWindow

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2703:
---
Priority: Major  (was: Blocker)

> KafkaIO: watermark outside the bounds of BoundedWindow
> --
>
> Key: BEAM-2703
> URL: https://issues.apache.org/jira/browse/BEAM-2703
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Chris Pettitt
>Assignee: Raghu Angadi
>
> KafkaIO appears to use an incorrect lower bound for it's initial watermark 
> with respect to BoundedWindow.TIMESTAMP_MIN_VALUE.
> KafkaIO's initial watermark:
> new Instant(Long.MIN_VALUE) -> -9223372036854775808
> BoundedWindow.TIMESTAMP_MIN_VALUE:
> new Instant(TimeUnit.MICROSECONDS.toMillis(Long.MIN_VALUE)) -> 
> -9223372036854775
> The difference is that the last three digits have been truncated due to the 
> micro to millis conversion.
> This difference can cause errors in runners that assert that the input 
> watermark can never regress as KafkaIO gives a value below the lower bound 
> when no messages have been received yet. For consistency it would probably be 
> best for it to use BoundedWindow.TIMESTAMP_MIN_VALUE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2703) KafkaIO: watermark outside the bounds of BoundedWindow

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2703:
---
Fix Version/s: 2.2.0

> KafkaIO: watermark outside the bounds of BoundedWindow
> --
>
> Key: BEAM-2703
> URL: https://issues.apache.org/jira/browse/BEAM-2703
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Chris Pettitt
>Assignee: Raghu Angadi
>Priority: Blocker
>
> KafkaIO appears to use an incorrect lower bound for it's initial watermark 
> with respect to BoundedWindow.TIMESTAMP_MIN_VALUE.
> KafkaIO's initial watermark:
> new Instant(Long.MIN_VALUE) -> -9223372036854775808
> BoundedWindow.TIMESTAMP_MIN_VALUE:
> new Instant(TimeUnit.MICROSECONDS.toMillis(Long.MIN_VALUE)) -> 
> -9223372036854775
> The difference is that the last three digits have been truncated due to the 
> micro to millis conversion.
> This difference can cause errors in runners that assert that the input 
> watermark can never regress as KafkaIO gives a value below the lower bound 
> when no messages have been received yet. For consistency it would probably be 
> best for it to use BoundedWindow.TIMESTAMP_MIN_VALUE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2979) Race condition between KafkaIO.UnboundedKafkaReader.getWatermark() and KafkaIO.UnboundedKafkaReader.advance()

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2979:
---
Priority: Blocker  (was: Major)

> Race condition between KafkaIO.UnboundedKafkaReader.getWatermark() and 
> KafkaIO.UnboundedKafkaReader.advance()
> -
>
> Key: BEAM-2979
> URL: https://issues.apache.org/jira/browse/BEAM-2979
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wesley Tanaka
>Assignee: Raghu Angadi
>Priority: Blocker
> Fix For: 2.2.0
>
>
> getWatermark() looks like this:
> {noformat}
> @Override
> public Instant getWatermark() {
>   if (curRecord == null) {
> LOG.debug("{}: getWatermark() : no records have been read yet.", 
> name);
> return initialWatermark;
>   }
>   return source.spec.getWatermarkFn() != null
>   ? source.spec.getWatermarkFn().apply(curRecord) : curTimestamp;
> }
> {noformat}
> advance() has code in it that looks like this:
> {noformat}
>   curRecord = null; // user coders below might throw.
>   // apply user deserializers.
>   // TODO: write records that can't be deserialized to a 
> "dead-letter" additional output.
>   KafkaRecord record = new KafkaRecord(
>   rawRecord.topic(),
>   rawRecord.partition(),
>   rawRecord.offset(),
>   consumerSpEL.getRecordTimestamp(rawRecord),
>   keyDeserializerInstance.deserialize(rawRecord.topic(), 
> rawRecord.key()),
>   valueDeserializerInstance.deserialize(rawRecord.topic(), 
> rawRecord.value()));
>   curTimestamp = (source.spec.getTimestampFn() == null)
>   ? Instant.now() : source.spec.getTimestampFn().apply(record);
>   curRecord = record;
> {noformat}
> There's a race condition between these two blocks of code which is exposed at 
> the very least in the FlinkRunner, which calls getWatermark() periodically 
> from a timer.
> The symptom of the race condition is a stack trace that looks like this (SDK 
> 2.0.0):
> {noformat}
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed.
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:910)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:853)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:853)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: TimerException{java.lang.NullPointerException}
>   at 
> org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run(SystemProcessingTimeService.java:220)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>   at org.apache.beam.sdk.io.kafka.KafkaIO$Read$1.apply(KafkaIO.java:568)
>   at org.apache.beam.sdk.io.kafka.KafkaIO$Read$1.apply(KafkaIO.java:565)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$UnboundedKafkaReader.getWatermark(KafkaIO.java:1210)
>   at 
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.onProcessingTime(UnboundedSourceWrapper.java:431)
>   at 
> org.apache.flink.streaming.runtime.tasks.SystemProcessing

[jira] [Updated] (BEAM-2979) Race condition between KafkaIO.UnboundedKafkaReader.getWatermark() and KafkaIO.UnboundedKafkaReader.advance()

2017-10-26 Thread Raghu Angadi (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi updated BEAM-2979:
---
Fix Version/s: 2.2.0

> Race condition between KafkaIO.UnboundedKafkaReader.getWatermark() and 
> KafkaIO.UnboundedKafkaReader.advance()
> -
>
> Key: BEAM-2979
> URL: https://issues.apache.org/jira/browse/BEAM-2979
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Wesley Tanaka
>Assignee: Raghu Angadi
> Fix For: 2.2.0
>
>
> getWatermark() looks like this:
> {noformat}
> @Override
> public Instant getWatermark() {
>   if (curRecord == null) {
> LOG.debug("{}: getWatermark() : no records have been read yet.", 
> name);
> return initialWatermark;
>   }
>   return source.spec.getWatermarkFn() != null
>   ? source.spec.getWatermarkFn().apply(curRecord) : curTimestamp;
> }
> {noformat}
> advance() has code in it that looks like this:
> {noformat}
>   curRecord = null; // user coders below might throw.
>   // apply user deserializers.
>   // TODO: write records that can't be deserialized to a 
> "dead-letter" additional output.
>   KafkaRecord record = new KafkaRecord(
>   rawRecord.topic(),
>   rawRecord.partition(),
>   rawRecord.offset(),
>   consumerSpEL.getRecordTimestamp(rawRecord),
>   keyDeserializerInstance.deserialize(rawRecord.topic(), 
> rawRecord.key()),
>   valueDeserializerInstance.deserialize(rawRecord.topic(), 
> rawRecord.value()));
>   curTimestamp = (source.spec.getTimestampFn() == null)
>   ? Instant.now() : source.spec.getTimestampFn().apply(record);
>   curRecord = record;
> {noformat}
> There's a race condition between these two blocks of code which is exposed at 
> the very least in the FlinkRunner, which calls getWatermark() periodically 
> from a timer.
> The symptom of the race condition is a stack trace that looks like this (SDK 
> 2.0.0):
> {noformat}
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed.
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:910)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:853)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:853)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
>   at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
>   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>   at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: TimerException{java.lang.NullPointerException}
>   at 
> org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run(SystemProcessingTimeService.java:220)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>   at org.apache.beam.sdk.io.kafka.KafkaIO$Read$1.apply(KafkaIO.java:568)
>   at org.apache.beam.sdk.io.kafka.KafkaIO$Read$1.apply(KafkaIO.java:565)
>   at 
> org.apache.beam.sdk.io.kafka.KafkaIO$UnboundedKafkaReader.getWatermark(KafkaIO.java:1210)
>   at 
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.onProcessingTime(UnboundedSourceWrapper.java:431)
>   at 
> org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run(SystemProcessi

svn commit: r22688 - /release/beam/KEYS

2017-10-26 Thread altay

Author: altay
Date: Thu Oct 26 18:54:02 2017
New Revision: 22688

Log:
Add public key for Reuven Lax


Modified:
release/beam/KEYS

Modified: release/beam/KEYS
==
--- release/beam/KEYS (original)
+++ release/beam/KEYS Thu Oct 26 18:54:02 2017
@@ -366,3 +366,57 @@ ihzdeN4MN35i3Gtxz4vGxvDNpADwSZMTmFT5wW5B
 WIp7YH3KH93zhbhpTgq0vvKT/U6DivZbwJzG6pMpRzOUtpfdx6I7+GhBAc3YKZU=
 =nw7v
 -END PGP PUBLIC KEY BLOCK-
+pub   4096R/B98B7708 2017-10-26
+uid  Reuven Lax (CODE SIGNING KEY) 
+sub   4096R/7153A296 2017-10-26
+-BEGIN PGP PUBLIC KEY BLOCK-
+
+mQINBFnxgUoBEADAhVyE47eaJfRLTbODLSXWbmbArwn+yyWzcdPlnVRE++lRiYi+
+HWmE3snGN5xG/yMW4D//kE9zZ/5bDUhvfczoMozixlg0wNOGhWD3Qnr0qrMD
+uXlZ91++OnHAWX9bl7TI9bvPYtyhm408VtzexCsyegjicZSw6mQr/Hjae0WsQZgc
+yrrpSWquvrSIdWtWid1DAC8wWcYoP3he/6XwzMpT2vjF+NDoX2V2x/potuUqGpIA
+IoX0LrJ77mm/YvP+uq3F04ay5zEfpFEqz0VfM4m6vmDy2sIjOkflR3dq6zP6mIeJ
+jHWGmbcwreAqZGnFs6LzP186ok92xD+mAXma3hYArfw1208NjqNOSSTkaSgWmXsa
+xiGT8sFualUCm8PBNYyFMCXyH6JV7/LUdjmpSxDbex96NqkPR8SKR9nDnDSnsBkS
+hLrhuulgeRL+wI/QCl+mf+UWbt7R6KlGz1z8Y74y2YcSRmpz3E4GQkRnKo0IhA+E
+cxCyfuuiXPgXvdB97a2VWyO+6zylC/jGxJ0L79LU7Yi9tp3CClsEPMnuo4IkUC+w
+1bgRlh0f9feqvbBZSPr5yjsbGUXbQim2asqDl+nzGSJcNiz0FdfMo0cAjCATycLQ
+cfTDiLpaFVjGHY1WM3egama7Ara0mTVnFRaIquLpWjz0VzT+30Z+wLqr8wARAQAB
+tDBSZXV2ZW4gTGF4IChDT0RFIFNJR05JTkcgS0VZKSA8cmVsYXhAZ29vZ2xlLmNv
+bT6JAjcEEwECACECGwMCHgECF4AFAlnxgdoFCwkIBwMFFQoJCAsFFgIDAQAACgkQ
+O8cuG7mLdwjkvw/+LuEvfdtStb3s9Llre/1cc3SRubPBENDN9FDKN2X+ZbgEzR0t
+hX5IE2suqrVCrBMZJjT5KEXlp8V6XUA6UFgcYv9JVhJ7y/JgQa+z+xJwZTMUUeGi
+BL4VqKJJD8NS4pyPli210ppusP/GF3qZAdirOdABOEM2v+U7E04lUDhA32AuUalB
+//MHD6kD8QiAeZPfR8IbRv2nFvput/8N03c6SqdxlYROyaWWhn+QtPxasD3VZ9I9
+ZO+JTMSWMUgLNBfyKZbXJKtI7ULSoy7WnoA3Yh3x7OS40RiQ+YlTxSgywtR5jrFC
+00AjUq9Lb4Gy4WMSwAfPFGaPwSSIJp8/cZo/sXuCXbZn947UOIaPaFDs3O4uGv+g
+5TfP9sXGqprl2pZp9O83U1EGz09NhWpvDkDODSKXfFNq9ARlm0792G/dk0FN65JO
+JAJ6SNSr1Ru77A2mond4T0UY5JWH0JBcI3HqfMxZ07dkJb/LGzLHnscDtfeRsSHc
+Vb+Cu9FNaotd8P2gjiE2+ruUy0+q1AVGKm7BSSxK6dKag8QcgXARlbv/zxDsXqZt
+eMBwmMXHhZgoG1ha6Ml92sU8cwR1mlZvQd82XkyP6sD53CA34GdxWn37ACVOjJC8
+GH3woWxYmFoGygYtcBxihPWHg7TqzuZ4qXmeG/v6u4hLQtI3LCUFbDxQ5cG5Ag0E
+WfGBSgEQALGfc9gK2frDMm7VKS4GuEyAMydsdkfhA/W5811ebn/KhKq/h4wWCiTf
+omyWBrjnW3U6okee6g1JqO0X06QYOftKzynBVSmod6ODzb8Y/gb3DA4WO5mXNabE
+nQ1gWoLzdHfzs3hRhiE+acTnwcDb9g4V71Da5Lhcxg9n/1f/7bEmLM6JPl5bRgCm
+EPSrrY6gsZHNROug7FKqpUEt8YaKr/qLc5fTUIskwQYawv5AMqKIa1qFwI1v2InU
+1r9kCmoGwWEyyrF3yfTwPRiYnSLB0TarbW8OVTUE/Wh8Ny1h8gWU56+alMj0jRRq
+ZWh2QeGeerFBqLDF0CtqB9j/bW6sD97lgzKkybYAPHE/oFZXUYUtQLI3iFrKUdil
+MIYhf6N+Vnl3EJMxT05rwM0uA3Oun9IuzmdQCatwuqcnNEQ2B3P3SkQ1EQ4sUlGs
+Rh+mTG0FkFZvrQX0Hav6K73XheI/tdHEsJDSwQSqr+ZEPm344YwNLd80XG3LidQu
+1CuMwLK64IiH3s2ekfRYQTPKUoaXlBTe3YuLeYHpJqfBGppFwtgpxfGoJSHjHNVI
+SUEQFj1PBqR2AtANtKZ6BU0lWHU2wGpx81L9iw8jdhaTAxRLg834s/bTUNd22jrL
+JsLT6m+F/AGNUlvm48M4GshvJHRZbuufmr9D3KL3BaPELTxsvCi7ABEBAAGJAh8E
+GAECAAkFAlnxgUoCGwwACgkQO8cuG7mLdwirxBAAlCslH8mS656Z+R2eajsp1s/J
+raJnBWUejwfyBvRi1uoRz0I8sxYsJsFIvmwYr6slcYg92AZPcff/7mwXqz3KDsed
+S/ShE2qp2oFezNFrZyTyWpXJkm+z/P44idMzlvhvC7I0iP6kpEp+LFFsN+lLImgk
+Y9ezg7C/3T4431cpO4gGoqn2NjRh3xyd0rN9xYect72YYtvf6jOWIS/syWNnWDz9
+ho33DWBX8btfGnbO5nbkwcm8Tkp22Pp9Bu2xzNM6CP7sF1iEK26v9i3O1m9gYi++
+JigTkDK9GJ51kvz7KYmPLl9Y7sECotcuVlXeqwa7RTMcHjes8zv9BduQtKKy6zSA
+FF6UcSTc1RWvHOAst97r7u+UAkVkUiSEN0/w89HZDG01WdxeGnvYNt/ogAnxgr0A
+8WLoKH8/T/8PXxd7XCLDAErGla7lUXqz1jPshWrMekkaiAHA1B0m7/5CEjAZixqX
+DO2rP/4EehKHOdXXF3w4x4hLFX2Fhjgi+WsYfbXif8RNYhHGiHyY0Jp7mmE4J/y5
+Nb/wMDwqDEE3cHMWJ/sOgH+MT6deEEHe4mWgUycRv2cKt8qgxn25VjtrdDTem0vt
+UAlocdzKzckER+P3B8Iu7jDdC5zT+dntZRNUaXxpXricLSPyXYGw7/a4+15xMNbx
+S6KI2Zy5KAqSs5do93Y=
+=NkhN
+-END PGP PUBLIC KEY BLOCK-

svn commit: r22687 - /dev/beam/KEYS

2017-10-26 Thread altay

Author: altay
Date: Thu Oct 26 18:51:56 2017
New Revision: 22687

Log:
Add public key for Reuven Lax


Modified:
dev/beam/KEYS

Modified: dev/beam/KEYS
==
--- dev/beam/KEYS (original)
+++ dev/beam/KEYS Thu Oct 26 18:51:56 2017
@@ -366,6 +366,9 @@ ihzdeN4MN35i3Gtxz4vGxvDNpADwSZMTmFT5wW5B
 WIp7YH3KH93zhbhpTgq0vvKT/U6DivZbwJzG6pMpRzOUtpfdx6I7+GhBAc3YKZU=
 =nw7v
 -END PGP PUBLIC KEY BLOCK-
+pub   4096R/B98B7708 2017-10-26
+uid  Reuven Lax (CODE SIGNING KEY) 
+sub   4096R/7153A296 2017-10-26
 -BEGIN PGP PUBLIC KEY BLOCK-
 
 mQINBFnxgUoBEADAhVyE47eaJfRLTbODLSXWbmbArwn+yyWzcdPlnVRE++lRiYi+

svn commit: r22686 - /dev/beam/KEYS

2017-10-26 Thread altay

Author: altay
Date: Thu Oct 26 18:47:44 2017
New Revision: 22686

Log:
Add Reuven Lax's key.


Modified:
dev/beam/KEYS

Modified: dev/beam/KEYS
==
--- dev/beam/KEYS (original)
+++ dev/beam/KEYS Thu Oct 26 18:47:44 2017
@@ -366,3 +366,54 @@ ihzdeN4MN35i3Gtxz4vGxvDNpADwSZMTmFT5wW5B
 WIp7YH3KH93zhbhpTgq0vvKT/U6DivZbwJzG6pMpRzOUtpfdx6I7+GhBAc3YKZU=
 =nw7v
 -END PGP PUBLIC KEY BLOCK-
+-BEGIN PGP PUBLIC KEY BLOCK-
+
+mQINBFnxgUoBEADAhVyE47eaJfRLTbODLSXWbmbArwn+yyWzcdPlnVRE++lRiYi+
+HWmE3snGN5xG/yMW4D//kE9zZ/5bDUhvfczoMozixlg0wNOGhWD3Qnr0qrMD
+uXlZ91++OnHAWX9bl7TI9bvPYtyhm408VtzexCsyegjicZSw6mQr/Hjae0WsQZgc
+yrrpSWquvrSIdWtWid1DAC8wWcYoP3he/6XwzMpT2vjF+NDoX2V2x/potuUqGpIA
+IoX0LrJ77mm/YvP+uq3F04ay5zEfpFEqz0VfM4m6vmDy2sIjOkflR3dq6zP6mIeJ
+jHWGmbcwreAqZGnFs6LzP186ok92xD+mAXma3hYArfw1208NjqNOSSTkaSgWmXsa
+xiGT8sFualUCm8PBNYyFMCXyH6JV7/LUdjmpSxDbex96NqkPR8SKR9nDnDSnsBkS
+hLrhuulgeRL+wI/QCl+mf+UWbt7R6KlGz1z8Y74y2YcSRmpz3E4GQkRnKo0IhA+E
+cxCyfuuiXPgXvdB97a2VWyO+6zylC/jGxJ0L79LU7Yi9tp3CClsEPMnuo4IkUC+w
+1bgRlh0f9feqvbBZSPr5yjsbGUXbQim2asqDl+nzGSJcNiz0FdfMo0cAjCATycLQ
+cfTDiLpaFVjGHY1WM3egama7Ara0mTVnFRaIquLpWjz0VzT+30Z+wLqr8wARAQAB
+tDBSZXV2ZW4gTGF4IChDT0RFIFNJR05JTkcgS0VZKSA8cmVsYXhAZ29vZ2xlLmNv
+bT6JAjcEEwECACECGwMCHgECF4AFAlnxgdoFCwkIBwMFFQoJCAsFFgIDAQAACgkQ
+O8cuG7mLdwjkvw/+LuEvfdtStb3s9Llre/1cc3SRubPBENDN9FDKN2X+ZbgEzR0t
+hX5IE2suqrVCrBMZJjT5KEXlp8V6XUA6UFgcYv9JVhJ7y/JgQa+z+xJwZTMUUeGi
+BL4VqKJJD8NS4pyPli210ppusP/GF3qZAdirOdABOEM2v+U7E04lUDhA32AuUalB
+//MHD6kD8QiAeZPfR8IbRv2nFvput/8N03c6SqdxlYROyaWWhn+QtPxasD3VZ9I9
+ZO+JTMSWMUgLNBfyKZbXJKtI7ULSoy7WnoA3Yh3x7OS40RiQ+YlTxSgywtR5jrFC
+00AjUq9Lb4Gy4WMSwAfPFGaPwSSIJp8/cZo/sXuCXbZn947UOIaPaFDs3O4uGv+g
+5TfP9sXGqprl2pZp9O83U1EGz09NhWpvDkDODSKXfFNq9ARlm0792G/dk0FN65JO
+JAJ6SNSr1Ru77A2mond4T0UY5JWH0JBcI3HqfMxZ07dkJb/LGzLHnscDtfeRsSHc
+Vb+Cu9FNaotd8P2gjiE2+ruUy0+q1AVGKm7BSSxK6dKag8QcgXARlbv/zxDsXqZt
+eMBwmMXHhZgoG1ha6Ml92sU8cwR1mlZvQd82XkyP6sD53CA34GdxWn37ACVOjJC8
+GH3woWxYmFoGygYtcBxihPWHg7TqzuZ4qXmeG/v6u4hLQtI3LCUFbDxQ5cG5Ag0E
+WfGBSgEQALGfc9gK2frDMm7VKS4GuEyAMydsdkfhA/W5811ebn/KhKq/h4wWCiTf
+omyWBrjnW3U6okee6g1JqO0X06QYOftKzynBVSmod6ODzb8Y/gb3DA4WO5mXNabE
+nQ1gWoLzdHfzs3hRhiE+acTnwcDb9g4V71Da5Lhcxg9n/1f/7bEmLM6JPl5bRgCm
+EPSrrY6gsZHNROug7FKqpUEt8YaKr/qLc5fTUIskwQYawv5AMqKIa1qFwI1v2InU
+1r9kCmoGwWEyyrF3yfTwPRiYnSLB0TarbW8OVTUE/Wh8Ny1h8gWU56+alMj0jRRq
+ZWh2QeGeerFBqLDF0CtqB9j/bW6sD97lgzKkybYAPHE/oFZXUYUtQLI3iFrKUdil
+MIYhf6N+Vnl3EJMxT05rwM0uA3Oun9IuzmdQCatwuqcnNEQ2B3P3SkQ1EQ4sUlGs
+Rh+mTG0FkFZvrQX0Hav6K73XheI/tdHEsJDSwQSqr+ZEPm344YwNLd80XG3LidQu
+1CuMwLK64IiH3s2ekfRYQTPKUoaXlBTe3YuLeYHpJqfBGppFwtgpxfGoJSHjHNVI
+SUEQFj1PBqR2AtANtKZ6BU0lWHU2wGpx81L9iw8jdhaTAxRLg834s/bTUNd22jrL
+JsLT6m+F/AGNUlvm48M4GshvJHRZbuufmr9D3KL3BaPELTxsvCi7ABEBAAGJAh8E
+GAECAAkFAlnxgUoCGwwACgkQO8cuG7mLdwirxBAAlCslH8mS656Z+R2eajsp1s/J
+raJnBWUejwfyBvRi1uoRz0I8sxYsJsFIvmwYr6slcYg92AZPcff/7mwXqz3KDsed
+S/ShE2qp2oFezNFrZyTyWpXJkm+z/P44idMzlvhvC7I0iP6kpEp+LFFsN+lLImgk
+Y9ezg7C/3T4431cpO4gGoqn2NjRh3xyd0rN9xYect72YYtvf6jOWIS/syWNnWDz9
+ho33DWBX8btfGnbO5nbkwcm8Tkp22Pp9Bu2xzNM6CP7sF1iEK26v9i3O1m9gYi++
+JigTkDK9GJ51kvz7KYmPLl9Y7sECotcuVlXeqwa7RTMcHjes8zv9BduQtKKy6zSA
+FF6UcSTc1RWvHOAst97r7u+UAkVkUiSEN0/w89HZDG01WdxeGnvYNt/ogAnxgr0A
+8WLoKH8/T/8PXxd7XCLDAErGla7lUXqz1jPshWrMekkaiAHA1B0m7/5CEjAZixqX
+DO2rP/4EehKHOdXXF3w4x4hLFX2Fhjgi+WsYfbXif8RNYhHGiHyY0Jp7mmE4J/y5
+Nb/wMDwqDEE3cHMWJ/sOgH+MT6deEEHe4mWgUycRv2cKt8qgxn25VjtrdDTem0vt
+UAlocdzKzckER+P3B8Iu7jDdC5zT+dntZRNUaXxpXricLSPyXYGw7/a4+15xMNbx
+S6KI2Zy5KAqSs5do93Y=
+=NkhN
+-END PGP PUBLIC KEY BLOCK-

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5109

2017-10-26 Thread Apache Jenkins Server

See 


Changes:

[herohde] [BEAM-3005] Add resource limits to provision proto

[herohde] [BEAM-3005] Set JVM max heap size in java container

[herohde] Declare .go and Dockerfile as text in gitattributes

[herohde] Update rat exclusion for python and Go protos

[herohde] CR: [BEAM-3005] Set JVM max heap size in java container

--
[...truncated 1.36 MB...]
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.avro:avro-ipc:jar:tests:1.7.7 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding com.twitter:chill_2.10:jar:0.5.0 from 
the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding com.twitter:chill-java:jar:0.5.0 from 
the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.xbean:xbean-asm5-shaded:jar:4.4 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.hadoop:hadoop-client:jar:2.7.3 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.hadoop:hadoop-common:jar:2.7.3 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from 
the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.directory.api:api-util:jar:1.0.0-M20 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.curator:curator-client:jar:2.7.1 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.3 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.3 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.3 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.3 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.hadoop:hadoop-yarn-api:jar:2.7.3 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.7.3 from the shaded 
jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.hadoop:hadoop-annotations:jar:2.7.3 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.spark:spark-launcher_2.10:jar:1.6.3 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.spark:spark-network-common_2.10:jar:1.6.3 from the shaded jar.
2017-10-26T18:44:13.031 [INFO] Excluding 
org.apache.spark:spark-network-shuffle_2.10:jar:1.6.3 from the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding 
org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding 
org.apache.spark:spark-unsafe_2.10:jar:1.6.3 from the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding net.java.dev.jets3t:jets3t:jar:0.7.1 
from the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding 
commons-httpclient:commons-httpclient:jar:3.1 from the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding 
org.apache.curator:curator-recipes:jar:2.4.0 from the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding 
org.apache.curator:curator-framework:jar:2.4.0 from the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding 
org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016 from the shaded 
jar.
2017-10-26T18:44:13.032 [INFO] Excluding 
org.apache.commons:commons-math3:jar:3.4.1 from the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding org.slf4j:jul-to-slf4j:jar:1.7.10 from 
the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding org.slf4j:jcl-over-slf4j:jar:1.7.10 
from the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded 
jar.
2017-10-26T18:44:13.032 [INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.10 
from the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding com.ning:compress-lzf:jar:1.0.3 from 
the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding net.jpountz.lz4:lz4:jar:1.3.0 from the 
shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding 
org.roaringbitmap:RoaringBitmap:jar:0.5.11 from the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding commons-net:commons-net:jar:2.2 from 
the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding 
com.typesafe.akka:akka-remote_2.10:jar:2.3.11 from the shaded jar.
2017-10-26T18:44:13.032 [INFO] Excluding 
com.typesafe.akka:akk

[jira] [Commented] (BEAM-2566) Java SDK harness should not depend on any runner

2017-10-26 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220989#comment-16220989
 ] 

ASF GitHub Bot commented on BEAM-2566:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4016


> Java SDK harness should not depend on any runner
> 
>
> Key: BEAM-2566
> URL: https://issues.apache.org/jira/browse/BEAM-2566
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Luke Cwik
>  Labels: portability
>
> Right now there is a dependency on the Dataflow runner. I believe this is 
> legacy due to using {{CloudObject}} temporarily but I do not claim to 
> understand the full nature of the dependency.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4016: [BEAM-2566] Decouple SDK harness from Dataflow runn...

2017-10-26 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4016


---

[3/3] beam git commit: [BEAM-2566] Decouple SDK harness from Dataflow runner by elevating experiments and SDK harness configuration to java-sdk-core.

2017-10-26 Thread lcwik

[BEAM-2566] Decouple SDK harness from Dataflow runner by elevating experiments 
and SDK harness configuration to java-sdk-core.

This closes #4016


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e01c78da
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e01c78da
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e01c78da

Branch: refs/heads/master
Commit: e01c78da7d713a23910c4bfb27d4177f6247098a
Parents: cde3b7b 77d0058
Author: Luke Cwik 
Authored: Thu Oct 26 11:40:32 2017 -0700
Committer: Luke Cwik 
Committed: Thu Oct 26 11:40:32 2017 -0700

--
 .../construction/ArtifactServiceStagerTest.java |   3 +
 .../options/DataflowPipelineDebugOptions.java   |  23 +--
 .../options/DataflowWorkerLoggingOptions.java   |   6 +
 .../DefaultPipelineOptionsRegistrar.java|   2 +
 .../beam/sdk/options/ExperimentalOptions.java   |  38 
 .../beam/sdk/options/SdkHarnessOptions.java | 173 +++
 .../beam/sdk/options/SdkHarnessOptionsTest.java |  76 
 sdks/java/harness/pom.xml   |   6 -
 .../harness/channel/ManagedChannelFactory.java  |   4 +-
 .../BeamFnDataBufferingOutboundObserver.java|   4 +-
 .../fn/harness/logging/BeamFnLoggingClient.java |  30 ++--
 .../harness/stream/StreamObserverFactory.java   |   4 +-
 .../logging/BeamFnLoggingClientTest.java|  12 +-
 13 files changed, 327 insertions(+), 54 deletions(-)
--

[1/3] beam git commit: Add missing @RunWith to test.

2017-10-26 Thread lcwik

Repository: beam
Updated Branches:
  refs/heads/master cde3b7b2d -> e01c78da7


Add missing @RunWith to test.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/77d00584
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/77d00584
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/77d00584

Branch: refs/heads/master
Commit: 77d00584a7b18c1442a190283133202c4bb33916
Parents: e5290f1
Author: Luke Cwik 
Authored: Thu Oct 19 15:04:37 2017 -0700
Committer: Luke Cwik 
Committed: Thu Oct 26 11:39:53 2017 -0700

--
 .../beam/runners/core/construction/ArtifactServiceStagerTest.java | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/77d00584/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ArtifactServiceStagerTest.java
--
diff --git 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ArtifactServiceStagerTest.java
 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ArtifactServiceStagerTest.java
index 13bd8dd..ffd023e 100644
--- 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ArtifactServiceStagerTest.java
+++ 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ArtifactServiceStagerTest.java
@@ -45,10 +45,13 @@ import org.junit.Before;
 import org.junit.Rule;
 import org.junit.Test;
 import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
 
 /**
  * Tests for {@link ArtifactServiceStager}.
  */
+@RunWith(JUnit4.class)
 public class ArtifactServiceStagerTest {
   @Rule public TemporaryFolder temp = new TemporaryFolder();

1 2 >

1 - 100 of 138 matches

Mail list logo