date:20171019

[jira] [Commented] (BEAM-3073) Connect to Apache ignite via JdbcIO sdk

2017-10-19 Thread Rick Lin (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212248#comment-16212248
 ] 

Rick Lin commented on BEAM-3073:


Hi,
In my opinion:

* The JDBC driver, org.apache.ignite.IgniteJdbcDriver, is only used to store 
data to Database(MySQL, PostgreSQL..) on disk via ignite. That is not for beam 
to output data into ignite.
* Yes, i am looking forward to the SDK for ignite I/O

Thanks for the update on the feature:).
Rick


> Connect to Apache ignite via JdbcIO sdk
> ---
>
> Key: BEAM-3073
> URL: https://issues.apache.org/jira/browse/BEAM-3073
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Rick Lin
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Hi all,
> {color:#14892c}I tried to connect Apache Ignite(In-memory) via the beam's 
> sdk:org.apache.beam.sdk.io.jdbc.JdbcIO
> Here, i am not sure if the JdbcIO sdk only is provided for some specific 
> Database: MySQL(disk), postgreSQL(disk)?{color}
> my java test code is as follows:
> import java.sql.PreparedStatement;
> import java.sql.SQLException;
> import java.util.ArrayList;
> import java.util.List;
> import org.apache.beam.sdk.Pipeline;
> import org.apache.beam.sdk.io.jdbc.JdbcIO;
> import org.apache.beam.sdk.options.PipelineOptionsFactory;
> import org.apache.beam.sdk.transforms.Create;
> import org.apache.beam.sdk.values.KV;
> import org.apache.beam.sdk.values.PCollection;
> public class BeamtoJDBC {
>   public static void main(String[] args) {
>   Integer[] value=new Integer[] {1,2,3,4,5};
>   List> dataList = new ArrayList<>();
>   int n=value.length;
>   int count=0;
>   for (int i=0; i   {
>   dataList.add(KV.of(count,value[i]));
>   count=count+1;  
>   }
>   
>   Pipeline p = 
> Pipeline.create(PipelineOptionsFactory.fromArgs(args).withValidation().create());
>   
>   PCollection> data=p.apply("create data 
> with time",Create.of(dataList));
>   data.apply(JdbcIO.>write()
>   
> .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration
>   
> .create("org.apache.ignite.IgniteJdbcDriver", 
> "jdbc:ignite://localhost:11211/")
>   )   
>   .withPreparedStatementSetter(new 
> JdbcIO.PreparedStatementSetter>() {
>   public void setParameters(KV Integer> element, PreparedStatement query)
>   throws SQLException {
>   query.setInt(1, 
> element.getKey());
>   query.setInt(2, 
> element.getValue());
>   }
>   })
>   );
>   p.run();
>   }
> }
> {color:#d04437}my error message is: 
> " InvocationTargetException: org.apache.beam.sdk.util.UserCodeException: 
> java.sql.SQLException: Cannot create PoolableConnectionFactory 
> (Failed to establish connection.): Failed to get future result due to waiting 
> timed out. "{color}
> {color:#14892c}I would like to know whether the connection between beam and 
> ignite is feasible or not?{color}
> Thanks
> Rick



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3073) Connect to Apache ignite via JdbcIO sdk

2017-10-19 Thread JIRA


[ 
https://issues.apache.org/jira/browse/BEAM-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212192#comment-16212192
 ] 

Jean-Baptiste Onofré commented on BEAM-3073:


That's two aspects:
- as Ignite provides a JDBC driver, it should work with JdbcIO even if it's not 
optimal
- I created Redis (and preparing Hazelcast) IO, so I can do the same for 
Ignite. I will tackle that.

> Connect to Apache ignite via JdbcIO sdk
> ---
>
> Key: BEAM-3073
> URL: https://issues.apache.org/jira/browse/BEAM-3073
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Rick Lin
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Hi all,
> {color:#14892c}I tried to connect Apache Ignite(In-memory) via the beam's 
> sdk:org.apache.beam.sdk.io.jdbc.JdbcIO
> Here, i am not sure if the JdbcIO sdk only is provided for some specific 
> Database: MySQL(disk), postgreSQL(disk)?{color}
> my java test code is as follows:
> import java.sql.PreparedStatement;
> import java.sql.SQLException;
> import java.util.ArrayList;
> import java.util.List;
> import org.apache.beam.sdk.Pipeline;
> import org.apache.beam.sdk.io.jdbc.JdbcIO;
> import org.apache.beam.sdk.options.PipelineOptionsFactory;
> import org.apache.beam.sdk.transforms.Create;
> import org.apache.beam.sdk.values.KV;
> import org.apache.beam.sdk.values.PCollection;
> public class BeamtoJDBC {
>   public static void main(String[] args) {
>   Integer[] value=new Integer[] {1,2,3,4,5};
>   List> dataList = new ArrayList<>();
>   int n=value.length;
>   int count=0;
>   for (int i=0; i   {
>   dataList.add(KV.of(count,value[i]));
>   count=count+1;  
>   }
>   
>   Pipeline p = 
> Pipeline.create(PipelineOptionsFactory.fromArgs(args).withValidation().create());
>   
>   PCollection> data=p.apply("create data 
> with time",Create.of(dataList));
>   data.apply(JdbcIO.>write()
>   
> .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration
>   
> .create("org.apache.ignite.IgniteJdbcDriver", 
> "jdbc:ignite://localhost:11211/")
>   )   
>   .withPreparedStatementSetter(new 
> JdbcIO.PreparedStatementSetter>() {
>   public void setParameters(KV Integer> element, PreparedStatement query)
>   throws SQLException {
>   query.setInt(1, 
> element.getKey());
>   query.setInt(2, 
> element.getValue());
>   }
>   })
>   );
>   p.run();
>   }
> }
> {color:#d04437}my error message is: 
> " InvocationTargetException: org.apache.beam.sdk.util.UserCodeException: 
> java.sql.SQLException: Cannot create PoolableConnectionFactory 
> (Failed to establish connection.): Failed to get future result due to waiting 
> timed out. "{color}
> {color:#14892c}I would like to know whether the connection between beam and 
> ignite is feasible or not?{color}
> Thanks
> Rick



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3394

2017-10-19 Thread Apache Jenkins Server

See 


--
[...truncated 884.79 KB...]
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apach

[jira] [Commented] (BEAM-3081) Our findbugs config does not actually use Nullable annotations effectively

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212108#comment-16212108
 ] 

ASF GitHub Bot commented on BEAM-3081:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4021

[BEAM-3081] Findbugs: NonNull by default

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @jkff since you mentioned the time findbugs spent, I thought I would try 
to get more value out of it anyhow.

I have wondered for a while why `@Nullable` annotations didn't take care of 
themselves, since this is maybe the most important thing findbugs ought to be 
gaining us. Turns out it is simply turned off by default. It has to be turned 
on per-package, as far as I can tell. That is just as well, because some 
packages are in better shape than others.

I addressed some low-hanging fruit to demonstrate, and the changes I made 
were universally improvements IMO, so that's nice evidence in favor of taking 
this further.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam NonNull-by-default

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4021.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4021


commit b6a23a93efebe7973ad602faa7796ff50bb71769
Author: Kenneth Knowles 
Date:   2017-10-19T22:07:12Z

Add dep on Apache-licensed findbugs-annotations implementation

Very useful library, previously unavailable under an acceptable license.  
This
is a compatible library reimplemented, discussed and now used by many Apache
projects, including Avro, Falcon, Geode, HBase, Nifi, and Rya.

commit 5f83a2dbfcc6c8f9fc892037cff426a552696905
Author: Kenneth Knowles 
Date:   2017-10-20T02:27:21Z

Make Java core SDK root dir NonNull by default

commit 4ccf818735d0b67b928390e7fbd7218820de
Author: Kenneth Knowles 
Date:   2017-10-20T02:42:44Z

NonNull by default in sdk/coders

commit b702fb102ec94c8737b55ccdce8c99087dd54fcb
Author: Kenneth Knowles 
Date:   2017-10-20T02:43:02Z

NonNull by default in sdk/annotations

commit bcb2f4563b1331600e64f2a1458e4d126e94d6a9
Author: Kenneth Knowles 
Date:   2017-10-20T02:50:47Z

NonNull by default in sdk/runners

commit 136fc4ce3648bd6eebaf8233c6269d49a0b75a11
Author: Kenneth Knowles 
Date:   2017-10-20T02:54:46Z

NonNull by default in sdk/state

commit f16feba0febb9e42fc78db63ae64c26f949c2b3a
Author: Kenneth Knowles 
Date:   2017-10-20T03:08:36Z

NonNull by default for sdk/testing

commit b92643d2b8b6c482deae408774868d72fc7848bb
Author: Kenneth Knowles 
Date:   2017-10-20T03:26:41Z

Ignore findbugs in AutoValue generated classes

commit 9d34d49de28e412aa7147442765f1012bf5d4634
Author: Kenneth Knowles 
Date:   2017-10-20T03:26:59Z

NonNull by default in metrics




> Our findbugs config does not actually use Nullable annotations effectively
> --
>
> Key: BEAM-3081
> URL: https://issues.apache.org/jira/browse/BEAM-3081
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> We use {{@Nullable}} annotations mostly appropriately, but in fact our 
> findbugs config was not delivering value based on these annotations, because 
> it does not default to {{@NonNull}}. We can and should set this default.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4021: [BEAM-3081] Findbugs: NonNull by default

2017-10-19 Thread kennknowles

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4021

[BEAM-3081] Findbugs: NonNull by default

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @jkff since you mentioned the time findbugs spent, I thought I would try 
to get more value out of it anyhow.

I have wondered for a while why `@Nullable` annotations didn't take care of 
themselves, since this is maybe the most important thing findbugs ought to be 
gaining us. Turns out it is simply turned off by default. It has to be turned 
on per-package, as far as I can tell. That is just as well, because some 
packages are in better shape than others.

I addressed some low-hanging fruit to demonstrate, and the changes I made 
were universally improvements IMO, so that's nice evidence in favor of taking 
this further.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam NonNull-by-default

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4021.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4021


commit b6a23a93efebe7973ad602faa7796ff50bb71769
Author: Kenneth Knowles 
Date:   2017-10-19T22:07:12Z

Add dep on Apache-licensed findbugs-annotations implementation

Very useful library, previously unavailable under an acceptable license.  
This
is a compatible library reimplemented, discussed and now used by many Apache
projects, including Avro, Falcon, Geode, HBase, Nifi, and Rya.

commit 5f83a2dbfcc6c8f9fc892037cff426a552696905
Author: Kenneth Knowles 
Date:   2017-10-20T02:27:21Z

Make Java core SDK root dir NonNull by default

commit 4ccf818735d0b67b928390e7fbd7218820de
Author: Kenneth Knowles 
Date:   2017-10-20T02:42:44Z

NonNull by default in sdk/coders

commit b702fb102ec94c8737b55ccdce8c99087dd54fcb
Author: Kenneth Knowles 
Date:   2017-10-20T02:43:02Z

NonNull by default in sdk/annotations

commit bcb2f4563b1331600e64f2a1458e4d126e94d6a9
Author: Kenneth Knowles 
Date:   2017-10-20T02:50:47Z

NonNull by default in sdk/runners

commit 136fc4ce3648bd6eebaf8233c6269d49a0b75a11
Author: Kenneth Knowles 
Date:   2017-10-20T02:54:46Z

NonNull by default in sdk/state

commit f16feba0febb9e42fc78db63ae64c26f949c2b3a
Author: Kenneth Knowles 
Date:   2017-10-20T03:08:36Z

NonNull by default for sdk/testing

commit b92643d2b8b6c482deae408774868d72fc7848bb
Author: Kenneth Knowles 
Date:   2017-10-20T03:26:41Z

Ignore findbugs in AutoValue generated classes

commit 9d34d49de28e412aa7147442765f1012bf5d4634
Author: Kenneth Knowles 
Date:   2017-10-20T03:26:59Z

NonNull by default in metrics




---

[jira] [Created] (BEAM-3081) Our findbugs config does not actually use Nullable annotations effectively

2017-10-19 Thread Kenneth Knowles (JIRA)

Kenneth Knowles created BEAM-3081:
-

 Summary: Our findbugs config does not actually use Nullable 
annotations effectively
 Key: BEAM-3081
 URL: https://issues.apache.org/jira/browse/BEAM-3081
 Project: Beam
  Issue Type: Improvement
  Components: build-system
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


We use {{@Nullable}} annotations mostly appropriately, but in fact our findbugs 
config was not delivering value based on these annotations, because it does not 
default to {{@NonNull}}. We can and should set this default.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PerformanceTests_Python #468

2017-10-19 Thread Apache Jenkins Server

See 


Changes:

[kenn] Add standalone version of seed job

[altay] Created Java snippets file

[kenn] Unit test to repro NPE in PTransformTranslation

[kenn] Do not crash when RawPTransform has null spec

[chamikara] Avoids generating proto files for Windows if grpcio-tools is not

[chamikara] Adding lull tracking for python sampler

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 49c3927905d606990ea1d36f1dea0a8c86c11f30 (origin/master)
Commit message: "This closes #3936"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 49c3927905d606990ea1d36f1dea0a8c86c11f30
 > git rev-list 3360b1f68e8ad3cb63fbb65c94ac3ed06f9b3ee9 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4382114150302897002.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4233578133467752264.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1255705823327773009.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/local/lib/python2.7

[jira] [Commented] (BEAM-3073) Connect to Apache ignite via JdbcIO sdk

2017-10-19 Thread Rick Lin (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212010#comment-16212010
 ] 

Rick Lin commented on BEAM-3073:


Hi,
Apache Ignite is similar to Redis or Hazelcast (in-memory key-value database).
I think that the JdbcIO sdk is not suitable for Ignite.
So, developing the in-memory IO sdk for Ignite is needed.
Thanks

> Connect to Apache ignite via JdbcIO sdk
> ---
>
> Key: BEAM-3073
> URL: https://issues.apache.org/jira/browse/BEAM-3073
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Rick Lin
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Hi all,
> {color:#14892c}I tried to connect Apache Ignite(In-memory) via the beam's 
> sdk:org.apache.beam.sdk.io.jdbc.JdbcIO
> Here, i am not sure if the JdbcIO sdk only is provided for some specific 
> Database: MySQL(disk), postgreSQL(disk)?{color}
> my java test code is as follows:
> import java.sql.PreparedStatement;
> import java.sql.SQLException;
> import java.util.ArrayList;
> import java.util.List;
> import org.apache.beam.sdk.Pipeline;
> import org.apache.beam.sdk.io.jdbc.JdbcIO;
> import org.apache.beam.sdk.options.PipelineOptionsFactory;
> import org.apache.beam.sdk.transforms.Create;
> import org.apache.beam.sdk.values.KV;
> import org.apache.beam.sdk.values.PCollection;
> public class BeamtoJDBC {
>   public static void main(String[] args) {
>   Integer[] value=new Integer[] {1,2,3,4,5};
>   List> dataList = new ArrayList<>();
>   int n=value.length;
>   int count=0;
>   for (int i=0; i   {
>   dataList.add(KV.of(count,value[i]));
>   count=count+1;  
>   }
>   
>   Pipeline p = 
> Pipeline.create(PipelineOptionsFactory.fromArgs(args).withValidation().create());
>   
>   PCollection> data=p.apply("create data 
> with time",Create.of(dataList));
>   data.apply(JdbcIO.>write()
>   
> .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration
>   
> .create("org.apache.ignite.IgniteJdbcDriver", 
> "jdbc:ignite://localhost:11211/")
>   )   
>   .withPreparedStatementSetter(new 
> JdbcIO.PreparedStatementSetter>() {
>   public void setParameters(KV Integer> element, PreparedStatement query)
>   throws SQLException {
>   query.setInt(1, 
> element.getKey());
>   query.setInt(2, 
> element.getValue());
>   }
>   })
>   );
>   p.run();
>   }
> }
> {color:#d04437}my error message is: 
> " InvocationTargetException: org.apache.beam.sdk.util.UserCodeException: 
> java.sql.SQLException: Cannot create PoolableConnectionFactory 
> (Failed to establish connection.): Failed to get future result due to waiting 
> timed out. "{color}
> {color:#14892c}I would like to know whether the connection between beam and 
> ignite is feasible or not?{color}
> Thanks
> Rick



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2927) Python SDK support for portable side input

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211992#comment-16211992
 ] 

ASF GitHub Bot commented on BEAM-2927:
--

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/4020

[BEAM-2927] Python SDK support for portable side input

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam side-inputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4020.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4020


commit 45c046308e1afeb6d4fcf18a8a3861a12e9933f9
Author: Robert Bradshaw 
Date:   2017-10-19T19:11:44Z

Implement FnApi side inputs in Python.

commit 202dfc74e58e9736bb40ed0382fa03dd072b7ab4
Author: Robert Bradshaw 
Date:   2017-10-19T23:59:34Z

cleanup

commit 198c864aca35b6bca1313e2f216dbb40c73b27e0
Author: Robert Bradshaw 
Date:   2017-10-20T00:02:02Z

Revert unneeded changes.

commit 3dfe862220afbc5e57d7ac674a329cb5e66164cf
Author: Robert Bradshaw 
Date:   2017-10-20T00:20:40Z

lint

commit abc8cd910559a5a8ac40f752d59091e6a2c38cd3
Author: Robert Bradshaw 
Date:   2017-10-20T00:48:59Z

more cleanup




> Python SDK support for portable side input
> --
>
> Key: BEAM-2927
> URL: https://issues.apache.org/jira/browse/BEAM-2927
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Robert Bradshaw
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4020: [BEAM-2927] Python SDK support for portable side in...

2017-10-19 Thread robertwb

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/4020

[BEAM-2927] Python SDK support for portable side input

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam side-inputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4020.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4020


commit 45c046308e1afeb6d4fcf18a8a3861a12e9933f9
Author: Robert Bradshaw 
Date:   2017-10-19T19:11:44Z

Implement FnApi side inputs in Python.

commit 202dfc74e58e9736bb40ed0382fa03dd072b7ab4
Author: Robert Bradshaw 
Date:   2017-10-19T23:59:34Z

cleanup

commit 198c864aca35b6bca1313e2f216dbb40c73b27e0
Author: Robert Bradshaw 
Date:   2017-10-20T00:02:02Z

Revert unneeded changes.

commit 3dfe862220afbc5e57d7ac674a329cb5e66164cf
Author: Robert Bradshaw 
Date:   2017-10-20T00:20:40Z

lint

commit abc8cd910559a5a8ac40f752d59091e6a2c38cd3
Author: Robert Bradshaw 
Date:   2017-10-20T00:48:59Z

more cleanup




---

Build failed in Jenkins: beam_PostCommit_Python_Verify #3393

2017-10-19 Thread Apache Jenkins Server

See 


Changes:

[kenn] Add standalone version of seed job

[kenn] Unit test to repro NPE in PTransformTranslation

[kenn] Do not crash when RawPTransform has null spec

[chamikara] Avoids generating proto files for Windows if grpcio-tools is not

[chamikara] Adding lull tracking for python sampler

--
[...truncated 890.93 KB...]
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/

[jira] [Commented] (BEAM-3013) The Python worker should report lulls

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211896#comment-16211896
 ] 

ASF GitHub Bot commented on BEAM-3013:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3936


> The Python worker should report lulls
> -
>
> Key: BEAM-3013
> URL: https://issues.apache.org/jira/browse/BEAM-3013
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>
> Whenever too much time has been spent on the same state (e.g. > 5 minutes), 
> the worker should report it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3936: [BEAM-3013] Prototyping lull-tracking for Python

2017-10-19 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3936


---

[1/2] beam git commit: Adding lull tracking for python sampler

2017-10-19 Thread chamikara

Repository: beam
Updated Branches:
  refs/heads/master 41f16123b -> 49c392790


Adding lull tracking for python sampler


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/21cdc85c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/21cdc85c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/21cdc85c

Branch: refs/heads/master
Commit: 21cdc85cfa8a06208a7f0a6736cc7d5886d4c8de
Parents: 41f1612
Author: Pablo 
Authored: Thu Oct 19 12:50:46 2017 -0700
Committer: chamik...@google.com 
Committed: Thu Oct 19 16:02:03 2017 -0700

--
 .../apache_beam/runners/worker/statesampler.pyx  | 19 ---
 1 file changed, 16 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/21cdc85c/sdks/python/apache_beam/runners/worker/statesampler.pyx
--
diff --git a/sdks/python/apache_beam/runners/worker/statesampler.pyx 
b/sdks/python/apache_beam/runners/worker/statesampler.pyx
index f0527c6..1e37196 100644
--- a/sdks/python/apache_beam/runners/worker/statesampler.pyx
+++ b/sdks/python/apache_beam/runners/worker/statesampler.pyx
@@ -74,12 +74,16 @@ cdef inline int64_t get_nsec_time() nogil:
 class StateSamplerInfo(object):
   """Info for current state and transition statistics of StateSampler."""
 
-  def __init__(self, state_name, transition_count):
+  def __init__(self, state_name, transition_count, time_since_transition):
 self.state_name = state_name
 self.transition_count = transition_count
+self.time_since_transition = time_since_transition
 
   def __repr__(self):
-return '' % (self.state_name, 
self.transition_count)
+return (''
+% (self.state_name,
+   self.time_since_transition,
+   self.transition_count))
 
 
 # Default period for sampling current state of pipeline execution.
@@ -105,6 +109,7 @@ cdef class StateSampler(object):
   cdef pythread.PyThread_type_lock lock
 
   cdef public int64_t state_transition_count
+  cdef int64_t time_since_transition
 
   cdef int32_t current_state_index
 
@@ -122,6 +127,8 @@ cdef class StateSampler(object):
 self.scoped_states_by_name = {}
 
 self.current_state_index = 0
+self.time_since_transition = 0
+self.state_transition_count = 0
 unknown_state = ScopedState(self, 'unknown', self.current_state_index)
 pythread.PyThread_acquire_lock(self.lock, pythread.WAIT_LOCK)
 self.scoped_states_by_index = [unknown_state]
@@ -142,6 +149,7 @@ cdef class StateSampler(object):
   def run(self):
 cdef int64_t last_nsecs = get_nsec_time()
 cdef int64_t elapsed_nsecs
+cdef int64_t latest_transition_count = self.state_transition_count
 with nogil:
   while True:
 usleep(self.sampling_period_ms * 1000)
@@ -155,6 +163,10 @@ cdef class StateSampler(object):
   nsecs_ptr = &(PyList_GET_ITEM(
   self.scoped_states_by_index, self.current_state_index)).nsecs
   nsecs_ptr[0] += elapsed_nsecs
+  if latest_transition_count != self.state_transition_count:
+self.time_since_transition = 0
+latest_transition_count = self.state_transition_count
+  self.time_since_transition += elapsed_nsecs
   last_nsecs += elapsed_nsecs
 finally:
   pythread.PyThread_release_lock(self.lock)
@@ -182,7 +194,8 @@ cdef class StateSampler(object):
 """Returns StateSamplerInfo with transition statistics."""
 return StateSamplerInfo(
 self.scoped_states_by_index[self.current_state_index].name,
-self.state_transition_count)
+self.state_transition_count,
+self.time_since_transition)
 
   # TODO(pabloem): Make state_name required once all callers migrate,
   #   and the legacy path is removed.

[2/2] beam git commit: This closes #3936

2017-10-19 Thread chamikara

This closes #3936


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/49c39279
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/49c39279
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/49c39279

Branch: refs/heads/master
Commit: 49c3927905d606990ea1d36f1dea0a8c86c11f30
Parents: 41f1612 21cdc85
Author: chamik...@google.com 
Authored: Thu Oct 19 16:02:19 2017 -0700
Committer: chamik...@google.com 
Committed: Thu Oct 19 16:02:19 2017 -0700

--
 .../apache_beam/runners/worker/statesampler.pyx  | 19 ---
 1 file changed, 16 insertions(+), 3 deletions(-)
--

[GitHub] beam pull request #4019: [BEAM-3066] Make the main seed job standalone

2017-10-19 Thread kennknowles

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4019

[BEAM-3066] Make the main seed job standalone

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @jasonkuster 

This is a follow-up to #4004. That PR added a standalone seed job that 
would continue to work even if I screwed up the main seed job. But if I 
understand correctly, the "standalone seed job" does not have permissions to 
create new jobs. I didn't realize this at the time, but either way my plan 
would be mostly unchanged. The standalone job did succeed, so it has the power 
to restore existing jobs if not to create new ones.

This switches the main seed job to have the same contents, leaving the 
standalone job as a backup so we can restore easily.

The end goal is this: we "never" touch the seed job again, and no tweaking 
of other jobs can affect it. Then we can start to improve them safely.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam seed-job-swap

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4019.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4019


commit 987eb53977a9f47efa2ad6fdd58dc9611779c2fc
Author: Kenneth Knowles 
Date:   2017-10-19T22:55:03Z

Make the main seed job standalone




---

[jira] [Commented] (BEAM-3066) Jenkins seed job should be resilient and independent of other jobs' configurations

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211893#comment-16211893
 ] 

ASF GitHub Bot commented on BEAM-3066:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4019

[BEAM-3066] Make the main seed job standalone

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @jasonkuster 

This is a follow-up to #4004. That PR added a standalone seed job that 
would continue to work even if I screwed up the main seed job. But if I 
understand correctly, the "standalone seed job" does not have permissions to 
create new jobs. I didn't realize this at the time, but either way my plan 
would be mostly unchanged. The standalone job did succeed, so it has the power 
to restore existing jobs if not to create new ones.

This switches the main seed job to have the same contents, leaving the 
standalone job as a backup so we can restore easily.

The end goal is this: we "never" touch the seed job again, and no tweaking 
of other jobs can affect it. Then we can start to improve them safely.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam seed-job-swap

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4019.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4019


commit 987eb53977a9f47efa2ad6fdd58dc9611779c2fc
Author: Kenneth Knowles 
Date:   2017-10-19T22:55:03Z

Make the main seed job standalone




> Jenkins seed job should be resilient and independent of other jobs' 
> configurations
> --
>
> Key: BEAM-3066
> URL: https://issues.apache.org/jira/browse/BEAM-3066
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2927) Python SDK support for portable side input

2017-10-19 Thread Robert Bradshaw (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Bradshaw reassigned BEAM-2927:
-

Assignee: Robert Bradshaw  (was: Ahmet Altay)

> Python SDK support for portable side input
> --
>
> Key: BEAM-2927
> URL: https://issues.apache.org/jira/browse/BEAM-2927
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Robert Bradshaw
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3066) Jenkins seed job should be resilient and independent of other jobs' configurations

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211879#comment-16211879
 ] 

ASF GitHub Bot commented on BEAM-3066:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4004


> Jenkins seed job should be resilient and independent of other jobs' 
> configurations
> --
>
> Key: BEAM-3066
> URL: https://issues.apache.org/jira/browse/BEAM-3066
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4004: [BEAM-3066] Add standalone version of seed job

2017-10-19 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4004


---

[1/2] beam git commit: Add standalone version of seed job

2017-10-19 Thread kenn

Repository: beam
Updated Branches:
  refs/heads/master 07b6b0115 -> 41f16123b


Add standalone version of seed job


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/58b33699
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/58b33699
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/58b33699

Branch: refs/heads/master
Commit: 58b33699ef8917ebed8d62d7ec42de6eb651639c
Parents: 1974b92
Author: Kenneth Knowles 
Authored: Mon Oct 16 21:20:52 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Oct 19 11:10:23 2017 -0700

--
 .test-infra/jenkins/job_seed_standalone.groovy | 114 
 1 file changed, 114 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/58b33699/.test-infra/jenkins/job_seed_standalone.groovy
--
diff --git a/.test-infra/jenkins/job_seed_standalone.groovy 
b/.test-infra/jenkins/job_seed_standalone.groovy
new file mode 100644
index 000..badb3c3
--- /dev/null
+++ b/.test-infra/jenkins/job_seed_standalone.groovy
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// Defines the seed job, which creates or updates all other Jenkins projects.
+job('beam_SeedJob_Standalone') {
+  description('Automatically configures all Apache Beam Jenkins projects 
based' +
+  ' on Jenkins DSL groovy files checked into the code repository.')
+
+  properties {
+githubProjectUrl('https://github.com/apache/beam/')
+  }
+
+  // Restrict to only run on Jenkins executors labeled 'beam'
+  label('beam')
+
+  logRotator {
+daysToKeep(14)
+  }
+
+  scm {
+git {
+  remote {
+github('apache/beam')
+
+// ${ghprBuildId} is not interpolated by groovy, but passed through to 
Jenkins where it
+// refers to the environment variable
+refspec(['+refs/head/*:refs/remotes/origin/*',
+ 
'+refs/pull/${ghprPullId}/*:refs/remotes/origin/pr/${ghprPullId}/*']
+.join(' '))
+
+// The variable ${sha1} is not interpolated by groovy, but a parameter 
of the Jenkins job
+branch('${sha1}')
+
+extensions {
+  cleanAfterCheckout()
+}
+  }
+}
+  }
+
+  parameters {
+// Setup for running this job from a pull request
+stringParam(
+'sha1',
+'master',
+'Commit id or refname (eg: origin/pr/4001/head) you want to build 
against.')
+  }
+
+  wrappers {
+timeout {
+  absolute(5)
+  abortBuild()
+}
+  }
+
+  triggers {
+// Run once per day
+cron('0 */5 * * *')
+
+githubPullRequest {
+  admins(['asfbot'])
+  useGitHubHooks()
+  orgWhitelist(['apache'])
+  allowMembersOfWhitelistedOrgsAsAdmin()
+  permitAll()
+
+  // Also run when manually kicked on a pull request
+  triggerPhrase('Run Standalone Seed Job')
+  onlyTriggerPhrase()
+
+  extensions {
+commitStatus {
+  context("Jenkins: Standalone Seed Job")
+}
+
+buildStatus {
+  completedStatus('SUCCESS', '--none--')
+  completedStatus('FAILURE', '--none--')
+  completedStatus('ERROR', '--none--')
+}
+  }
+}
+  }
+
+  // If anything goes wrong, mail the main dev list, because it is a big deal
+  publishers {
+mailer('d...@beam.apache.org', false, true)
+  }
+
+  steps {
+dsl {
+  // A list or a glob of other groovy files to process.
+  external('.test-infra/jenkins/job_*.groovy')
+
+  // If a job is removed from the script, disable it (rather than 
deleting).
+  removeAction('DISABLE')
+}
+  }
+}

[2/2] beam git commit: This closes #4004: Add standalone version of seed job

2017-10-19 Thread kenn

This closes #4004: Add standalone version of seed job


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/41f16123
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/41f16123
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/41f16123

Branch: refs/heads/master
Commit: 41f16123b7a0f2b3b317ce9bbb87c360242332b2
Parents: 07b6b01 58b3369
Author: Kenneth Knowles 
Authored: Thu Oct 19 15:42:07 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Oct 19 15:42:07 2017 -0700

--
 .test-infra/jenkins/job_seed_standalone.groovy | 114 
 1 file changed, 114 insertions(+)
--

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #4197

2017-10-19 Thread Apache Jenkins Server

See 


--
GitHub pull request #4018 of commit 6915ce2518d9db1832a525fea914fad7663da197, 
no merge conflicts.
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

Cloning the remote Git repository
Cloning repository https://github.com/apache/beam.git
 > git init 
 > 
 >  # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/4018/*:refs/remotes/origin/pr/4018/*
 > git rev-parse refs/remotes/origin/pr/4018/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/4018/merge^{commit} # timeout=10
 > git rev-parse origin/pr/4018/merge^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/4018/*:refs/remotes/origin/pr/4018/*
 > git rev-parse refs/remotes/origin/pr/4018/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/4018/merge^{commit} # timeout=10
 > git rev-parse origin/pr/4018/merge^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/4018/*:refs/remotes/origin/pr/4018/*
 > git rev-parse refs/remotes/origin/pr/4018/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/4018/merge^{commit} # timeout=10
 > git rev-parse origin/pr/4018/merge^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch 
configuration for this job.
Not sending mail to unregistered user valen...@google.com

[jira] [Commented] (BEAM-3065) Proto generation installation is broken for Windows

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211867#comment-16211867
 ] 

ASF GitHub Bot commented on BEAM-3065:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4007


> Proto generation installation is broken for Windows
> ---
>
> Key: BEAM-3065
> URL: https://issues.apache.org/jira/browse/BEAM-3065
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Chamikara Jayalath
>Assignee: Robert Bradshaw
>
> "python setup.py sdist" and "python setup.py install" raises following error 
> for Windows. Failure happens when running gen_protos.py. Robert, seems like 
> you added this script. Can you take a look ?
> C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\gen_protos.py:48:
>  UserWarning: Installing grpcio-tools is recommended for development.
>   warnings.warn('Installing grpcio-tools is recommended for development.')
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "c:\python27\Lib\multiprocessing\forking.py", line 380, in main
> prepare(preparation_data)
>   File "c:\python27\Lib\multiprocessing\forking.py", line 510, in prepare
> '__parents_main__', file, path_name, etc
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\setup.py",
>  line 203, in 
> 'test': generate_protos_first(test),
>   File "c:\python27\Lib\distutils\core.py", line 151, in setup
> dist.run_commands()
>   File "c:\python27\Lib\distutils\dist.py", line 953, in run_commands
> self.run_command(cmd)
>   File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
> cmd_obj.run()
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install.py",
>  line 67, in run
> self.do_egg_install()
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install.py",
>  line 109, in do_egg_install
> self.run_command('bdist_egg')
>   File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
> self.distribution.run_command(command)
>   File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
> cmd_obj.run()
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\bdist_egg.py",
>  line 169, in run
> cmd = self.call_command('install_lib', warn_dir=0)
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\bdist_egg.py",
>  line 155, in call_command
> self.run_command(cmdname)
>   File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
> self.distribution.run_command(command)
>   File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
> cmd_obj.run()
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\env1\lib\site-packages\setuptools\command\install_lib.py",
>  line 11, in run
> self.build()
>   File "c:\python27\Lib\distutils\command\install_lib.py", line 109, in build
> self.run_command('build_py')
>   File "c:\python27\Lib\distutils\cmd.py", line 326, in run_command
> self.distribution.run_command(command)
>   File "c:\python27\Lib\distutils\dist.py", line 972, in run_command
> cmd_obj.run()
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\setup.py",
>  line 143, in run
> gen_protos.generate_proto_files()
>   File 
> "C:\Users\chamikara\beam_testing\windows_test_10_16_2017\beam\sdks\python\gen_protos.py",
>  line 84, in generate_proto_files
> p.start()
>   File "c:\python27\Lib\multiprocessing\process.py", line 130, in start
> self._popen = Popen(self)
>   File "c:\python27\Lib\multiprocessing\forking.py", line 258, in __init__
> cmd = get_command_line() + [rhandle]
>   File "c:\python27\Lib\multiprocessing\forking.py", line 358, in 
> get_command_line
> is not going to be frozen to produce a Windows executable.''')
> RuntimeError:
> Attempt to start a new process before the current process
> has finished its bootstrapping phase.
> This probably means that you are on Windows and you have
> forgotten to use the proper idiom in the main module:
> if __name__ == '__main__':
> freeze_support()
> ...
> The "freeze_support()" line can be omitted if the program
> is not going to be frozen to produce a Windows executable.
> Traceback (most recent call last):
>   File "setup.py", line 203, in 
> 'test': generate_protos_first(test),
>   File "c:\python27\Lib\distutils\core.py", line 151, in setup
> dist.run_commands()
>

[GitHub] beam pull request #4007: [BEAM-3065] Avoids generating proto files for Windo...

2017-10-19 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4007


---

[2/2] beam git commit: This closes #4007

2017-10-19 Thread chamikara

This closes #4007


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/07b6b011
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/07b6b011
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/07b6b011

Branch: refs/heads/master
Commit: 07b6b0115d6c4c07aadb40a099adbbb876cb8ea1
Parents: 1039f5b f472439
Author: chamik...@google.com 
Authored: Thu Oct 19 15:29:28 2017 -0700
Committer: chamik...@google.com 
Committed: Thu Oct 19 15:29:28 2017 -0700

--
 sdks/python/gen_protos.py | 8 
 1 file changed, 8 insertions(+)
--

[1/2] beam git commit: Avoids generating proto files for Windows if grpcio-tools is not installed.

2017-10-19 Thread chamikara

Repository: beam
Updated Branches:
  refs/heads/master 1039f5b96 -> 07b6b0115


Avoids generating proto files for Windows if grpcio-tools is not installed.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f4724397
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f4724397
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f4724397

Branch: refs/heads/master
Commit: f4724397c54b31fe67517f92cab2b238aed8d527
Parents: 1039f5b
Author: chamik...@google.com 
Authored: Tue Oct 17 18:46:40 2017 -0700
Committer: chamik...@google.com 
Committed: Thu Oct 19 15:29:16 2017 -0700

--
 sdks/python/gen_protos.py | 8 
 1 file changed, 8 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f4724397/sdks/python/gen_protos.py
--
diff --git a/sdks/python/gen_protos.py b/sdks/python/gen_protos.py
index 59d6cb7..c7bf55f 100644
--- a/sdks/python/gen_protos.py
+++ b/sdks/python/gen_protos.py
@@ -22,6 +22,7 @@ import logging
 import multiprocessing
 import os
 import pkg_resources
+import platform
 import shutil
 import subprocess
 import sys
@@ -76,6 +77,13 @@ def generate_proto_files(force=False):
 try:
   from grpc_tools import protoc
 except ImportError:
+  if platform.system() == 'Windows':
+# For Windows, grpcio-tools has to be installed manually.
+raise RuntimeError(
+'Cannot generate protos for Windows since grpcio-tools package is '
+'not installed. Please install this package manually '
+'using \'pip install grpcio-tools\'.')
+
   # Use a subprocess to avoid messing with this process' path and imports.
   # Note that this requires a separate module from setup.py for Windows:
   # https://docs.python.org/2/library/multiprocessing.html#windows

[3/3] beam git commit: This closes #4018: Do not crash when RawPTransform has null spec

2017-10-19 Thread kenn

This closes #4018: Do not crash when RawPTransform has null spec


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1039f5b9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1039f5b9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1039f5b9

Branch: refs/heads/master
Commit: 1039f5b9682fa6aa5fba256110c63caf4d0da41f
Parents: 0f096b1 d879d27
Author: Kenneth Knowles 
Authored: Thu Oct 19 14:28:46 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Oct 19 14:28:46 2017 -0700

--
 .../construction/PTransformTranslation.java |  8 -
 .../construction/PTransformTranslationTest.java | 35 ++--
 2 files changed, 40 insertions(+), 3 deletions(-)
--

[1/3] beam git commit: Unit test to repro NPE in PTransformTranslation

2017-10-19 Thread kenn

Repository: beam
Updated Branches:
  refs/heads/master 0f096b12e -> 1039f5b96


Unit test to repro NPE in PTransformTranslation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/acbbf1f5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/acbbf1f5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/acbbf1f5

Branch: refs/heads/master
Commit: acbbf1f5f41a6afc73fe129882de4474db58df2a
Parents: 3360b1f
Author: Kenneth Knowles 
Authored: Thu Oct 19 13:49:31 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Oct 19 13:49:42 2017 -0700

--
 .../construction/PTransformTranslationTest.java | 35 ++--
 1 file changed, 33 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/acbbf1f5/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformTranslationTest.java
--
diff --git 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformTranslationTest.java
 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformTranslationTest.java
index 2ec96e8..36f912c 100644
--- 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformTranslationTest.java
+++ 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PTransformTranslationTest.java
@@ -31,9 +31,9 @@ import java.util.Collections;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import javax.annotation.Nullable;
 import org.apache.beam.model.pipeline.v1.RunnerApi;
 import org.apache.beam.model.pipeline.v1.RunnerApi.Components;
-import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.io.CountingSource;
 import org.apache.beam.sdk.io.GenerateSequence;
@@ -42,6 +42,7 @@ import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.values.KV;
@@ -49,6 +50,7 @@ import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionTuple;
 import org.apache.beam.sdk.values.PCollectionView;
+import org.apache.beam.sdk.values.PDone;
 import org.apache.beam.sdk.values.PValue;
 import org.apache.beam.sdk.values.TupleTag;
 import org.apache.beam.sdk.values.TupleTagList;
@@ -69,17 +71,24 @@ public class PTransformTranslationTest {
 // This pipeline exists for construction, not to run any test.
 // TODO: Leaf node with understood payload - i.e. validate payloads
 ToAndFromProtoSpec readLeaf = 
ToAndFromProtoSpec.leaf(read(TestPipeline.create()));
+
 ToAndFromProtoSpec readMultipleInAndOut =
 ToAndFromProtoSpec.leaf(multiMultiParDo(TestPipeline.create()));
+
 TestPipeline compositeReadPipeline = TestPipeline.create();
 ToAndFromProtoSpec compositeRead =
 ToAndFromProtoSpec.composite(
 generateSequence(compositeReadPipeline),
 ToAndFromProtoSpec.leaf(read(compositeReadPipeline)));
+
+ToAndFromProtoSpec rawLeafNullSpec =
+
ToAndFromProtoSpec.leaf(rawPTransformWithNullSpec(TestPipeline.create()));
+
 return ImmutableList.builder()
 .add(readLeaf)
 .add(readMultipleInAndOut)
 .add(compositeRead)
+.add(rawLeafNullSpec)
 // TODO: Composite with multiple children
 // TODO: Composite with a composite child
 .build();
@@ -139,7 +148,7 @@ public class PTransformTranslationTest {
   // Sanity call
   components.getExistingPTransformId(child.getTransform());
 }
-PTransform convert = PTransformTranslation
+RunnerApi.PTransform convert = PTransformTranslation
 .toProto(spec.getTransform(), childTransforms, components);
 // Make sure the converted transform is registered. Convert it 
independently, but if this is a
 // child spec, the child must be in the components.
@@ -166,6 +175,28 @@ public class PTransformTranslationTest {
 "ReadTheCount", pipeline.begin().expand(), pcollection.expand(), 
transform, pipeline);
   }
 
+  private static AppliedPTransform rawPTransformWithNullSpec(Pipeline 
pipeline) {
+PTransformTranslation.RawPTransform rawPTransform =
+new PTransformTranslation.RawPTransform() {
+  @Override
+  public String getUrn() {
+return "fake/urn";
+  }
+
+  @Nullable
+  @

[GitHub] beam pull request #4018: Do not crash when RawPTransform has null spec

2017-10-19 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4018


---

[2/3] beam git commit: Do not crash when RawPTransform has null spec

2017-10-19 Thread kenn

Do not crash when RawPTransform has null spec


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d879d279
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d879d279
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d879d279

Branch: refs/heads/master
Commit: d879d2791e51dee5baddbdbe51f757e44b55a109
Parents: acbbf1f
Author: Kenneth Knowles 
Authored: Thu Oct 19 12:45:43 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Oct 19 14:28:05 2017 -0700

--
 .../runners/core/construction/PTransformTranslation.java | 8 +++-
 1 file changed, 7 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d879d279/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
index 8e6829b..a3a5a1f 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
@@ -179,7 +179,13 @@ public class PTransformTranslation {
 if (transform instanceof RawPTransform) {
   // The raw transform was parsed in the context of other components; this 
puts it in the
   // context of our current serialization
-  transformBuilder.setSpec(((RawPTransform) 
transform).migrate(components));
+  FunctionSpec spec = ((RawPTransform) 
transform).migrate(components);
+
+  // A composite transform is permitted to have a null spec. There are 
also some pseudo-
+  // primitives not yet supported by the portability framework that have 
null specs
+  if (spec != null) {
+transformBuilder.setSpec(spec);
+  }
 } else if (KNOWN_PAYLOAD_TRANSLATORS.containsKey(transform.getClass())) {
   transformBuilder.setSpec(
   KNOWN_PAYLOAD_TRANSLATORS

[jira] [Commented] (BEAM-2933) Gearpump support for portable side input

2017-10-19 Thread Luke Cwik (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211810#comment-16211810
 ] 

Luke Cwik commented on BEAM-2933:
-

https://github.com/apache/beam/pull/4011 migrated to using a multimap as the 
materialization format for side inputs.

The migration used a trivial multimap -> iterable -> multimap conversion within 
the Gearpump runner for first pass implementation purposes. Note that this is 
no different then the current materialization from a performance perspective it 
just moves this logic within the purview of the runner. Consider improving the 
internal materialization format from this trivial and unoptimal mapping.

> Gearpump support for portable side input
> 
>
> Key: BEAM-2933
> URL: https://issues.apache.org/jira/browse/BEAM-2933
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-gearpump
>Reporter: Henning Rohde
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-2930) Flink support for portable side input

2017-10-19 Thread Luke Cwik (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211805#comment-16211805
 ] 

Luke Cwik edited comment on BEAM-2930 at 10/19/17 9:54 PM:
---

https://github.com/apache/beam/pull/4011 migrated to using a multimap as the 
materialization format for side inputs.

The migration used a trivial multimap -> iterable -> multimap conversion within 
the Flink runner for first pass implementation purposes. Note that this is no 
different then the current materialization from a performance perspective it 
just moves this logic within the purview of the runner. Consider improving the 
internal materialization format from this trivial and unoptimal mapping.


was (Author: lcwik):
https://github.com/apache/beam/pull/4011 migrated to using a multimap as the 
materialization format for side inputs.

The migration used a trivial multimap -> iterable -> multimap conversion within 
the Flink runner for first pass implementation purposes. Note that this is no 
different then the current materialization from a performance perspective it 
just moves this logic within the purview of the runner.

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2932) Spark support for portable side input

2017-10-19 Thread Luke Cwik (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211808#comment-16211808
 ] 

Luke Cwik commented on BEAM-2932:
-

https://github.com/apache/beam/pull/4011 migrated to using a multimap as the 
materialization format for side inputs.

The migration used a trivial multimap -> iterable -> multimap conversion within 
the Spark runner for first pass implementation purposes. Note that this is no 
different then the current materialization from a performance perspective it 
just moves this logic within the purview of the runner. Consider improving the 
internal materialization format from this trivial and unoptimal mapping.

> Spark support for portable side input
> -
>
> Key: BEAM-2932
> URL: https://issues.apache.org/jira/browse/BEAM-2932
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Henning Rohde
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-2931) Apex support for portable side input

2017-10-19 Thread Luke Cwik (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211806#comment-16211806
 ] 

Luke Cwik edited comment on BEAM-2931 at 10/19/17 9:54 PM:
---

https://github.com/apache/beam/pull/4011 migrated to using a multimap as the 
materialization format for side inputs.

The migration used a trivial multimap -> iterable -> multimap conversion within 
the Apex runner for first pass implementation purposes. Note that this is no 
different then the current materialization from a performance perspective it 
just moves this logic within the purview of the runner. Consider improving the 
internal materialization format from this trivial and unoptimal mapping.


was (Author: lcwik):
https://github.com/apache/beam/pull/4011 migrated to using a multimap as the 
materialization format for side inputs.

The migration used a trivial multimap -> iterable -> multimap conversion within 
the Apex runner for first pass implementation purposes. Note that this is no 
different then the current materialization from a performance perspective it 
just moves this logic within the purview of the runner.

> Apex support for portable side input
> 
>
> Key: BEAM-2931
> URL: https://issues.apache.org/jira/browse/BEAM-2931
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-apex
>Reporter: Henning Rohde
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2931) Apex support for portable side input

2017-10-19 Thread Luke Cwik (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211806#comment-16211806
 ] 

Luke Cwik commented on BEAM-2931:
-

https://github.com/apache/beam/pull/4011 migrated to using a multimap as the 
materialization format for side inputs.

The migration used a trivial multimap -> iterable -> multimap conversion within 
the Apex runner for first pass implementation purposes. Note that this is no 
different then the current materialization from a performance perspective it 
just moves this logic within the purview of the runner.

> Apex support for portable side input
> 
>
> Key: BEAM-2931
> URL: https://issues.apache.org/jira/browse/BEAM-2931
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-apex
>Reporter: Henning Rohde
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2930) Flink support for portable side input

2017-10-19 Thread Luke Cwik (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211805#comment-16211805
 ] 

Luke Cwik commented on BEAM-2930:
-

https://github.com/apache/beam/pull/4011 migrated to using a multimap as the 
materialization format for side inputs.

The migration used a trivial multimap -> iterable -> multimap conversion within 
the Flink runner for first pass implementation purposes. Note that this is no 
different then the current materialization from a performance perspective it 
just moves this logic within the purview of the runner.

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>  Labels: portability
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3080) Improve the side input materialization for the DirectRunner/ULR from iterable to storing the multimap directly

2017-10-19 Thread Luke Cwik (JIRA)

Luke Cwik created BEAM-3080:
---

 Summary: Improve the side input materialization for the 
DirectRunner/ULR from iterable to storing the multimap directly
 Key: BEAM-3080
 URL: https://issues.apache.org/jira/browse/BEAM-3080
 Project: Beam
  Issue Type: Sub-task
  Components: runner-direct
Reporter: Luke Cwik
Priority: Minor


https://github.com/apache/beam/pull/4011 migrated to using a multimap as the 
materialization format for side inputs.

The migration used a trivial multimap -> iterable -> multimap conversion within 
the DirectRunner for first pass implementation purposes. Note that this is no 
different then the current materialization from a performance perspective it 
just moves this logic within the purview of the runner.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3392

2017-10-19 Thread Apache Jenkins Server

See 


Changes:

[altay] Created Java snippets file

--
[...truncated 888.53 KB...]
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
co

[beam] Git Push Summary

2017-10-19 Thread kenn

Repository: beam
Updated Branches:
  refs/heads/samza-runner [created] 0f096b12e

[jira] [Updated] (BEAM-3079) Samza runner

2017-10-19 Thread Kenneth Knowles (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-3079:
--
Issue Type: Wish  (was: Bug)

> Samza runner
> 
>
> Key: BEAM-3079
> URL: https://issues.apache.org/jira/browse/BEAM-3079
> Project: Beam
>  Issue Type: Wish
>  Components: runner-ideas
>Reporter: Xinyu Liu
>Assignee: Kenneth Knowles
>
> Apache Samza is a distributed data-processing platform which supports both 
> stream and batch processing. It'll be awesome if we can run BEAM's advanced 
> data transform and multi-language sdks on top of Samza.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-3079) Samza runner

2017-10-19 Thread Xinyu Liu (JIRA)

Xinyu Liu created BEAM-3079:
---

 Summary: Samza runner
 Key: BEAM-3079
 URL: https://issues.apache.org/jira/browse/BEAM-3079
 Project: Beam
  Issue Type: Bug
  Components: runner-ideas
Reporter: Xinyu Liu
Assignee: Kenneth Knowles


Apache Samza is a distributed data-processing platform which supports both 
stream and batch processing. It'll be awesome if we can run BEAM's advanced 
data transform and multi-language sdks on top of Samza.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #5071

2017-10-19 Thread Apache Jenkins Server

See 


--
[...truncated 1.17 MB...]
2017-10-19T20:09:44.205 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/razorvine/pyrolite/4.9/pyrolite-4.9.jar
2017-10-19T20:09:44.258 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/razorvine/pyrolite/4.9/pyrolite-4.9.jar
 (92 KB at 8.6 KB/sec)
2017-10-19T20:09:44.258 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/sf/py4j/py4j/0.9/py4j-0.9.jar
2017-10-19T20:09:44.303 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/spark/spark-core_2.10/1.6.3/spark-core_2.10-1.6.3.jar
 (10968 KB at 1029.0 KB/sec)
2017-10-19T20:09:44.303 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar
2017-10-19T20:09:44.307 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/sf/py4j/py4j/0.9/py4j-0.9.jar (83 KB 
at 7.7 KB/sec)
2017-10-19T20:09:44.308 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/spark/spark-sql_2.10/2.0.0/spark-sql_2.10-2.0.0.jar
2017-10-19T20:09:44.335 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar
 (3 KB at 0.3 KB/sec)
2017-10-19T20:09:44.335 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/univocity/univocity-parsers/2.1.1/univocity-parsers-2.1.1.jar
2017-10-19T20:09:44.384 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-underfs-s3/0.8.2/tachyon-underfs-s3-0.8.2.jar
 (494 KB at 46.0 KB/sec)
2017-10-19T20:09:44.384 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/spark/spark-sketch_2.10/2.0.0/spark-sketch_2.10-2.0.0.jar
2017-10-19T20:09:44.421 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/spark/spark-sketch_2.10/2.0.0/spark-sketch_2.10-2.0.0.jar
 (29 KB at 2.7 KB/sec)
2017-10-19T20:09:44.421 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/spark/spark-catalyst_2.10/2.0.0/spark-catalyst_2.10-2.0.0.jar
2017-10-19T20:09:44.447 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/univocity/univocity-parsers/2.1.1/univocity-parsers-2.1.1.jar
 (284 KB at 26.3 KB/sec)
2017-10-19T20:09:44.448 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/janino/janino/2.7.8/janino-2.7.8.jar
2017-10-19T20:09:44.475 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/tachyonproject/tachyon-client/0.8.2/tachyon-client-0.8.2.jar
 (2238 KB at 206.6 KB/sec)
2017-10-19T20:09:44.475 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/spark/spark-tags_2.10/2.0.0/spark-tags_2.10-2.0.0.jar
2017-10-19T20:09:44.511 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/spark/spark-tags_2.10/2.0.0/spark-tags_2.10-2.0.0.jar
 (15 KB at 1.4 KB/sec)
2017-10-19T20:09:44.511 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scalatest/scalatest_2.10/2.2.6/scalatest_2.10-2.2.6.jar
2017-10-19T20:09:44.700 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/janino/janino/2.7.8/janino-2.7.8.jar
 (599 KB at 54.2 KB/sec)
2017-10-19T20:09:44.700 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/parquet/parquet-column/1.7.0/parquet-column-1.7.0.jar
2017-10-19T20:09:45.082 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/parquet/parquet-column/1.7.0/parquet-column-1.7.0.jar
 (896 KB at 78.3 KB/sec)
2017-10-19T20:09:45.082 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/parquet/parquet-common/1.7.0/parquet-common-1.7.0.jar
2017-10-19T20:09:45.111 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/parquet/parquet-common/1.7.0/parquet-common-1.7.0.jar
 (22 KB at 1.8 KB/sec)
2017-10-19T20:09:45.111 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/parquet/parquet-encoding/1.7.0/parquet-encoding-1.7.0.jar
2017-10-19T20:09:45.223 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/parquet/parquet-encoding/1.7.0/parquet-encoding-1.7.0.jar
 (279 KB at 24.1 KB/sec)
2017-10-19T20:09:45.224 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/parquet/parquet-generator/1.7.0/parquet-generator-1.7.0.jar
2017-10-19T20:09:45.259 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/parquet/parquet-generator/1.7.0/parquet-generator-1.7.0.jar
 (21 KB at 1.8 KB/sec)
2017-10-19T20:09:45.260 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/parquet/parquet-hadoop/1.7.0/parquet-hadoop-1.7.0.jar
2017-10-19T20:09:45.360 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/parquet/parquet-hadoop/1.7.0/parquet-hadoop-1.7.0.jar
 (205 KB at 17.5 KB/sec)
2017-10-19T20:09:45.360 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/parquet/parquet-format/2.3.0-incubating/parquet-format-2.3.0-incubating.jar
2017-10-19T20:09:45.547

[GitHub] beam pull request #4018: Do not crash when RawPTransform has null spec

2017-10-19 Thread kennknowles

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4018

Do not crash when RawPTransform has null spec

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh this is the right short-term fix
CC: @jkff 

Superficial issue: misuse of generated proto code. I've confirmed this fix 
eliminates the NPE currently happening on `SplittableDoFnTest` in the Dataflow 
postcommit.

Larger issue: we stage a portable pipeline and the Dataflow job description 
has (or will have) pointers into that portable pipeline for the runner harness 
to extract portable payloads. So setting up `ProcessKeyedElements` to have a 
`null` payload or to be `NotSerializable` is incorrect. Instead, I think it has 
to be a standardized transform with a portable URN/payload specification. That 
will be a prerequisite to supporting SDF portably, unless we have a different 
implementation strategy.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam SplittableParDo-portability

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4018.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4018


commit 8c96011578ccc12c7f78488afdb7ba6041984037
Author: Kenneth Knowles 
Date:   2017-10-19T19:45:43Z

Do not crash when RawPTransform has null spec




---

[GitHub] beam pull request #4017: Java DataflowRunner: snapshot pipeline before apply...

2017-10-19 Thread kennknowles

Github user kennknowles closed the pull request at:

https://github.com/apache/beam/pull/4017


---

[GitHub] beam pull request #4017: Java DataflowRunner: snapshot pipeline before apply...

2017-10-19 Thread kennknowles

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/4017

Java DataflowRunner: snapshot pipeline before applying overrides

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @tgroh I think you'll see what I did wrong here...


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam SplittableDoFnTest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4017.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4017


commit 3e85056b43b024186589a47064bff95c2123ce80
Author: Kenneth Knowles 
Date:   2017-10-19T19:31:34Z

Java DataflowRunner: snapshot pipeline before applying overrides




---

[jira] [Assigned] (BEAM-2566) Java SDK harness should not depend on any runner

2017-10-19 Thread Kenneth Knowles (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-2566:
-

Assignee: Luke Cwik  (was: Thomas Groh)

> Java SDK harness should not depend on any runner
> 
>
> Key: BEAM-2566
> URL: https://issues.apache.org/jira/browse/BEAM-2566
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Luke Cwik
>  Labels: portability
>
> Right now there is a dependency on the Dataflow runner. I believe this is 
> legacy due to using {{CloudObject}} temporarily but I do not claim to 
> understand the full nature of the dependency.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3900: Created Java snippets file

2017-10-19 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3900


---

[1/2] beam git commit: Created Java snippets file

2017-10-19 Thread altay

Repository: beam
Updated Branches:
  refs/heads/master 3360b1f68 -> 0f096b12e


Created Java snippets file


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9f1db3f1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9f1db3f1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9f1db3f1

Branch: refs/heads/master
Commit: 9f1db3f11680c08407bc8dd0101f8b047c090620
Parents: 3360b1f
Author: David Cavazos 
Authored: Tue Sep 26 10:38:38 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Oct 19 12:10:45 2017 -0700

--
 .../examples/website_snippets/Snippets.java |  87 ++
 .../examples/website_snippets/SnippetsTest.java | 114 +++
 .../apache_beam/examples/snippets/snippets.py   |  45 +++-
 .../examples/snippets/snippets_test.py  |  38 ---
 4 files changed, 239 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/9f1db3f1/examples/java8/src/main/java/org/apache/beam/examples/website_snippets/Snippets.java
--
diff --git 
a/examples/java8/src/main/java/org/apache/beam/examples/website_snippets/Snippets.java
 
b/examples/java8/src/main/java/org/apache/beam/examples/website_snippets/Snippets.java
new file mode 100644
index 000..f17171e
--- /dev/null
+++ 
b/examples/java8/src/main/java/org/apache/beam/examples/website_snippets/Snippets.java
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.examples;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.join.CoGbkResult;
+import org.apache.beam.sdk.transforms.join.CoGroupByKey;
+import org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TupleTag;
+
+/**
+ * Code snippets used in webdocs.
+ */
+public class Snippets {
+
+  /* Helper function to format results in coGroupByKeyTuple */
+  public static String formatCoGbkResults(
+  String name, Iterable emails, Iterable phones) {
+
+List emailsList = new ArrayList<>();
+for (String elem : emails) {
+  emailsList.add("'" + elem + "'");
+}
+Collections.sort(emailsList);
+String emailsStr = "[" + String.join(", ", emailsList) + "]";
+
+List phonesList = new ArrayList<>();
+for (String elem : phones) {
+  phonesList.add("'" + elem + "'");
+}
+Collections.sort(phonesList);
+String phonesStr = "[" + String.join(", ", phonesList) + "]";
+
+return name + "; " + emailsStr + "; " + phonesStr;
+  }
+
+  public static PCollection coGroupByKeyTuple(
+  TupleTag emailsTag,
+  TupleTag phonesTag,
+  PCollection> emails,
+  PCollection> phones) {
+
+// [START CoGroupByKeyTuple]
+PCollection> results =
+KeyedPCollectionTuple
+.of(emailsTag, emails)
+.and(phonesTag, phones)
+.apply(CoGroupByKey.create());
+
+PCollection contactLines = results.apply(ParDo.of(
+  new DoFn, String>() {
+@ProcessElement
+public void processElement(ProcessContext c) {
+  KV e = c.element();
+  String name = e.getKey();
+  Iterable emailsIter = e.getValue().getAll(emailsTag);
+  Iterable phonesIter = e.getValue().getAll(phonesTag);
+  String formattedResult = Snippets.formatCoGbkResults(name, 
emailsIter, phonesIter);
+  c.output(formattedResult);
+}
+  }
+));
+// [END CoGroupByKeyTuple]
+return contactLines;
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/9f1db3f1/examples/java8/src/test/java/org/apache/beam/examples/website_snippets/SnippetsTest.java
--
diff --git 
a/examples/java8/src/test/java/org/apache/beam/examples/website_snippets/Snippet

[2/2] beam git commit: This closes #3900

2017-10-19 Thread altay

This closes #3900


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0f096b12
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0f096b12
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0f096b12

Branch: refs/heads/master
Commit: 0f096b12e0df7ae6034d5ba1234a80837b635b30
Parents: 3360b1f 9f1db3f
Author: Ahmet Altay 
Authored: Thu Oct 19 12:10:49 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Oct 19 12:10:49 2017 -0700

--
 .../examples/website_snippets/Snippets.java |  87 ++
 .../examples/website_snippets/SnippetsTest.java | 114 +++
 .../apache_beam/examples/snippets/snippets.py   |  45 +++-
 .../examples/snippets/snippets_test.py  |  38 ---
 4 files changed, 239 insertions(+), 45 deletions(-)
--

[jira] [Commented] (BEAM-2566) Java SDK harness should not depend on any runner

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211556#comment-16211556
 ] 

ASF GitHub Bot commented on BEAM-2566:
--

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/4016

[BEAM-2566] Decouple SDK harness from Dataflow runner by elevating 
experiments and SDK harness configuration to java-sdk-core.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
I'm entertaining name changes on the options interfaces or any suggestions 
on how to make it easier to migrate from DataflowWorkerLoggingOptions to 
SdkHarnessOptions for logging configuration.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam2566

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4016.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4016


commit 8b524c64e203c26ddab05d8adb4f86711fecfadc
Author: Luke Cwik 
Date:   2017-10-19T18:26:04Z

[BEAM-2566] Decouple SDK harness from Dataflow runner by elevating 
experiments and SDK harness configuration to java-sdk-core.




> Java SDK harness should not depend on any runner
> 
>
> Key: BEAM-2566
> URL: https://issues.apache.org/jira/browse/BEAM-2566
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>  Labels: portability
>
> Right now there is a dependency on the Dataflow runner. I believe this is 
> legacy due to using {{CloudObject}} temporarily but I do not claim to 
> understand the full nature of the dependency.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4016: [BEAM-2566] Decouple SDK harness from Dataflow runn...

2017-10-19 Thread lukecwik

GitHub user lukecwik opened a pull request:

https://github.com/apache/beam/pull/4016

[BEAM-2566] Decouple SDK harness from Dataflow runner by elevating 
experiments and SDK harness configuration to java-sdk-core.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
I'm entertaining name changes on the options interfaces or any suggestions 
on how to make it easier to migrate from DataflowWorkerLoggingOptions to 
SdkHarnessOptions for logging configuration.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam2566

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/4016.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4016


commit 8b524c64e203c26ddab05d8adb4f86711fecfadc
Author: Luke Cwik 
Date:   2017-10-19T18:26:04Z

[BEAM-2566] Decouple SDK harness from Dataflow runner by elevating 
experiments and SDK harness configuration to java-sdk-core.




---

Build failed in Jenkins: beam_PerformanceTests_Python #467

2017-10-19 Thread Apache Jenkins Server

See 


Changes:

[valentyn] Pin runner harness also for official BEAM releases.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 3360b1f68e8ad3cb63fbb65c94ac3ed06f9b3ee9 (origin/master)
Commit message: "This closes #4012"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 3360b1f68e8ad3cb63fbb65c94ac3ed06f9b3ee9
 > git rev-list 1974b920e4b3bbe8549e25fe789f9dada13c1769 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3026968475203325506.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7113176142870879777.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins557549793118684602.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins84117455060170455.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.

[jira] [Created] (BEAM-3078) Source split operations do not support metrics reported correctly

2017-10-19 Thread Pablo Estrada (JIRA)

Pablo Estrada created BEAM-3078:
---

 Summary: Source split operations do not support metrics reported 
correctly
 Key: BEAM-3078
 URL: https://issues.apache.org/jira/browse/BEAM-3078
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Reporter: Pablo Estrada
Assignee: Pablo Estrada






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3391

2017-10-19 Thread Apache Jenkins Server

See 


Changes:

[valentyn] Pin runner harness also for official BEAM releases.

--
[...truncated 885.41 KB...]
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.3.0.dev0/a

[jira] [Commented] (BEAM-3011) Pin Runner harness container image in Python SDK

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211313#comment-16211313
 ] 

ASF GitHub Bot commented on BEAM-3011:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4012


> Pin Runner harness container image in Python SDK
> 
>
> Key: BEAM-3011
> URL: https://issues.apache.org/jira/browse/BEAM-3011
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Blocker
>  Labels: portability
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4012: Pin runner harness also for official BEAM releases.

2017-10-19 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/4012


---

[2/2] beam git commit: This closes #4012

2017-10-19 Thread altay

This closes #4012


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3360b1f6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3360b1f6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3360b1f6

Branch: refs/heads/master
Commit: 3360b1f68e8ad3cb63fbb65c94ac3ed06f9b3ee9
Parents: 1974b92 7155931
Author: Ahmet Altay 
Authored: Thu Oct 19 09:31:53 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Oct 19 09:31:53 2017 -0700

--
 .../runners/dataflow/internal/apiclient_test.py | 30 +---
 .../runners/dataflow/internal/dependency.py |  7 +
 2 files changed, 33 insertions(+), 4 deletions(-)
--

[1/2] beam git commit: Pin runner harness also for official BEAM releases.

2017-10-19 Thread altay

Repository: beam
Updated Branches:
  refs/heads/master 1974b920e -> 3360b1f68


Pin runner harness also for official BEAM releases.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7155931f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7155931f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7155931f

Branch: refs/heads/master
Commit: 7155931ff9eaf5fb85765e9d515469f5e6bd5bf9
Parents: 1974b92
Author: Valentyn Tymofieiev 
Authored: Wed Oct 18 14:25:33 2017 -0700
Committer: Valentyn Tymofieiev 
Committed: Wed Oct 18 14:39:44 2017 -0700

--
 .../runners/dataflow/internal/apiclient_test.py | 30 +---
 .../runners/dataflow/internal/dependency.py |  7 +
 2 files changed, 33 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7155931f/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
--
diff --git 
a/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py 
b/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
index fa4f89a..ecd6003 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py
@@ -156,10 +156,14 @@ class UtilTest(unittest.TestCase):
 'apache_beam.runners.dataflow.internal.dependency.pkg_resources'
 '.get_distribution',
 mock.MagicMock(return_value=distribution)):
-  env = apiclient.Environment([], pipeline_options, '2.2.0')
+  env = apiclient.Environment([], #packages
+  pipeline_options,
+  '2.0.0') #any environment version
   self.assertIn(override, env.proto.experiments)
 
-  def test_harness_override_absent_in_unreleased_sdk(self):
+  @mock.patch('apache_beam.runners.dataflow.internal.dependency.'
+  'beam_version.__version__', '2.2.0')
+  def test_harness_override_present_in_beam_releases(self):
 pipeline_options = PipelineOptions(
 ['--temp_location', 'gs://any-location/temp', '--streaming'])
 override = ''.join(
@@ -170,8 +174,26 @@ class UtilTest(unittest.TestCase):
 'apache_beam.runners.dataflow.internal.dependency.pkg_resources'
 '.get_distribution',
 mock.Mock(side_effect=pkg_resources.DistributionNotFound())):
-  env = apiclient.Environment([], pipeline_options, '2.2.0')
-  self.assertNotIn(override, env.proto.experiments)
+  env = apiclient.Environment([], #packages
+  pipeline_options,
+  '2.0.0') #any environment version
+  self.assertIn(override, env.proto.experiments)
+
+  @mock.patch('apache_beam.runners.dataflow.internal.dependency.'
+  'beam_version.__version__', '2.2.0-dev')
+  def test_harness_override_absent_in_unreleased_sdk(self):
+pipeline_options = PipelineOptions(
+['--temp_location', 'gs://any-location/temp', '--streaming'])
+with mock.patch(
+'apache_beam.runners.dataflow.internal.dependency.pkg_resources'
+'.get_distribution',
+mock.Mock(side_effect=pkg_resources.DistributionNotFound())):
+  env = apiclient.Environment([], #packages
+  pipeline_options,
+  '2.0.0') #any environment version
+  if env.proto.experiments:
+for experiment in env.proto.experiments:
+  self.assertNotIn('runner_harness_container_image=', experiment)
 
 
 if __name__ == '__main__':

http://git-wip-us.apache.org/repos/asf/beam/blob/7155931f/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/dependency.py 
b/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
index 123fc49..c1edf7d 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
@@ -500,9 +500,16 @@ def get_runner_harness_container_image():
   """
   try:
 version = pkg_resources.get_distribution(GOOGLE_PACKAGE_NAME).version
+# Pin runner harness for Dataflow releases.
 return (DATAFLOW_CONTAINER_IMAGE_REPOSITORY + '/' + 'harness' + ':' +
 version)
   except pkg_resources.DistributionNotFound:
+# Pin runner harness for BEAM releases.
+if 'dev' not in beam_version.__version__:
+  return (DATAFLOW_CONTAINER_IMAGE_REPOSITORY + '/' + 'harness' + ':' +
+  beam_version.__version__)
+# Don't pin runner harness for BEAM head so that we can notice
+# potential incompatibility between runner and sdk harnesse

Build failed in Jenkins: beam_PostCommit_Python_Verify #3390

2017-10-19 Thread Apache Jenkins Server

See 


--
[...truncated 884.48 KB...]
copying apache_beam/portability/api/standard_window_fns_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.

[jira] [Closed] (BEAM-3019) make AvroIOWriteTransformTest more generic

2017-10-19 Thread Etienne Chauchot (JIRA)


 [ 
https://issues.apache.org/jira/browse/BEAM-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Etienne Chauchot closed BEAM-3019.
--
   Resolution: Won't Do
Fix Version/s: Not applicable

I'm closing this ticket and the related PR because it's coverage is included in 
PR #3999 that also merges {{AvroIOTransformTest}} into {{AvroIOTest}} 

> make AvroIOWriteTransformTest more generic
> --
>
> Key: BEAM-3019
> URL: https://issues.apache.org/jira/browse/BEAM-3019
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-extensions, testing
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
> Fix For: Not applicable
>
>
> make {{runTestWrite()}} more generic to be able to use {{GenericRecord[]}} as 
> input for {{writeGenericRecords}} test in place of {{AvroGeneratedUser}}
> make {{readAvroFile()}} generic to be able to read {{GenericRecords}} using 
> {{GenericDatumReader}} for {{writeGenericRecords}} test



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-3019) make AvroIOWriteTransformTest more generic

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210953#comment-16210953
 ] 

ASF GitHub Bot commented on BEAM-3019:
--

Github user echauchot closed the pull request at:

https://github.com/apache/beam/pull/3948


> make AvroIOWriteTransformTest more generic
> --
>
> Key: BEAM-3019
> URL: https://issues.apache.org/jira/browse/BEAM-3019
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-extensions, testing
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>
> make {{runTestWrite()}} more generic to be able to use {{GenericRecord[]}} as 
> input for {{writeGenericRecords}} test in place of {{AvroGeneratedUser}}
> make {{readAvroFile()}} generic to be able to read {{GenericRecords}} using 
> {{GenericDatumReader}} for {{writeGenericRecords}} test



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3948: [BEAM-3019] Make AvroIOWriteTransformTest more gene...

2017-10-19 Thread echauchot

Github user echauchot closed the pull request at:

https://github.com/apache/beam/pull/3948


---

Build failed in Jenkins: beam_PerformanceTests_Python #466

2017-10-19 Thread Apache Jenkins Server

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 1974b920e4b3bbe8549e25fe789f9dada13c1769 (origin/master)
Commit message: "This closes #3977: [BEAM-2963] Stage pipeline in 
DataflowRunner"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1974b920e4b3bbe8549e25fe789f9dada13c1769
 > git rev-list 1974b920e4b3bbe8549e25fe789f9dada13c1769 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3972513318357171114.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4765995492248647682.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2756610790617509435.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in /usr/lib/python2.7/dist-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7814949316910416754.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev)
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apach

[GitHub] beam pull request #3903: [BEAM-2530] Fix javadoc generation on Java 9 and up...

2017-10-19 Thread iemejia

Github user iemejia closed the pull request at:

https://github.com/apache/beam/pull/3903


---

[jira] [Commented] (BEAM-2530) Make Beam compatible with Java 9

2017-10-19 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16210918#comment-16210918
 ] 

ASF GitHub Bot commented on BEAM-2530:
--

Github user iemejia closed the pull request at:

https://github.com/apache/beam/pull/3903


> Make Beam compatible with Java 9
> 
>
> Key: BEAM-2530
> URL: https://issues.apache.org/jira/browse/BEAM-2530
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Affects Versions: Not applicable
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: java9
> Fix For: Not applicable
>
>
> Java 9 seems to be finally been released this year, this is a JIRA to keep 
> track of the needed changes to support Beam on Java 9.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #4008: Fix error and warnings on KafkaIO javadoc generatio...

2017-10-19 Thread iemejia

Github user iemejia closed the pull request at:

https://github.com/apache/beam/pull/4008


---

Build failed in Jenkins: beam_PostCommit_Python_Verify #3389

2017-10-19 Thread Apache Jenkins Server

See 


--
[...truncated 883.47 KB...]
copying apache_beam/portability/api/standard_window_fns_pb2.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/portability/api/standard_window_fns_pb2_grpc.py -> 
apache-beam-2.3.0.dev0/apache_beam/portability/api
copying apache_beam/runners/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.pxd -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/common_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/pipeline_context_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.3.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4193

2017-10-19 Thread Apache Jenkins Server

See

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #5068

2017-10-19 Thread Apache Jenkins Server

See

[jira] [Created] (BEAM-3077) Add the supported Checkstyle-IDEA plugin version

2017-10-19 Thread Uri Silberstein (JIRA)

Uri Silberstein created BEAM-3077:
-

 Summary: Add the supported Checkstyle-IDEA plugin version
 Key: BEAM-3077
 URL: https://issues.apache.org/jira/browse/BEAM-3077
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Uri Silberstein
Assignee: Uri Silberstein
Priority: Minor


Need to specify the exact Checkstyle-IDEA plugin version, as the latest fails 
to load beam codestyle



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

71 matches

Mail list logo