[jira] [Updated] (BEAM-793) JdbcIO can create a deadlock when parallelism is greater than 1

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-793:
--
Fix Version/s: 2.2.0

> JdbcIO can create a deadlock when parallelism is greater than 1
> ---
>
> Key: BEAM-793
> URL: https://issues.apache.org/jira/browse/BEAM-793
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 2.2.0
>
>
> With the following JdbcIO configuration, if the parallelism is greater than 
> 1, we can have a {{Deadlock found when trying to get lock; try restarting 
> transaction}}.
> {code}
> MysqlDataSource dbCfg = new MysqlDataSource();
> dbCfg.setDatabaseName("db");
> dbCfg.setUser("user");
> dbCfg.setPassword("pass");
> dbCfg.setServerName("localhost");
> dbCfg.setPortNumber(3306);
> p.apply(Create.of(data))
> .apply(JdbcIO. Long>>write()
> 
> .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(dbCfg))
> .withStatement("INSERT INTO 
> smth(loc,event_type,hash,begin_date,end_date) VALUES(?, ?, ?, ?, ?) ON 
> DUPLICATE KEY UPDATE event_type=VALUES(event_type),end_date=VALUES(end_date)")
> .withPreparedStatementSetter(new 
> JdbcIO.PreparedStatementSetter Long>>() {
> public void setParameters(Tuple5 Integer, ByteString, Long, Long> element, PreparedStatement statement)
> throws Exception {
> statement.setInt(1, element.f0);
> statement.setInt(2, element.f1);
> statement.setBytes(3, 
> element.f2.toByteArray());
> statement.setLong(4, element.f3);
> statement.setLong(5, element.f4);
> }
> }));
> {code}
> This can happen due to the {{autocommit}}. I'm going to investigate.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-793) JdbcIO can create a deadlock when parallelism is greater than 1

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174287#comment-16174287
 ] 

Jean-Baptiste Onofré commented on BEAM-793:
---

Thanks for the update Guillaume, I will work on this one today in order to 
include in 2.2.0.

> JdbcIO can create a deadlock when parallelism is greater than 1
> ---
>
> Key: BEAM-793
> URL: https://issues.apache.org/jira/browse/BEAM-793
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 2.2.0
>
>
> With the following JdbcIO configuration, if the parallelism is greater than 
> 1, we can have a {{Deadlock found when trying to get lock; try restarting 
> transaction}}.
> {code}
> MysqlDataSource dbCfg = new MysqlDataSource();
> dbCfg.setDatabaseName("db");
> dbCfg.setUser("user");
> dbCfg.setPassword("pass");
> dbCfg.setServerName("localhost");
> dbCfg.setPortNumber(3306);
> p.apply(Create.of(data))
> .apply(JdbcIO. Long>>write()
> 
> .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(dbCfg))
> .withStatement("INSERT INTO 
> smth(loc,event_type,hash,begin_date,end_date) VALUES(?, ?, ?, ?, ?) ON 
> DUPLICATE KEY UPDATE event_type=VALUES(event_type),end_date=VALUES(end_date)")
> .withPreparedStatementSetter(new 
> JdbcIO.PreparedStatementSetter Long>>() {
> public void setParameters(Tuple5 Integer, ByteString, Long, Long> element, PreparedStatement statement)
> throws Exception {
> statement.setInt(1, element.f0);
> statement.setInt(2, element.f1);
> statement.setBytes(3, 
> element.f2.toByteArray());
> statement.setLong(4, element.f3);
> statement.setLong(5, element.f4);
> }
> }));
> {code}
> This can happen due to the {{autocommit}}. I'm going to investigate.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2701) use a custom implementation of java.io.ObjectInputStream

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-2701:
--

Assignee: Jean-Baptiste Onofré  (was: Luke Cwik)

> use a custom implementation of java.io.ObjectInputStream
> 
>
> Key: BEAM-2701
> URL: https://issues.apache.org/jira/browse/BEAM-2701
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Jean-Baptiste Onofré
>
> java.io.ObjectInputStream should override resolve[Proxy]Class using the TCCL 
> to support any classloader and not fallback into some JVM pitfall using 
> another classloader (default). This will enable beam to use any classloader 
> instead of requiring to run in the JVM using java serialization.
> {code}
> @Override
> protected Class resolveClass(final ObjectStreamClass classDesc) throws 
> IOException, ClassNotFoundException {
> final String n = classDesc.getName();
> final ClassLoader classloader = getClassloader();
> try {
> return Class.forName(n, false, classloader);
> } catch (ClassNotFoundException e) {
> if (n.equals("boolean")) {
> return boolean.class;
> }
> if (n.equals("byte")) {
> return byte.class;
> }
> if (n.equals("char")) {
> return char.class;
> }
> if (n.equals("short")) {
> return short.class;
> }
> if (n.equals("int")) {
> return int.class;
> }
> if (n.equals("long")) {
> return long.class;
> }
> if (n.equals("float")) {
> return float.class;
> }
> if (n.equals("double")) {
> return double.class;
> }
> //Last try - Let runtime try and find it.
> return Class.forName(n, false, null);
> }
> }
> @Override
> protected Class resolveProxyClass(final String[] interfaces) throws 
> IOException, ClassNotFoundException {
> final Class[] cinterfaces = new Class[interfaces.length];
> for (int i = 0; i < interfaces.length; i++) {
> cinterfaces[i] = getClassloader().loadClass(interfaces[i]);
> }
> try {
> return Proxy.getProxyClass(getClassloader(), cinterfaces);
> } catch (IllegalArgumentException e) {
> throw new ClassNotFoundException(null, e);
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2970) Add comparator function to equal_to

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174188#comment-16174188
 ] 

ASF GitHub Bot commented on BEAM-2970:
--

GitHub user sarahwalters opened a pull request:

https://github.com/apache/beam/pull/3878

[BEAM-2970] Add contains_in_any_order matcher

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [X] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [X] Each commit in the pull request should have a meaningful subject 
line and body.
 - [X] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [X] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [X] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

The Python testing utilities already provide an equal_to matcher, but the 
equal_to matcher assumes that the lists it's comparing contain objects which 
can be compared using < and ==. In particular, classes don't always define < 
and ==. contains_in_any_order is more general than equal_to: it doesn't rely on 
< and ==; instead, it takes as an argument a list of matcher functions, 
allowing the caller to provide its own definition of equality.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sarahwalters/beam contains_in_any_order

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3878.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3878


commit 295c936aaf247788a441a8b0575ee057588f35cb
Author: Sarah Walters 
Date:   2017-09-20T23:08:06Z

Add a contains_in_any_order matcher.

The testing utilities already provide an equal_to matcher,
but the equal_to matcher assumes that the lists it's comparing
contain objects which can be compared using < and ==. In
particular, classes don't always define < and ==.
contains_in_any_order is more general than equal_to -- it
doesn't rely on < and ==; instead, it takes as an argument
a list of matcher functions, allowing the caller to provide
its own definition of equality.

commit 44dd2c6e7b27113d4cfd47a0b502a4e0b6cb87af
Author: Sarah Walters 
Date:   2017-09-21T02:41:45Z

Fix contains_in_any_order so it ignores order

Now, contains_in_any_order performs a breadth-first search to find
a mapping from items to predicates they satisfy. With this change,
it becomes acceptable to specify predicates that match multiple
items -> there's no danger of a predicate meant to match item i
matching item j instead and being unavailable to match item i.




> Add comparator function to equal_to
> ---
>
> Key: BEAM-2970
> URL: https://issues.apache.org/jira/browse/BEAM-2970
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Sarah Walters
>Assignee: Ahmet Altay
>Priority: Minor
>
> The equal_to function provided by testing/util.py 
> (https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/util.py#L54)
>  assumes that the actual and expected lists can be sorted using Python's 
> sorted method (which relies on the < operator) and compared using the == 
> operator.
> If this isn't the case, equal_to sometimes reports False incorrectly, when 
> the expected and actual lists are in different orders.
> Add a comparator function to equal_to in order to allow callers to define a 
> total order.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3878: [BEAM-2970] Add contains_in_any_order matcher

[sarahwalters]

GitHub user sarahwalters opened a pull request:

https://github.com/apache/beam/pull/3878

[BEAM-2970] Add contains_in_any_order matcher

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [X] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [X] Each commit in the pull request should have a meaningful subject 
line and body.
 - [X] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [X] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [X] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

The Python testing utilities already provide an equal_to matcher, but the 
equal_to matcher assumes that the lists it's comparing contain objects which 
can be compared using < and ==. In particular, classes don't always define < 
and ==. contains_in_any_order is more general than equal_to: it doesn't rely on 
< and ==; instead, it takes as an argument a list of matcher functions, 
allowing the caller to provide its own definition of equality.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sarahwalters/beam contains_in_any_order

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3878.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3878


commit 295c936aaf247788a441a8b0575ee057588f35cb
Author: Sarah Walters 
Date:   2017-09-20T23:08:06Z

Add a contains_in_any_order matcher.

The testing utilities already provide an equal_to matcher,
but the equal_to matcher assumes that the lists it's comparing
contain objects which can be compared using < and ==. In
particular, classes don't always define < and ==.
contains_in_any_order is more general than equal_to -- it
doesn't rely on < and ==; instead, it takes as an argument
a list of matcher functions, allowing the caller to provide
its own definition of equality.

commit 44dd2c6e7b27113d4cfd47a0b502a4e0b6cb87af
Author: Sarah Walters 
Date:   2017-09-21T02:41:45Z

Fix contains_in_any_order so it ignores order

Now, contains_in_any_order performs a breadth-first search to find
a mapping from items to predicates they satisfy. With this change,
it becomes acceptable to specify predicates that match multiple
items -> there's no danger of a predicate meant to match item i
matching item j instead and being unavailable to match item i.




---

Build failed in Jenkins: beam_PostCommit_Python_Verify #3180

[Apache Jenkins Server]

See 


--
[...truncated 44.46 KB...]
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.2.0.dev0)
  Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting oauth2client<4.0.0,>=2.0.1 (from apache-beam==2.2.0.dev0)
Collecting protobuf<=3.3.0,>=3.2.0 (from apache-beam==2.2.0.dev0)
  Using cached protobuf-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.2.0.dev0)
Collecting six<1.11,>=1.9 (from apache-beam==2.2.0.dev0)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.2.0.dev0)
  Using cached typing-3.6.2-py2-none-any.whl
Requirement already satisfied: enum34>=1.0.4 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Requirement already satisfied: futures>=2.2.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1_modules-0.1.4-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1-0.3.6-py2.py3-none-any.whl
Requirement already satisfied: setuptools in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
protobuf<=3.3.0,>=3.2.0->apache-beam==2.2.0.dev0)
Building wheels for collected packages: apache-beam
  Running setup.py bdist_wheel for apache-beam: started
  Running setup.py bdist_wheel for apache-beam: finished with status 'error'
  Complete output from command 

 -u -c "import setuptools, 
tokenize;__file__='/tmp/pip-KDhvT1-build/setup.py';f=getattr(tokenize, 'open', 
open)(__file__);code=f.read().replace('\r\n', 
'\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d 
/tmp/tmpPYvuNEpip-wheel- --python-tag cp27:
  
:351:
 UserWarning: Normalizing '2.2.0.dev' to '2.2.0.dev0'
normalized_version,
  running bdist_wheel
  running build
  running build_py
  Traceback (most recent call last):
File "", line 1, in 
File "/tmp/pip-KDhvT1-build/setup.py", line 203, in 
  'test': generate_protos_first(test),
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
  dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
  self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File 
"
 line 204, in run
  self.run_command('build')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
  self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/tmp/pip-KDhvT1-build/setup.py", line 143, in run
  gen_protos.generate_proto_files()
File "gen_protos.py", line 65, in generate_proto_files
  'Not in apache git tree; unable to find proto definitions.')
  RuntimeError: Not in apache git tree; unable to find proto definitions.
  
  
  Failed building wheel for apache-beam
  Running setup.py clean for apache-beam
Failed to build apache-beam
Installing collected packages: avro, crcmod, dill, httplib2, pbr, six, 
funcsigs, mock, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pyyaml, 
typing, apache-beam
  Found existing installation: six 1.11.0
Uninstalling six-1.11.0:
  Successfully uninstalled six-1.11.0
  Found existing installation: protobuf 3.4.0
Uninstalling protobuf-3.4.0:
  Successfully uninstalled protobuf-3.4.0
  Running setup.py install for apache-beam: started
Running setup.py install for apache-beam: finished with status 'error'
Complete output from 

[jira] [Commented] (BEAM-2970) Add comparator function to equal_to

[Sarah Walters (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174158#comment-16174158
 ] 

Sarah Walters commented on BEAM-2970:
-

Spoke with robertwb over email and decided that a contains_in_any_order 
function is a more appropriate way to address this issue.

> Add comparator function to equal_to
> ---
>
> Key: BEAM-2970
> URL: https://issues.apache.org/jira/browse/BEAM-2970
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Sarah Walters
>Assignee: Ahmet Altay
>Priority: Minor
>
> The equal_to function provided by testing/util.py 
> (https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/util.py#L54)
>  assumes that the actual and expected lists can be sorted using Python's 
> sorted method (which relies on the < operator) and compared using the == 
> operator.
> If this isn't the case, equal_to sometimes reports False incorrectly, when 
> the expected and actual lists are in different orders.
> Add a comparator function to equal_to in order to allow callers to define a 
> total order.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-2970) Add comparator function to equal_to

[Sarah Walters (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174158#comment-16174158
 ] 

Sarah Walters edited comment on BEAM-2970 at 9/21/17 2:48 AM:
--

Spoke with robertwb over email and agreed that a contains_in_any_order function 
is a more appropriate way to address this issue.


was (Author: sarahwalters):
Spoke with robertwb over email and decided that a contains_in_any_order 
function is a more appropriate way to address this issue.

> Add comparator function to equal_to
> ---
>
> Key: BEAM-2970
> URL: https://issues.apache.org/jira/browse/BEAM-2970
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Sarah Walters
>Assignee: Ahmet Altay
>Priority: Minor
>
> The equal_to function provided by testing/util.py 
> (https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/util.py#L54)
>  assumes that the actual and expected lists can be sorted using Python's 
> sorted method (which relies on the < operator) and compared using the == 
> operator.
> If this isn't the case, equal_to sometimes reports False incorrectly, when 
> the expected and actual lists are in different orders.
> Add a comparator function to equal_to in order to allow callers to define a 
> total order.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4013

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4840

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_JDBC #214

[Apache Jenkins Server]

See 


Changes:

[kirpichov] Marks TikaIO as experimental to allow backward-incompatible changes

[aljoscha.krettek] [BEAM-2377] Allow cross compilation (2.10,2.11) for flink 
runner

[tgroh] Add PipelineOptionsTranslation

[tgroh] Add Nullable getters to MetricsContainerImpl

[robertwb] Exclude incompatible six release, part 2

[robertwb] Enable progress request handling in python SDK harness

[robertwb] Minor cleanup.

[robertwb] Revert "[BEAM-2377] Allow cross compilation (2.10,2.11) for flink

--
[...truncated 160.11 KB...]
unable to connect to a server to handle "replicationcontrollers": failed to 
negotiate an api version; server supports: map[], client supports: 
map[componentconfig/v1alpha1:{} batch/v1:{} autoscaling/v1:{} 
authorization.k8s.io/v1beta1:{} v1:{} metrics/v1alpha1:{} extensions/v1beta1:{}]

2017-09-21 00:49:43,947 0372d89a MainThread beam_integration_benchmark(1/1) 
ERRORRetrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2017-09-21 00:50:02,339 0372d89a MainThread beam_integration_benchmark(1/1) 
INFO Running: /usr/lib/google-cloud-sdk/bin/kubectl 
--kubeconfig=/home/jenkins/.kube/config delete -f 

2017-09-21 00:50:02,411 0372d89a MainThread beam_integration_benchmark(1/1) 
INFO Ran /usr/lib/google-cloud-sdk/bin/kubectl 
--kubeconfig=/home/jenkins/.kube/config delete -f 

 Got return code (1).
STDOUT: 
STDERR: unable to connect to a server to handle "services": failed to negotiate 
an api version; server supports: map[], client supports: 
map[componentconfig/v1alpha1:{} batch/v1:{} autoscaling/v1:{} 
authorization.k8s.io/v1beta1:{} v1:{} metrics/v1alpha1:{} extensions/v1beta1:{}]
unable to connect to a server to handle "replicationcontrollers": failed to 
negotiate an api version; server supports: map[], client supports: map[v1:{} 
metrics/v1alpha1:{} extensions/v1beta1:{} componentconfig/v1alpha1:{} 
batch/v1:{} autoscaling/v1:{} authorization.k8s.io/v1beta1:{}]

2017-09-21 00:50:02,412 0372d89a MainThread beam_integration_benchmark(1/1) 
ERRORRetrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2017-09-21 00:50:32,085 0372d89a MainThread beam_integration_benchmark(1/1) 
INFO Running: /usr/lib/google-cloud-sdk/bin/kubectl 
--kubeconfig=/home/jenkins/.kube/config delete -f 

2017-09-21 00:50:32,113 0372d89a MainThread beam_integration_benchmark(1/1) 
INFO Ran /usr/lib/google-cloud-sdk/bin/kubectl 
--kubeconfig=/home/jenkins/.kube/config delete -f 

 Got return code (1).
STDOUT: 
STDERR: unable to connect to a server to handle "services": failed to negotiate 
an api version; server supports: map[], client supports: 
map[extensions/v1beta1:{} componentconfig/v1alpha1:{} batch/v1:{} 
autoscaling/v1:{} authorization.k8s.io/v1beta1:{} v1:{} metrics/v1alpha1:{}]
unable to connect to a server to handle "replicationcontrollers": failed to 
negotiate an api version; server supports: map[], client supports: 
map[autoscaling/v1:{} authorization.k8s.io/v1beta1:{} v1:{} metrics/v1alpha1:{} 
extensions/v1beta1:{} componentconfig/v1alpha1:{} batch/v1:{}]

2017-09-21 00:50:32,114 0372d89a MainThread beam_integration_benchmark(1/1) 
ERRORRetrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2017-09-21 00:50:55,749 0372d89a MainThread beam_integration_benchmark(1/1) 
INFO Running: /usr/lib/google-cloud-sdk/bin/kubectl 
--kubeconfig=/home/jenkins/.kube/config delete -f 

2017-09-21 00:50:55,778 0372d89a MainThread beam_integration_benchmark(1/1) 
INFO Ran /usr/lib/google-cloud-sdk/bin/kubectl 
--kubeconfig=/home/jenkins/.kube/config delete -f 

 Got return code (1).
STDOUT: 
STDERR: unable to connect to a server to handle "services": failed to negotiate 
an api version; server supports: map[], client supports: 
map[authorization.k8s.io/v1beta1:{} v1:{} metrics/v1alpha1:{} 
extensions/v1beta1:{} componentconfig/v1alpha1:{} batch/v1:{} autoscaling/v1:{}]
unable to connect to a server to handle "replicationcontrollers": failed to 
negotiate an api version; server supports: map[], client supports: 
map[batch/v1:{} autoscaling/v1:{} authorization.k8s.io/v1beta1:{} v1:{} 
metrics/v1alpha1:{} 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4838

[Apache Jenkins Server]

See 


Changes:

[robertwb] Enable progress request handling in python SDK harness

[robertwb] Minor cleanup.

--
[...truncated 4.28 MB...]
2017-09-21T00:49:57.733 [INFO] --- findbugs-maven-plugin:3.0.4:findbugs 
(findbugs) @ beam-runners-flink_2.11 ---
2017-09-21T00:49:57.739 [INFO] Fork Value is true
[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

[WARNING] Cannot calculate digest of mojo class, because mojo wasn't loaded 
from a jar, but from: 

[JENKINS] Archiving disabled
2017-09-21T00:49:58.767 [INFO]  
   
2017-09-21T00:49:58.767 [INFO] 

2017-09-21T00:49:58.767 [INFO] Skipping Apache Beam :: Parent
2017-09-21T00:49:58.767 [INFO] This project has been banned from the build due 
to previous failures.
2017-09-21T00:49:58.767 [INFO] 

[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
2017-09-21T00:51:13.950 [INFO] 

2017-09-21T00:51:13.950 [INFO] Reactor Summary:
2017-09-21T00:51:13.950 [INFO] 
2017-09-21T00:51:13.951 [INFO] Apache Beam :: Parent 
.. SUCCESS [01:08 min]
2017-09-21T00:51:13.951 [INFO] Apache Beam :: SDKs :: Java :: Build Tools 
. SUCCESS [ 16.757 s]
2017-09-21T00:51:13.951 [INFO] Apache Beam :: SDKs 
 SUCCESS [  5.508 s]
2017-09-21T00:51:13.951 [INFO] Apache Beam :: SDKs :: Common 
.. SUCCESS [  2.875 s]
2017-09-21T00:51:13.951 [INFO] Apache Beam :: SDKs :: Common :: Runner API 
 SUCCESS [ 30.955 s]
2017-09-21T00:51:13.951 [INFO] Apache Beam :: SDKs :: Common :: Fn API 
 SUCCESS [ 18.446 s]
2017-09-21T00:51:13.951 [INFO] Apache Beam :: SDKs :: Java 
 SUCCESS [  3.032 s]
2017-09-21T00:51:13.951 [INFO] Apache Beam :: SDKs :: Java :: Core 
 SUCCESS [02:33 min]
2017-09-21T00:51:13.951 [INFO] Apache Beam :: Runners 
. SUCCESS [  2.945 s]
2017-09-21T00:51:13.951 [INFO] Apache Beam :: Runners :: Core Construction Java 
... SUCCESS [ 35.800 s]
2017-09-21T00:51:13.951 [INFO] Apache Beam :: Runners :: Core Java 
 SUCCESS [01:03 min]
2017-09-21T00:51:13.951 [INFO] Apache Beam :: Runners :: Direct Java 
.. SUCCESS [05:09 min]
2017-09-21T00:51:13.951 

[GitHub] beam pull request #3877: Do windowing in the Fn API runner itself.

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3877

Do windowing in the Fn API runner itself.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam fn-api-windowing

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3877.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3877


commit 76d41f1004c4771083387ebc1c23879617639ee7
Author: Robert Bradshaw 
Date:   2017-09-14T07:30:10Z

Execute windowing in Fn API runner.

commit 54c8a597c0635560afbdbfb1b1f33cb8a4d3b81f
Author: Robert Bradshaw 
Date:   2017-09-21T00:49:31Z

cleanup




---

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #3117

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #3886

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4837

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #3116

[Apache Jenkins Server]

See 


--
[...truncated 339.90 KB...]
2017-09-20T23:59:07.230 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.1.4/snappy-java-1.1.4.jar
 (1471 KB at 5881.8 KB/sec)
2017-09-20T23:59:07.230 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-all/1.3/hamcrest-all-1.3.jar
2017-09-20T23:59:07.232 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.6.8/byte-buddy-1.6.8.jar
 (2706 KB at 10737.3 KB/sec)
2017-09-20T23:59:07.232 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.7.25/slf4j-jdk14-1.7.25.jar
2017-09-20T23:59:07.248 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.8.9/jackson-dataformat-yaml-2.8.9.jar
 (40 KB at 149.0 KB/sec)
2017-09-20T23:59:07.248 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/mockito/mockito-all/1.9.5/mockito-all-1.9.5.jar
2017-09-20T23:59:07.258 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.7.25/slf4j-jdk14-1.7.25.jar
 (9 KB at 29.7 KB/sec)
2017-09-20T23:59:07.259 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar
2017-09-20T23:59:07.273 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/yaml/snakeyaml/1.17/snakeyaml-1.17.jar 
(268 KB at 911.9 KB/sec)
2017-09-20T23:59:07.273 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar
2017-09-20T23:59:07.279 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-all/1.3/hamcrest-all-1.3.jar
 (300 KB at 1001.3 KB/sec)
2017-09-20T23:59:07.279 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm/4.0/asm-4.0.jar
2017-09-20T23:59:07.305 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar
 (65 KB at 197.2 KB/sec)
2017-09-20T23:59:07.305 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar
2017-09-20T23:59:07.308 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm/4.0/asm-4.0.jar (45 KB at 
137.0 KB/sec)
2017-09-20T23:59:07.308 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/objenesis/objenesis/1.2/objenesis-1.2.jar
2017-09-20T23:59:07.309 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar
 (355 KB at 1078.8 KB/sec)
2017-09-20T23:59:07.330 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar
 (5 KB at 13.9 KB/sec)
2017-09-20T23:59:07.336 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/objenesis/objenesis/1.2/objenesis-1.2.jar
 (36 KB at 98.9 KB/sec)
2017-09-20T23:59:07.343 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/auto/value/auto-value/1.4.1/auto-value-1.4.1.jar
 (1470 KB at 4048.1 KB/sec)
2017-09-20T23:59:07.380 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/mockito/mockito-all/1.9.5/mockito-all-1.9.5.jar
 (1545 KB at 3860.0 KB/sec)
2017-09-20T23:59:07.386 [INFO] 
2017-09-20T23:59:07.386 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-sdks-java-core ---
2017-09-20T23:59:07.387 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-09-20T23:59:07.503 [INFO] 
2017-09-20T23:59:07.503 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce) @ beam-sdks-java-core ---
2017-09-20T23:59:08.074 [INFO] 
2017-09-20T23:59:08.074 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce-banned-dependencies) @ beam-sdks-java-core ---
2017-09-20T23:59:08.189 [INFO] 
2017-09-20T23:59:08.190 [INFO] --- jacoco-maven-plugin:0.7.8:prepare-agent 
(default) @ beam-sdks-java-core ---
2017-09-20T23:59:08.193 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/junit/junit/4.8.2/junit-4.8.2.pom
2017-09-20T23:59:08.218 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/junit/junit/4.8.2/junit-4.8.2.pom (2 KB at 
57.9 KB/sec)
2017-09-20T23:59:08.220 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.2/maven-artifact-manager-2.0.2.pom
2017-09-20T23:59:08.246 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.2/maven-artifact-manager-2.0.2.pom
 (2 KB at 52.4 KB/sec)
2017-09-20T23:59:08.248 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/2.0.2/maven-repository-metadata-2.0.2.pom
2017-09-20T23:59:08.273 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #3885

[Apache Jenkins Server]

See 


--
[...truncated 338.75 KB...]
2017-09-20T23:59:00.639 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/auto/auto-common/0.3/auto-common-0.3.jar
 (39 KB at 112.3 KB/sec)
2017-09-20T23:59:00.639 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/auto/value/auto-value/1.4.1/auto-value-1.4.1.jar
2017-09-20T23:59:00.640 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.6/commons-lang3-3.6.jar
 (484 KB at 1421.3 KB/sec)
2017-09-20T23:59:00.641 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.8.9/jackson-dataformat-yaml-2.8.9.jar
2017-09-20T23:59:00.647 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.1.4/snappy-java-1.1.4.jar
 (1471 KB at 4237.6 KB/sec)
2017-09-20T23:59:00.647 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/joda-time/joda-time/2.4/joda-time-2.4.jar 
(573 KB at 1649.7 KB/sec)
2017-09-20T23:59:00.647 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/yaml/snakeyaml/1.17/snakeyaml-1.17.jar
2017-09-20T23:59:00.647 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-all/1.3/hamcrest-all-1.3.jar
2017-09-20T23:59:00.678 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.8.9/jackson-dataformat-yaml-2.8.9.jar
 (40 KB at 106.5 KB/sec)
2017-09-20T23:59:00.678 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.7.25/slf4j-jdk14-1.7.25.jar
2017-09-20T23:59:00.682 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.6.8/byte-buddy-1.6.8.jar
 (2706 KB at 7064.7 KB/sec)
2017-09-20T23:59:00.682 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/mockito/mockito-all/1.9.5/mockito-all-1.9.5.jar
2017-09-20T23:59:00.694 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/yaml/snakeyaml/1.17/snakeyaml-1.17.jar 
(268 KB at 683.3 KB/sec)
2017-09-20T23:59:00.694 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar
2017-09-20T23:59:00.700 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-all/1.3/hamcrest-all-1.3.jar
 (300 KB at 754.1 KB/sec)
2017-09-20T23:59:00.700 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar
2017-09-20T23:59:00.705 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-jdk14/1.7.25/slf4j-jdk14-1.7.25.jar
 (9 KB at 20.6 KB/sec)
2017-09-20T23:59:00.705 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm/4.0/asm-4.0.jar
2017-09-20T23:59:00.733 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/reflectasm/reflectasm/1.07/reflectasm-1.07-shaded.jar
 (65 KB at 149.0 KB/sec)
2017-09-20T23:59:00.733 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar
2017-09-20T23:59:00.735 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm/4.0/asm-4.0.jar (45 KB at 
104.0 KB/sec)
2017-09-20T23:59:00.735 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/objenesis/objenesis/1.2/objenesis-1.2.jar
2017-09-20T23:59:00.748 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/kryo/kryo/2.21/kryo-2.21.jar
 (355 KB at 797.6 KB/sec)
2017-09-20T23:59:00.761 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/esotericsoftware/minlog/minlog/1.2/minlog-1.2.jar
 (5 KB at 10.6 KB/sec)
2017-09-20T23:59:00.767 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/objenesis/objenesis/1.2/objenesis-1.2.jar
 (36 KB at 75.9 KB/sec)
2017-09-20T23:59:00.791 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/auto/value/auto-value/1.4.1/auto-value-1.4.1.jar
 (1470 KB at 3017.4 KB/sec)
2017-09-20T23:59:00.841 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/mockito/mockito-all/1.9.5/mockito-all-1.9.5.jar
 (1545 KB at 2869.9 KB/sec)
2017-09-20T23:59:00.850 [INFO] 
2017-09-20T23:59:00.850 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-sdks-java-core ---
2017-09-20T23:59:00.852 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-09-20T23:59:00.971 [INFO] 
2017-09-20T23:59:00.971 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce) @ beam-sdks-java-core ---
2017-09-20T23:59:01.975 [INFO] 
2017-09-20T23:59:01.975 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce-banned-dependencies) @ beam-sdks-java-core ---

[GitHub] beam pull request #3875: Fix Jenkins presubmits by reverting #3255

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3875


---

[1/2] beam git commit: Revert "[BEAM-2377] Allow cross compilation (2.10, 2.11) for flink runner"

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master ddc0a7d83 -> a92c45f93


Revert "[BEAM-2377] Allow cross compilation (2.10,2.11) for flink runner"

This reverts commit ab975317e1aa532053b68ccc105e13afff0c0b1a.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e1548435
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e1548435
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e1548435

Branch: refs/heads/master
Commit: e1548435c45b1e4b349f55df1e37e1b6de8fc500
Parents: ddc0a7d
Author: Charles Chen 
Authored: Wed Sep 20 15:17:34 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Sep 20 16:56:36 2017 -0700

--
 examples/java/pom.xml   |  2 +-
 examples/java8/pom.xml  |  2 +-
 pom.xml | 16 +---
 runners/flink/pom.xml   | 14 +++---
 sdks/java/javadoc/pom.xml   |  2 +-
 .../src/main/resources/archetype-resources/pom.xml  |  2 +-
 .../src/main/resources/archetype-resources/pom.xml  |  2 +-
 7 files changed, 13 insertions(+), 27 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e1548435/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 817af33..ade4cac 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -95,7 +95,7 @@
   
 
   org.apache.beam
-  beam-runners-flink_${flink.scala.version}
+  beam-runners-flink_2.10
   runtime
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/e1548435/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index f27f6df..585d7b8 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -95,7 +95,7 @@
   
 
   org.apache.beam
-  beam-runners-flink_${flink.scala.version}
+  beam-runners-flink_2.10
   runtime
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/e1548435/pom.xml
--
diff --git a/pom.xml b/pom.xml
index f112c64..236645c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -154,7 +154,6 @@
 1.1.4
 0.10.1.0
 1.4
-2.11
 
 1.5.0.Final
 2.0
@@ -364,19 +363,6 @@
 
   
 
-
-
-  flink-scala-2.10
-  
-
-  flink-scala-2.10
-
-  
-  
-2.10
-  
-
-
   
 
   
@@ -620,7 +606,7 @@
 
   
 org.apache.beam
-beam-runners-flink_${flink.scala.version}
+beam-runners-flink_2.10
 ${project.version}
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/e1548435/runners/flink/pom.xml
--
diff --git a/runners/flink/pom.xml b/runners/flink/pom.xml
index 5c680c8..0ef1931 100644
--- a/runners/flink/pom.xml
+++ b/runners/flink/pom.xml
@@ -26,7 +26,7 @@
 ../pom.xml
   
 
-  beam-runners-flink_${flink.scala.version}
+  beam-runners-flink_2.10
   Apache Beam :: Runners :: Flink
   jar
 
@@ -165,7 +165,7 @@
 
 
   org.apache.flink
-  flink-clients_${flink.scala.version}
+  flink-clients_2.10
   ${flink.version}
 
 
@@ -189,13 +189,13 @@
 
 
   org.apache.flink
-  flink-runtime_${flink.scala.version}
+  flink-runtime_2.10
   ${flink.version}
 
 
 
   org.apache.flink
-  flink-streaming-java_${flink.scala.version}
+  flink-streaming-java_2.10
   ${flink.version}
 
 
@@ -210,7 +210,7 @@
 
 
   org.apache.flink
-  flink-runtime_${flink.scala.version}
+  flink-runtime_2.10
   ${flink.version}
   test-jar
   test
@@ -336,7 +336,7 @@
 
 
   org.apache.flink
-  flink-streaming-java_${flink.scala.version}
+  flink-streaming-java_2.10
   ${flink.version}
   test
   test-jar
@@ -344,7 +344,7 @@
 
 
   org.apache.flink
-  flink-test-utils_${flink.scala.version}
+  flink-test-utils_2.10
   ${flink.version}
   test
   

http://git-wip-us.apache.org/repos/asf/beam/blob/e1548435/sdks/java/javadoc/pom.xml
--
diff --git a/sdks/java/javadoc/pom.xml b/sdks/java/javadoc/pom.xml
index 1d90046..35f0b86 100644
--- a/sdks/java/javadoc/pom.xml
+++ b/sdks/java/javadoc/pom.xml
@@ -64,7 +64,7 @@
 
 
   org.apache.beam
-  beam-runners-flink_${flink.scala.version}
+  beam-runners-flink_2.10
 
 
 

[2/2] beam git commit: Closes #3875

[robertwb]

Closes #3875


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a92c45f9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a92c45f9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a92c45f9

Branch: refs/heads/master
Commit: a92c45f9347e326e71afdc9057cac7e053b6c3e6
Parents: ddc0a7d e154843
Author: Robert Bradshaw 
Authored: Wed Sep 20 16:56:42 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Sep 20 16:56:42 2017 -0700

--
 examples/java/pom.xml   |  2 +-
 examples/java8/pom.xml  |  2 +-
 pom.xml | 16 +---
 runners/flink/pom.xml   | 14 +++---
 sdks/java/javadoc/pom.xml   |  2 +-
 .../src/main/resources/archetype-resources/pom.xml  |  2 +-
 .../src/main/resources/archetype-resources/pom.xml  |  2 +-
 7 files changed, 13 insertions(+), 27 deletions(-)
--

[jira] [Commented] (BEAM-2975) Results of ReadableState.read() should be snapshots of the underlying state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174021#comment-16174021
 ] 

ASF GitHub Bot commented on BEAM-2975:
--

GitHub user dpmills opened a pull request:

https://github.com/apache/beam/pull/3876

[BEAM-2975] Clarify semantics of objects returned by state access

R: @tgroh 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dpmills/incubator-beam state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3876.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3876


commit 306f7004753715dab68ee0210dcf85e24c255049
Author: Daniel Mills 
Date:   2017-09-20T23:35:06Z

Clarify semantics of objects returned by state access




> Results of ReadableState.read() should be snapshots of the underlying state
> ---
>
> Key: BEAM-2975
> URL: https://issues.apache.org/jira/browse/BEAM-2975
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Daniel Mills
>Assignee: Daniel Mills
>Priority: Minor
>
> Future modification of state should not be reflected in previous calls to 
> read().  For example:
> @StateId("tag") BagState state;
> Iterable ints = state.read();
> state.add(17);
> // ints should still be empty here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3876: [BEAM-2975] Clarify semantics of objects returned b...

[dpmills]

GitHub user dpmills opened a pull request:

https://github.com/apache/beam/pull/3876

[BEAM-2975] Clarify semantics of objects returned by state access

R: @tgroh 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dpmills/incubator-beam state

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3876.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3876


commit 306f7004753715dab68ee0210dcf85e24c255049
Author: Daniel Mills 
Date:   2017-09-20T23:35:06Z

Clarify semantics of objects returned by state access




---

[jira] [Created] (BEAM-2975) Results of ReadableState.read() should be snapshots of the underlying state

[Daniel Mills (JIRA)]

Daniel Mills created BEAM-2975:
--

 Summary: Results of ReadableState.read() should be snapshots of 
the underlying state
 Key: BEAM-2975
 URL: https://issues.apache.org/jira/browse/BEAM-2975
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Daniel Mills
Assignee: Daniel Mills
Priority: Minor


Future modification of state should not be reflected in previous calls to 
read().  For example:

@StateId("tag") BagState state;
Iterable ints = state.read();
state.add(17);
// ints should still be empty here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2822) Add support for progress reporting in fn API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173992#comment-16173992
 ] 

ASF GitHub Bot commented on BEAM-2822:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3784


> Add support for progress reporting in fn API
> 
>
> Key: BEAM-2822
> URL: https://issues.apache.org/jira/browse/BEAM-2822
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Vikas Kedigehalli
>Assignee: Robert Bradshaw
>Priority: Minor
>  Labels: portability
>
> https://s.apache.org/beam-fn-api-progress-reporting
> Note that the ULR reference implementation, when ready, should be useful for 
> every runner.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3784: [BEAM-2822] Enable progress request handling in pyt...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3784


---

[2/3] beam git commit: Minor cleanup.

[robertwb]

Minor cleanup.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0fa7fe1d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0fa7fe1d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0fa7fe1d

Branch: refs/heads/master
Commit: 0fa7fe1d2d36cbe8a36825dfac7a02978d9ed2d7
Parents: a241eda
Author: Robert Bradshaw 
Authored: Wed Sep 20 16:14:39 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Sep 20 16:14:39 2017 -0700

--
 .../apache_beam/runners/worker/sdk_worker.py  | 18 +-
 1 file changed, 9 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/0fa7fe1d/sdks/python/apache_beam/runners/worker/sdk_worker.py
--
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker.py 
b/sdks/python/apache_beam/runners/worker/sdk_worker.py
index 97f1f59..3534e2b 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker.py
@@ -123,9 +123,9 @@ class SdkWorker(object):
   def register(self, request, instruction_id):
 for process_bundle_descriptor in request.process_bundle_descriptor:
   self.fns[process_bundle_descriptor.id] = process_bundle_descriptor
-return beam_fn_api_pb2.InstructionResponse(**{
-'instruction_id': instruction_id,
-'register': beam_fn_api_pb2.RegisterResponse()})
+return beam_fn_api_pb2.InstructionResponse(
+instruction_id=instruction_id,
+register=beam_fn_api_pb2.RegisterResponse())
 
   def process_bundle(self, request, instruction_id):
 bundle_processor.BundleProcessor(
@@ -133,11 +133,11 @@ class SdkWorker(object):
 self.state_handler,
 self.data_channel_factory).process_bundle(instruction_id)
 
-return beam_fn_api_pb2.InstructionResponse(**{
-'instruction_id': instruction_id,
-'process_bundle': beam_fn_api_pb2.ProcessBundleResponse()})
+return beam_fn_api_pb2.InstructionResponse(
+instruction_id=instruction_id,
+process_bundle=beam_fn_api_pb2.ProcessBundleResponse())
 
   def process_bundle_progress(self, request, instruction_id):
-return beam_fn_api_pb2.InstructionResponse(**{
-'instruction_id': instruction_id,
-'error': 'Not Supported'})
+return beam_fn_api_pb2.InstructionResponse(
+instruction_id=instruction_id,
+error='Not Supported')

[1/3] beam git commit: Enable progress request handling in python SDK harness

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master fe395900a -> ddc0a7d83


Enable progress request handling in python SDK harness


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a241eda6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a241eda6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a241eda6

Branch: refs/heads/master
Commit: a241eda684376fffa80a4f530446ae6cae800a06
Parents: fe39590
Author: Vikas Kedigehalli 
Authored: Tue Aug 29 12:32:38 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Sep 20 16:10:37 2017 -0700

--
 .../apache_beam/runners/worker/sdk_worker.py| 74 ++--
 1 file changed, 52 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a241eda6/sdks/python/apache_beam/runners/worker/sdk_worker.py
--
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker.py 
b/sdks/python/apache_beam/runners/worker/sdk_worker.py
index 1481797..97f1f59 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker.py
@@ -21,10 +21,11 @@ from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
 
+import functools
 import logging
 import Queue as queue
-import threading
 import traceback
+from concurrent import futures
 
 import grpc
 
@@ -38,6 +39,9 @@ class SdkHarness(object):
   def __init__(self, control_address):
 self._control_channel = grpc.insecure_channel(control_address)
 self._data_channel_factory = data_plane.GrpcClientDataChannelFactory()
+# TODO: Ensure thread safety to run with more than 1 thread.
+self._default_work_thread_pool = futures.ThreadPoolExecutor(max_workers=1)
+self._progress_thread_pool = futures.ThreadPoolExecutor(max_workers=1)
 
   def run(self):
 contol_stub = beam_fn_api_pb2.BeamFnControlStub(self._control_channel)
@@ -55,23 +59,44 @@ class SdkHarness(object):
   return
 yield response
 
-def process_requests():
-  for work_request in contol_stub.Control(get_responses()):
-logging.info('Got work %s', work_request.instruction_id)
+for work_request in contol_stub.Control(get_responses()):
+  logging.info('Got work %s', work_request.instruction_id)
+  request_type = work_request.WhichOneof('request')
+  if request_type == ['process_bundle_progress']:
+thread_pool = self._progress_thread_pool
+  else:
+thread_pool = self._default_work_thread_pool
+
+  # Need this wrapper to capture the original stack trace.
+  def do_instruction(request):
+try:
+  return self.worker.do_instruction(request)
+except Exception as e:  # pylint: disable=broad-except
+  traceback_str = traceback.format_exc(e)
+  raise StandardError("Error processing request. Original traceback "
+  "is\n%s\n" % traceback_str)
+
+  def handle_response(request, response_future):
 try:
-  response = self.worker.do_instruction(work_request)
-except Exception:  # pylint: disable=broad-except
+  response = response_future.result()
+except Exception as e:  # pylint: disable=broad-except
   logging.error(
   'Error processing instruction %s',
-  work_request.instruction_id,
+  request.instruction_id,
   exc_info=True)
   response = beam_fn_api_pb2.InstructionResponse(
-  instruction_id=work_request.instruction_id,
-  error=traceback.format_exc())
+  instruction_id=request.instruction_id,
+  error=str(e))
 responses.put(response)
-t = threading.Thread(target=process_requests)
-t.start()
-t.join()
+
+  thread_pool.submit(do_instruction, work_request).add_done_callback(
+  functools.partial(handle_response, work_request))
+
+logging.info("No more requests from control plane")
+logging.info("SDK Harness waiting for in-flight requests to complete")
+# Wait until existing requests are processed.
+self._progress_thread_pool.shutdown()
+self._default_work_thread_pool.shutdown()
 # get_responses may be blocked on responses.get(), but we need to return
 # control to its caller.
 responses.put(no_more_work)
@@ -89,20 +114,18 @@ class SdkWorker(object):
   def do_instruction(self, request):
 request_type = request.WhichOneof('request')
 if request_type:
-  # E.g. if register is set, this will construct
-  # InstructionResponse(register=self.register(request.register))
-  return beam_fn_api_pb2.InstructionResponse(**{
-  

[3/3] beam git commit: Closes #3784

[robertwb]

Closes #3784


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ddc0a7d8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ddc0a7d8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ddc0a7d8

Branch: refs/heads/master
Commit: ddc0a7d83c71b4effeda04c710d8ede99d71eed4
Parents: fe39590 0fa7fe1
Author: Robert Bradshaw 
Authored: Wed Sep 20 16:15:00 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Sep 20 16:15:00 2017 -0700

--
 .../apache_beam/runners/worker/sdk_worker.py| 74 ++--
 1 file changed, 52 insertions(+), 22 deletions(-)
--

[GitHub] beam pull request #3874: Update Beam Python version for 2.1.1 release

[charlesccychen]

Github user charlesccychen closed the pull request at:

https://github.com/apache/beam/pull/3874


---

[1/2] beam git commit: Closes #3874

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/release-2.1.1 955e0122e -> d6e6ea9c7


Closes #3874


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d6e6ea9c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d6e6ea9c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d6e6ea9c

Branch: refs/heads/release-2.1.1
Commit: d6e6ea9c72bbccd96c36ddcb60a513a0b420563a
Parents: 955e012 27fd33a
Author: Robert Bradshaw 
Authored: Wed Sep 20 15:40:40 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Sep 20 15:40:40 2017 -0700

--
 sdks/python/apache_beam/version.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[2/2] beam git commit: Update Beam Python version to 2.1.1

[robertwb]

Update Beam Python version to 2.1.1


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/27fd33ab
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/27fd33ab
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/27fd33ab

Branch: refs/heads/release-2.1.1
Commit: 27fd33ab3e5f81497e13432ccabf189261cb43a4
Parents: 955e012
Author: Charles Chen 
Authored: Wed Sep 20 14:57:40 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Sep 20 15:40:40 2017 -0700

--
 sdks/python/apache_beam/version.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/27fd33ab/sdks/python/apache_beam/version.py
--
diff --git a/sdks/python/apache_beam/version.py 
b/sdks/python/apache_beam/version.py
index 842be0a..6a5e629 100644
--- a/sdks/python/apache_beam/version.py
+++ b/sdks/python/apache_beam/version.py
@@ -18,4 +18,4 @@
 """Apache Beam SDK version information and utilities."""
 
 
-__version__ = '2.1.0'
+__version__ = '2.1.1'

beam git commit: Exclude incompatible six release, part 2

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/release-2.1.1 de56f6b28 -> 955e0122e


Exclude incompatible six release, part 2


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/955e0122
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/955e0122
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/955e0122

Branch: refs/heads/release-2.1.1
Commit: 955e0122e8acc6f8b0635a053c766088097237d0
Parents: de56f6b
Author: Charles Chen 
Authored: Wed Sep 20 12:26:39 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Sep 20 15:39:34 2017 -0700

--
 sdks/python/setup.py | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/955e0122/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 25ea5c7..de118ab 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -104,6 +104,9 @@ REQUIRED_PACKAGES = [
 'oauth2client>=2.0.1,<4.0.0',
 'protobuf>=3.2.0,<=3.3.0',
 'pyyaml>=3.12,<4.0.0',
+# Six 1.11.0 incompatible with apitools.
+# TODO(BEAM-2964): Remove the upper bound.
+'six>=1.9,<1.11',
 ]
 
 REQUIRED_SETUP_PACKAGES = [

[jira] [Commented] (BEAM-2964) Latest six (1.11.0) produces "metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173946#comment-16173946
 ] 

ASF GitHub Bot commented on BEAM-2964:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3872


> Latest six (1.11.0) produces "metaclass conflict: the metaclass of a derived 
> class must be a (non-strict) subclass of the metaclasses of all its bases"
> ---
>
> Key: BEAM-2964
> URL: https://issues.apache.org/jira/browse/BEAM-2964
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.1.0
> Environment: Python 2.7 in a virtualenv on MacOS
>Reporter: Steven Normore
>Assignee: Ahmet Altay
>
> $ virtualenv venv
> [...]
> $ source venv/bin/activate
> [...]
> $ pip install apache-beam[gcp]==2.1.0 six==1.11.0
> [...]
> $ python -c 'import apache_beam'
> Traceback (most recent call last):
>   File "", line 1, in 
>   File 
> "/Users/snormore/Workspace/beam-six/venv/lib/python2.7/site-packages/apache_beam/__init__.py",
>  line 78, in 
> from apache_beam import io
>   File 
> "/Users/snormore/Workspace/beam-six/venv/lib/python2.7/site-packages/apache_beam/io/__init__.py",
>  line 21, in 
> from apache_beam.io.avroio import *
>   File 
> "/Users/snormore/Workspace/beam-six/venv/lib/python2.7/site-packages/apache_beam/io/avroio.py",
>  line 29, in 
> from apache_beam.io import filebasedsource
>   File 
> "/Users/snormore/Workspace/beam-six/venv/lib/python2.7/site-packages/apache_beam/io/filebasedsource.py",
>  line 33, in 
> from apache_beam.io.filesystems import FileSystems
>   File 
> "/Users/snormore/Workspace/beam-six/venv/lib/python2.7/site-packages/apache_beam/io/filesystems.py",
>  line 31, in 
> from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem
>   File 
> "/Users/snormore/Workspace/beam-six/venv/lib/python2.7/site-packages/apache_beam/io/gcp/gcsfilesystem.py",
>  line 27, in 
> from apache_beam.io.gcp import gcsio
>   File 
> "/Users/snormore/Workspace/beam-six/venv/lib/python2.7/site-packages/apache_beam/io/gcp/gcsio.py",
>  line 36, in 
> from apache_beam.utils import retry
>   File 
> "/Users/snormore/Workspace/beam-six/venv/lib/python2.7/site-packages/apache_beam/utils/retry.py",
>  line 38, in 
> from apitools.base.py.exceptions import HttpError
>   File 
> "/Users/snormore/Workspace/beam-six/venv/lib/python2.7/site-packages/apitools/base/py/__init__.py",
>  line 21, in 
> from apitools.base.py.base_api import *
>   File 
> "/Users/snormore/Workspace/beam-six/venv/lib/python2.7/site-packages/apitools/base/py/base_api.py",
>  line 31, in 
> from apitools.base.protorpclite import message_types
>   File 
> "/Users/snormore/Workspace/beam-six/venv/lib/python2.7/site-packages/apitools/base/protorpclite/message_types.py",
>  line 25, in 
> from apitools.base.protorpclite import messages
>   File 
> "/Users/snormore/Workspace/beam-six/venv/lib/python2.7/site-packages/apitools/base/protorpclite/messages.py",
>  line 1165, in 
> class Field(six.with_metaclass(_FieldMeta, object)):
> TypeError: Error when calling the metaclass bases
> metaclass conflict: the metaclass of a derived class must be a 
> (non-strict) subclass of the metaclasses of all its bases
> (venv)
> $ pip install six==1.10.0
> [...]
> $ python -c 'import apache_beam'
> [success]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3872: [BEAM-2964] Exclude incompatible six release, part ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3872


---

[1/2] beam git commit: Closes #3872

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 59542c3cc -> fe395900a


Closes #3872


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fe395900
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fe395900
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fe395900

Branch: refs/heads/master
Commit: fe395900a80abba3fd60f10399f0273fb0206eb9
Parents: 59542c3 fe6e95b
Author: Robert Bradshaw 
Authored: Wed Sep 20 15:34:41 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Sep 20 15:34:41 2017 -0700

--
 sdks/python/setup.py | 3 +++
 1 file changed, 3 insertions(+)
--

[2/2] beam git commit: Exclude incompatible six release, part 2

[robertwb]

Exclude incompatible six release, part 2


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fe6e95b2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fe6e95b2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fe6e95b2

Branch: refs/heads/master
Commit: fe6e95b28da11b2556e449c3c8e6bc5c8f42dde2
Parents: 59542c3
Author: Charles Chen 
Authored: Wed Sep 20 12:26:39 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Sep 20 15:34:41 2017 -0700

--
 sdks/python/setup.py | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/fe6e95b2/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 2bc2e99..cac2ea7 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -104,6 +104,9 @@ REQUIRED_PACKAGES = [
 'oauth2client>=2.0.1,<4.0.0',
 'protobuf>=3.2.0,<=3.3.0',
 'pyyaml>=3.12,<4.0.0',
+# Six 1.11.0 incompatible with apitools.
+# TODO(BEAM-2964): Remove the upper bound.
+'six>=1.9,<1.11',
 'typing>=3.6.0,<3.7.0',
 ]
 

[GitHub] beam pull request #3875: Fix Jenkins presubmits by reverting #3255

[charlesccychen]

GitHub user charlesccychen opened a pull request:

https://github.com/apache/beam/pull/3875

Fix Jenkins presubmits by reverting #3255

Currently, presubmits at head are broken with the following error:

```
[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
ERROR: Failed to parse POMs
java.io.IOException: remote file operation failed: 
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2 at 
hudson.remoting.Channel@7b1b4155:beam7: hudson.remoting.ProxyException: 
hudson.maven.MavenModuleSetBuild$MavenExecutionException: 
org.apache.maven.project.ProjectBuildingException: Some problems were 
encountered while processing the POMs:
[ERROR] 'dependencies.dependency.version' for 
org.apache.beam:beam-runners-flink_2.10:jar is missing. @ line 68, column 21
[WARNING] 'artifactId' contains an expression but should be a constant. @ 
org.apache.beam:beam-runners-flink_${flink.scala.version}:2.2.0-SNAPSHOT, 
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/runners/flink/pom.xml,
 line 29, column 15

at hudson.FilePath.act(FilePath.java:993)
at hudson.FilePath.act(FilePath.java:975)
at 
hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.parsePoms(MavenModuleSetBuild.java:985)
at 
hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:690)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:490)
at hudson.model.Run.execute(Run.java:1735)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:542)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:405)
Caused by: hudson.remoting.ProxyException: 
hudson.maven.MavenModuleSetBuild$MavenExecutionException: 
org.apache.maven.project.ProjectBuildingException: Some problems were 
encountered while processing the POMs:
[...]
```

https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/14503/console

This change reverts the suspected cause #3255 to unbreak development at 
head.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/beam fix-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3875.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3875


commit 66dca6c33d5a4100a4aa4af1101d74902e347671
Author: Charles Chen 
Date:   2017-09-20T22:17:34Z

Revert "[BEAM-2377] Allow cross compilation (2.10,2.11) for flink runner"

This reverts commit ab975317e1aa532053b68ccc105e13afff0c0b1a.




---

[jira] [Commented] (BEAM-793) JdbcIO can create a deadlock when parallelism is greater than 1

[Guillaume Balaine (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173911#comment-16173911
 ] 

Guillaume Balaine commented on BEAM-793:


This is a major problem, it easily occurs if batches are available in quick 
succession and contain unordered data.
There needs to be a lock so that finishBundle does not try to write if the 
previous batch is not finished.

> JdbcIO can create a deadlock when parallelism is greater than 1
> ---
>
> Key: BEAM-793
> URL: https://issues.apache.org/jira/browse/BEAM-793
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> With the following JdbcIO configuration, if the parallelism is greater than 
> 1, we can have a {{Deadlock found when trying to get lock; try restarting 
> transaction}}.
> {code}
> MysqlDataSource dbCfg = new MysqlDataSource();
> dbCfg.setDatabaseName("db");
> dbCfg.setUser("user");
> dbCfg.setPassword("pass");
> dbCfg.setServerName("localhost");
> dbCfg.setPortNumber(3306);
> p.apply(Create.of(data))
> .apply(JdbcIO. Long>>write()
> 
> .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(dbCfg))
> .withStatement("INSERT INTO 
> smth(loc,event_type,hash,begin_date,end_date) VALUES(?, ?, ?, ?, ?) ON 
> DUPLICATE KEY UPDATE event_type=VALUES(event_type),end_date=VALUES(end_date)")
> .withPreparedStatementSetter(new 
> JdbcIO.PreparedStatementSetter Long>>() {
> public void setParameters(Tuple5 Integer, ByteString, Long, Long> element, PreparedStatement statement)
> throws Exception {
> statement.setInt(1, element.f0);
> statement.setInt(2, element.f1);
> statement.setBytes(3, 
> element.f2.toByteArray());
> statement.setLong(4, element.f3);
> statement.setLong(5, element.f4);
> }
> }));
> {code}
> This can happen due to the {{autocommit}}. I'm going to investigate.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #4836

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3874: Update Beam Python version for 2.1.1 release

[charlesccychen]

GitHub user charlesccychen opened a pull request:

https://github.com/apache/beam/pull/3874

Update Beam Python version for 2.1.1 release

R: @robertwb 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/beam update-211-version

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3874.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3874


commit 9b99e4c89a980f4cd81b3713f9ba2a2e9072772b
Author: Charles Chen 
Date:   2017-09-20T21:57:40Z

Update Beam Python version to 2.1.1




---

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #2430

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2973) Jenkins PreCommit broken due to missing dependency

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173824#comment-16173824
 ] 

ASF GitHub Bot commented on BEAM-2973:
--

GitHub user youngoli opened a pull request:

https://github.com/apache/beam/pull/3873

[BEAM-2973] Reverting change to attempt to fix Jenkins PreCommit

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

This reverts commit ab975317e1aa532053b68ccc105e13afff0c0b1a. That commit 
seems to be causing issues with Jenkins for whatever reason.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/youngoli/beam bugfix-beam2973

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3873.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3873


commit e698efc634f35c72af409577eccc2b541b379cba
Author: Daniel Oliveira 
Date:   2017-09-20T20:46:22Z

[BEAM-2973] Revert "[BEAM-2377] Allow cross compilation (2.10,2.11) for 
flink runner"

This reverts commit ab975317e1aa532053b68ccc105e13afff0c0b1a.




> Jenkins PreCommit broken due to missing dependency
> --
>
> Key: BEAM-2973
> URL: https://issues.apache.org/jira/browse/BEAM-2973
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Blocker
>
> Jenkins seems to be failing nearly immediately this morning for multiple 
> builds. The error is:
> ERROR: Failed to parse POMs
> java.io.IOException: remote file operation failed: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall at 
> hudson.remoting.Channel@7b1b4155:beam7: hudson.remoting.ProxyException: 
> hudson.maven.MavenModuleSetBuild$MavenExecutionException: 
> org.apache.maven.project.ProjectBuildingException: Some problems were 
> encountered while processing the POMs:
> [ERROR] 'dependencies.dependency.version' for 
> org.apache.beam:beam-runners-flink_2.10:jar is missing. @ line 68, column 21
> [WARNING] 'artifactId' contains an expression but should be a constant. @ 
> org.apache.beam:beam-runners-flink_${flink.scala.version}:2.2.0-SNAPSHOT, 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/flink/pom.xml,
>  line 29, column 15
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/14497/console
> I suspect this commit is causing the issue:
> https://github.com/apache/beam/commit/ab975317e1aa532053b68ccc105e13afff0c0b1a



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3873: [BEAM-2973] Reverting change to attempt to fix Jenk...

[youngoli]

GitHub user youngoli opened a pull request:

https://github.com/apache/beam/pull/3873

[BEAM-2973] Reverting change to attempt to fix Jenkins PreCommit

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

This reverts commit ab975317e1aa532053b68ccc105e13afff0c0b1a. That commit 
seems to be causing issues with Jenkins for whatever reason.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/youngoli/beam bugfix-beam2973

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3873.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3873


commit e698efc634f35c72af409577eccc2b541b379cba
Author: Daniel Oliveira 
Date:   2017-09-20T20:46:22Z

[BEAM-2973] Revert "[BEAM-2377] Allow cross compilation (2.10,2.11) for 
flink runner"

This reverts commit ab975317e1aa532053b68ccc105e13afff0c0b1a.




---

[GitHub] beam pull request #3868: Add Nullable Metric getter to MetricsContainerImpl.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3868


---

[2/2] beam git commit: This closes #3868

[tgroh]

This closes #3868


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/59542c3c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/59542c3c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/59542c3c

Branch: refs/heads/master
Commit: 59542c3cc796ac2aa2976ea17d385b141c1b0b38
Parents: a267ca8 8a25597
Author: Thomas Groh 
Authored: Wed Sep 20 13:34:59 2017 -0700
Committer: Thomas Groh 
Committed: Wed Sep 20 13:34:59 2017 -0700

--
 .../core/metrics/MetricsContainerImpl.java  | 40 
 .../core/metrics/MetricsContainerImplTest.java  | 10 +
 2 files changed, 50 insertions(+)
--

[jira] [Created] (BEAM-2974) Error when multiple copies of a PCollection are materialized (e.g. in multiple components of a returned tuple) in Eager Runner

[Robert Bradshaw (JIRA)]

Robert Bradshaw created BEAM-2974:
-

 Summary: Error when multiple copies of a PCollection are 
materialized (e.g. in multiple components of a returned tuple) in Eager Runner
 Key: BEAM-2974
 URL: https://issues.apache.org/jira/browse/BEAM-2974
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Robert Bradshaw
Assignee: Robert Bradshaw
 Fix For: 2.2.0






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2974) Error when multiple copies of a PCollection are materialized (e.g. in multiple components of a returned tuple) in Eager Runner

[Robert Bradshaw (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Bradshaw resolved BEAM-2974.
---
Resolution: Fixed

Also fixed as part of https://github.com/apache/beam/pull/3831

> Error when multiple copies of a PCollection are materialized (e.g. in 
> multiple components of a returned tuple) in Eager Runner
> --
>
> Key: BEAM-2974
> URL: https://issues.apache.org/jira/browse/BEAM-2974
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2973) Jenkins PreCommit broken due to missing dependency

[Daniel Oliveira (JIRA)]

Daniel Oliveira created BEAM-2973:
-

 Summary: Jenkins PreCommit broken due to missing dependency
 Key: BEAM-2973
 URL: https://issues.apache.org/jira/browse/BEAM-2973
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Daniel Oliveira
Assignee: Daniel Oliveira
Priority: Blocker


Jenkins seems to be failing nearly immediately this morning for multiple 
builds. The error is:

ERROR: Failed to parse POMs
java.io.IOException: remote file operation failed: 
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall at 
hudson.remoting.Channel@7b1b4155:beam7: hudson.remoting.ProxyException: 
hudson.maven.MavenModuleSetBuild$MavenExecutionException: 
org.apache.maven.project.ProjectBuildingException: Some problems were 
encountered while processing the POMs:
[ERROR] 'dependencies.dependency.version' for 
org.apache.beam:beam-runners-flink_2.10:jar is missing. @ line 68, column 21
[WARNING] 'artifactId' contains an expression but should be a constant. @ 
org.apache.beam:beam-runners-flink_${flink.scala.version}:2.2.0-SNAPSHOT, 
/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/flink/pom.xml,
 line 29, column 15

https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/14497/console

I suspect this commit is causing the issue:
https://github.com/apache/beam/commit/ab975317e1aa532053b68ccc105e13afff0c0b1a



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[1/2] beam git commit: Add Nullable getters to MetricsContainerImpl

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master a267ca89e -> 59542c3cc


Add Nullable getters to MetricsContainerImpl


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8a25597e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8a25597e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8a25597e

Branch: refs/heads/master
Commit: 8a25597eb5da41f2174dfcf4cea8d0fc230603e4
Parents: a267ca8
Author: Sunil Pedapudi 
Authored: Tue Sep 19 14:55:43 2017 -0700
Committer: Thomas Groh 
Committed: Wed Sep 20 13:33:20 2017 -0700

--
 .../core/metrics/MetricsContainerImpl.java  | 40 
 .../core/metrics/MetricsContainerImplTest.java  | 10 +
 2 files changed, 50 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/8a25597e/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
index 4b331e0..1d5ad72 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/MetricsContainerImpl.java
@@ -23,6 +23,7 @@ import static 
com.google.common.base.Preconditions.checkNotNull;
 import com.google.common.collect.ImmutableList;
 import java.io.Serializable;
 import java.util.Map;
+import javax.annotation.Nullable;
 import org.apache.beam.runners.core.construction.metrics.MetricKey;
 import org.apache.beam.runners.core.metrics.MetricUpdates.MetricUpdate;
 import org.apache.beam.sdk.annotations.Experimental;
@@ -80,21 +81,60 @@ public class MetricsContainerImpl implements Serializable, 
MetricsContainer {
 this.stepName = stepName;
   }
 
+  /**
+   * Return a {@code CounterCell} named {@code metricName}. If it doesn't 
exist, create a
+   * {@code Metric} with the specified name.
+   */
   @Override
   public CounterCell getCounter(MetricName metricName) {
 return counters.get(metricName);
   }
 
+  /**
+   * Return a {@code CounterCell} named {@code metricName}. If it doesn't 
exist, return
+   * {@code null}.
+   */
+  @Nullable
+  public CounterCell tryGetCounter(MetricName metricName) {
+return counters.tryGet(metricName);
+  }
+
+  /**
+   * Return a {@code DistributionCell} named {@code metricName}. If it doesn't 
exist, create a
+   * {@code Metric} with the specified name.
+   */
   @Override
   public DistributionCell getDistribution(MetricName metricName) {
 return distributions.get(metricName);
   }
 
+  /**
+   * Return a {@code DistributionCell} named {@code metricName}. If it doesn't 
exist, return
+   * {@code null}.
+   */
+  @Nullable
+  public DistributionCell tryGetDistribution(MetricName metricName) {
+return distributions.tryGet(metricName);
+  }
+
+  /**
+   * Return a {@code GaugeCell} named {@code metricName}. If it doesn't exist, 
create a
+   * {@code Metric} with the specified name.
+   */
   @Override
   public GaugeCell getGauge(MetricName metricName) {
 return gauges.get(metricName);
   }
 
+  /**
+   * Return a {@code GaugeCell} named {@code metricName}. If it doesn't exist, 
return
+   * {@code null}.
+   */
+  @Nullable
+  public GaugeCell tryGetGauge(MetricName metricName) {
+return gauges.tryGet(metricName);
+  }
+
   private 
   ImmutableList extractUpdates(MetricsMap cells) {
 ImmutableList.Builder updates = 
ImmutableList.builder();

http://git-wip-us.apache.org/repos/asf/beam/blob/8a25597e/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsContainerImplTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsContainerImplTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsContainerImplTest.java
index b304d3b..ab4b709 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsContainerImplTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/MetricsContainerImplTest.java
@@ -22,6 +22,7 @@ import static 
org.apache.beam.runners.core.metrics.MetricUpdateMatchers.metricUp
 import static org.hamcrest.Matchers.contains;
 import static org.hamcrest.Matchers.emptyIterable;
 import static 
org.hamcrest.collection.IsIterableContainingInAnyOrder.containsInAnyOrder;
+import static org.junit.Assert.assertEquals;
 import static 

Build failed in Jenkins: beam_PerformanceTests_JDBC #213

[Apache Jenkins Server]

See 


Changes:

[kirpichov] Marks TikaIO as experimental to allow backward-incompatible changes

[aljoscha.krettek] [BEAM-2377] Allow cross compilation (2.10,2.11) for flink 
runner

[tgroh] Add PipelineOptionsTranslation

--
[...truncated 142.28 KB...]
unable to connect to a server to handle "replicationcontrollers": failed to 
negotiate an api version; server supports: map[], client supports: map[v1:{} 
metrics/v1alpha1:{} extensions/v1beta1:{} componentconfig/v1alpha1:{} 
batch/v1:{} autoscaling/v1:{} authorization.k8s.io/v1beta1:{}]

2017-09-20 20:09:50,317 406b4cd6 MainThread beam_integration_benchmark(1/1) 
ERRORRetrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2017-09-20 20:10:13,851 406b4cd6 MainThread beam_integration_benchmark(1/1) 
INFO Running: /usr/lib/google-cloud-sdk/bin/kubectl 
--kubeconfig=/home/jenkins/.kube/config delete -f 

2017-09-20 20:10:13,882 406b4cd6 MainThread beam_integration_benchmark(1/1) 
INFO Ran /usr/lib/google-cloud-sdk/bin/kubectl 
--kubeconfig=/home/jenkins/.kube/config delete -f 

 Got return code (1).
STDOUT: 
STDERR: unable to connect to a server to handle "services": failed to negotiate 
an api version; server supports: map[], client supports: map[batch/v1:{} 
autoscaling/v1:{} authorization.k8s.io/v1beta1:{} v1:{} metrics/v1alpha1:{} 
extensions/v1beta1:{} componentconfig/v1alpha1:{}]
unable to connect to a server to handle "replicationcontrollers": failed to 
negotiate an api version; server supports: map[], client supports: 
map[authorization.k8s.io/v1beta1:{} v1:{} metrics/v1alpha1:{} 
extensions/v1beta1:{} componentconfig/v1alpha1:{} batch/v1:{} autoscaling/v1:{}]

2017-09-20 20:10:13,882 406b4cd6 MainThread beam_integration_benchmark(1/1) 
ERRORRetrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2017-09-20 20:10:34,507 406b4cd6 MainThread beam_integration_benchmark(1/1) 
INFO Running: /usr/lib/google-cloud-sdk/bin/kubectl 
--kubeconfig=/home/jenkins/.kube/config delete -f 

2017-09-20 20:10:34,536 406b4cd6 MainThread beam_integration_benchmark(1/1) 
INFO Ran /usr/lib/google-cloud-sdk/bin/kubectl 
--kubeconfig=/home/jenkins/.kube/config delete -f 

 Got return code (1).
STDOUT: 
STDERR: unable to connect to a server to handle "services": failed to negotiate 
an api version; server supports: map[], client supports: 
map[metrics/v1alpha1:{} extensions/v1beta1:{} componentconfig/v1alpha1:{} 
batch/v1:{} autoscaling/v1:{} authorization.k8s.io/v1beta1:{} v1:{}]
unable to connect to a server to handle "replicationcontrollers": failed to 
negotiate an api version; server supports: map[], client supports: 
map[componentconfig/v1alpha1:{} batch/v1:{} autoscaling/v1:{} 
authorization.k8s.io/v1beta1:{} v1:{} metrics/v1alpha1:{} extensions/v1beta1:{}]

2017-09-20 20:10:34,536 406b4cd6 MainThread beam_integration_benchmark(1/1) 
ERRORRetrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2017-09-20 20:10:51,218 406b4cd6 MainThread beam_integration_benchmark(1/1) 
INFO Running: /usr/lib/google-cloud-sdk/bin/kubectl 
--kubeconfig=/home/jenkins/.kube/config delete -f 

2017-09-20 20:10:51,248 406b4cd6 MainThread beam_integration_benchmark(1/1) 
INFO Ran /usr/lib/google-cloud-sdk/bin/kubectl 
--kubeconfig=/home/jenkins/.kube/config delete -f 

 Got return code (1).
STDOUT: 
STDERR: unable to connect to a server to handle "services": failed to negotiate 
an api version; server supports: map[], client supports: map[v1:{} 
metrics/v1alpha1:{} extensions/v1beta1:{} componentconfig/v1alpha1:{} 
batch/v1:{} autoscaling/v1:{} authorization.k8s.io/v1beta1:{}]
unable to connect to a server to handle "replicationcontrollers": failed to 
negotiate an api version; server supports: map[], client supports: 
map[batch/v1:{} autoscaling/v1:{} authorization.k8s.io/v1beta1:{} v1:{} 
metrics/v1alpha1:{} extensions/v1beta1:{} componentconfig/v1alpha1:{}]

2017-09-20 20:10:51,248 406b4cd6 MainThread beam_integration_benchmark(1/1) 
ERRORRetrying exception running IssueRetryableCommand: Command returned a 
non-zero exit code.

2017-09-20 20:11:17,902 406b4cd6 MainThread 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Apex #2429

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4834

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3872: Exclude incompatible six release, part 2

[charlesccychen]

GitHub user charlesccychen opened a pull request:

https://github.com/apache/beam/pull/3872

Exclude incompatible six release, part 2

R: @robertwb 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/beam fix-six-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3872.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3872


commit cb7e2a75d731334c445149df5bcdf10da4462005
Author: Charles Chen 
Date:   2017-09-20T19:26:39Z

Exclude incompatible six release, part 2




---

Jenkins build is back to normal : beam_PostCommit_Python_Verify #3174

[Apache Jenkins Server]

See 

beam git commit: [BEAM-2964] Exclude incompatible six release.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/release-2.1.1 656deff3e -> de56f6b28


[BEAM-2964] Exclude incompatible six release.

Upstream bugs being tracked at https://github.com/google/apitools/issues/175 
and https://github.com/benjaminp/six/issues/210


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/de56f6b2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/de56f6b2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/de56f6b2

Branch: refs/heads/release-2.1.1
Commit: de56f6b28cb0a8f5cded1fa7601957c9fcee6247
Parents: 656deff
Author: Robert Bradshaw 
Authored: Tue Sep 19 10:41:43 2017 -0700
Committer: Robert Bradshaw 
Committed: Wed Sep 20 12:09:04 2017 -0700

--
 sdks/python/setup.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/de56f6b2/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index da82466..25ea5c7 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -113,7 +113,9 @@ REQUIRED_SETUP_PACKAGES = [
 REQUIRED_TEST_PACKAGES = [
 'pyhamcrest>=1.9,<2.0',
 # Six required by nose plugins management.
-'six>=1.9',
+# Six 1.11.0 incompatible with apitools.
+# TODO(BEAM-2964): Remove the upper bound.
+'six>=1.9,<1.11',
 ]
 
 GCP_REQUIREMENTS = [

[beam] Git Push Summary

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/release-2.1.1 [created] 656deff3e

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4835

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add PipelineOptionsTranslation

--
[...truncated 1.37 MB...]
2017-09-20T19:05:50.372 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-auth/2.5.1/hadoop-auth-2.5.1.jar
2017-09-20T19:05:50.375 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar
 (76 KB at 105.2 KB/sec)
2017-09-20T19:05:50.375 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.jar
2017-09-20T19:05:50.391 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/lmax/disruptor/3.3.0/disruptor-3.3.0.jar
 (78 KB at 106.6 KB/sec)
2017-09-20T19:05:50.391 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop-compat/1.2.6/hbase-hadoop-compat-1.2.6-tests.jar
2017-09-20T19:05:50.402 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-auth/2.5.1/hadoop-auth-2.5.1.jar
 (52 KB at 69.2 KB/sec)
2017-09-20T19:05:50.402 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop2-compat/1.2.6/hbase-hadoop2-compat-1.2.6-tests.jar
2017-09-20T19:05:50.421 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop-compat/1.2.6/hbase-hadoop-compat-1.2.6-tests.jar
 (20 KB at 25.1 KB/sec)
2017-09-20T19:05:50.421 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-annotations/1.2.6/hbase-annotations-1.2.6.jar
2017-09-20T19:05:50.432 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.jar
 (424 KB at 549.6 KB/sec)
2017-09-20T19:05:50.433 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop2-compat/1.2.6/hbase-hadoop2-compat-1.2.6-tests.jar
 (30 KB at 38.0 KB/sec)
2017-09-20T19:05:50.450 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-annotations/1.2.6/hbase-annotations-1.2.6.jar
 (21 KB at 25.9 KB/sec)
2017-09-20T19:05:51.854 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-shaded-client/1.2.6/hbase-shaded-client-1.2.6.jar
 (29957 KB at 13647.7 KB/sec)
[JENKINS] Archiving disabled
2017-09-20T19:05:53.920 [INFO]  
   
2017-09-20T19:05:53.920 [INFO] 

2017-09-20T19:05:53.920 [INFO] Skipping Apache Beam :: Parent
2017-09-20T19:05:53.920 [INFO] This project has been banned from the build due 
to previous failures.
2017-09-20T19:05:53.920 [INFO] 

[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
2017-09-20T19:06:31.796 [INFO] 

2017-09-20T19:06:31.796 [INFO] Reactor Summary:
2017-09-20T19:06:31.796 [INFO] 
2017-09-20T19:06:31.797 [INFO] Apache Beam :: Parent 
.. SUCCESS [ 48.633 s]
2017-09-20T19:06:31.797 [INFO] Apache 

[jira] [Commented] (BEAM-2431) Model Runner interactions in RPC layer for Runner API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173648#comment-16173648
 ] 

ASF GitHub Bot commented on BEAM-2431:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3719


> Model Runner interactions in RPC layer for Runner API
> -
>
> Key: BEAM-2431
> URL: https://issues.apache.org/jira/browse/BEAM-2431
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Sourabh Bajaj
>  Labels: portability
> Fix For: 2.2.0
>
>
> The "Runner API" today is actually just a definition of what constitutes a 
> Beam pipeline. It needs to actually be a (very small) API.
> This would allow e.g. a Java-based job launcher to respond to launch requests 
> and state queries from a Python-based adapter.
> The expected API would be something like a distillation of the APIs for 
> PipelineRunner and PipelineResult (which is really "Job") via analyzing how 
> these both look in Java and Python.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3719: [BEAM-2431] Add PipelineOptionsTranslation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3719


---

[2/2] beam git commit: Add PipelineOptionsTranslation

[tgroh]

Add PipelineOptionsTranslation

This converts a PipelineOptions instance to and from a Protobuf Struct.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f7ebb620
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f7ebb620
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f7ebb620

Branch: refs/heads/master
Commit: f7ebb6201b5f6e0bb3c585733b6c934eef62c68b
Parents: a1835c6
Author: Thomas Groh 
Authored: Tue Aug 15 13:22:01 2017 -0700
Committer: Thomas Groh 
Committed: Wed Sep 20 10:33:30 2017 -0700

--
 .../PipelineOptionsTranslation.java |  51 +++
 .../PipelineOptionsTranslationTest.java | 143 +++
 2 files changed, 194 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f7ebb620/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
new file mode 100644
index 000..4cdca61
--- /dev/null
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslation.java
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.core.construction;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.protobuf.Struct;
+import com.google.protobuf.util.JsonFormat;
+import java.io.IOException;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.util.common.ReflectHelpers;
+
+/** Utilities for going to/from Runner API pipeline options. */
+public class PipelineOptionsTranslation {
+  private static final ObjectMapper MAPPER =
+  new ObjectMapper()
+  
.registerModules(ObjectMapper.findModules(ReflectHelpers.findClassLoader()));
+
+  /** Converts the provided {@link PipelineOptions} to a {@link Struct}. */
+  public static Struct toProto(PipelineOptions options) {
+Struct.Builder builder = Struct.newBuilder();
+try {
+  // The JSON format of a Protobuf Struct is the JSON object that is 
equivalent to that struct
+  // (with values encoded in a standard json-codeable manner). See Beam PR 
3719 for more.
+  JsonFormat.parser().merge(MAPPER.writeValueAsString(options), builder);
+  return builder.build();
+} catch (IOException e) {
+  throw new RuntimeException(e);
+}
+  }
+
+  /** Converts the provided {@link Struct} into {@link PipelineOptions}. */
+  public static PipelineOptions fromProto(Struct protoOptions) throws 
IOException {
+return MAPPER.readValue(JsonFormat.printer().print(protoOptions), 
PipelineOptions.class);
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/f7ebb620/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslationTest.java
--
diff --git 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslationTest.java
 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslationTest.java
new file mode 100644
index 000..eb59bac
--- /dev/null
+++ 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineOptionsTranslationTest.java
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file 

[1/2] beam git commit: This closes #3719

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master a1835c619 -> a267ca89e


This closes #3719


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a267ca89
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a267ca89
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a267ca89

Branch: refs/heads/master
Commit: a267ca89e6700402069992597253d761dd382574
Parents: a1835c6 f7ebb62
Author: Thomas Groh 
Authored: Wed Sep 20 10:33:30 2017 -0700
Committer: Thomas Groh 
Committed: Wed Sep 20 10:33:30 2017 -0700

--
 .../PipelineOptionsTranslation.java |  51 +++
 .../PipelineOptionsTranslationTest.java | 143 +++
 2 files changed, 194 insertions(+)
--

[jira] [Commented] (BEAM-2968) Master build broken due to error when running autocomplete_test.py

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173499#comment-16173499
 ] 

Xu Mingmin commented on BEAM-2968:
--

seems the error is gone, [~aviemzur].

> Master build broken due to error when running autocomplete_test.py
> --
>
> Key: BEAM-2968
> URL: https://issues.apache.org/jira/browse/BEAM-2968
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Aviem Zur
>Assignee: Ahmet Altay
>
> Master build fails with:
> {code}
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.codehaus.mojo:exec-maven-plugin:1.5.0:exec (setuptools-test) on 
> project beam-sdks-python: Command execution failed.
> {code}
> Last nightly build: 
> https://builds.apache.org/job/beam_Release_NightlySnapshot/536/
> All recent pre-commit builds: 
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PreCommit_Java_MavenInstall/
> {code}
> py27gcp runtests: commands[3] | python 
> apache_beam/examples/complete/autocomplete_test.py
> Traceback (most recent call last):
>   File "apache_beam/examples/complete/autocomplete_test.py", line 22, in 
> 
> import apache_beam as beam
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/__init__.py",
>  line 87, in 
> from apache_beam import io
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/io/__init__.py",
>  line 21, in 
> from apache_beam.io.avroio import *
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/io/avroio.py",
>  line 56, in 
> from apache_beam.io import filebasedsink
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/io/filebasedsink.py",
>  line 32, in 
> from apache_beam.io.filesystems import FileSystems
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/io/filesystems.py",
>  line 30, in 
> from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/io/gcp/gcsfilesystem.py",
>  line 27, in 
> from apache_beam.io.gcp import gcsio
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/io/gcp/gcsio.py",
>  line 37, in 
> from apache_beam.utils import retry
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/apache_beam/utils/retry.py",
>  line 40, in 
> from apitools.base.py.exceptions import HttpError
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/target/.tox/py27gcp/local/lib/python2.7/site-packages/apitools/base/py/__init__.py",
>  line 21, in 
> from apitools.base.py.base_api import *
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/target/.tox/py27gcp/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
>  line 31, in 
> from apitools.base.protorpclite import message_types
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/target/.tox/py27gcp/local/lib/python2.7/site-packages/apitools/base/protorpclite/message_types.py",
>  line 25, in 
> from apitools.base.protorpclite import messages
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/target/.tox/py27gcp/local/lib/python2.7/site-packages/apitools/base/protorpclite/messages.py",
>  line 1165, in 
> class Field(six.with_metaclass(_FieldMeta, object)):
> TypeError: Error when calling the metaclass bases
> metaclass conflict: the metaclass of a derived class must be a 
> (non-strict) subclass of the metaclasses of all its bases
> ERROR: InvocationError: 
> '/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/python/target/.tox/py27gcp/bin/python
>  apache_beam/examples/complete/autocomplete_test.py'
> ___ summary 
> 
> ERROR:   docs: commands failed
>   lint: commands succeeded
>   py27: commands succeeded
>   py27cython: commands succeeded
> ERROR:   py27gcp: commands failed
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2966) Allow subclasses of tuple, list, dict as pvalues.

[Robert Bradshaw (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Bradshaw resolved BEAM-2966.
---
Resolution: Fixed

> Allow subclasses of tuple, list, dict as pvalues.
> -
>
> Key: BEAM-2966
> URL: https://issues.apache.org/jira/browse/BEAM-2966
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2966) Allow subclasses of tuple, list, dict as pvalues.

[Robert Bradshaw (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Bradshaw updated BEAM-2966:
--
Fix Version/s: 2.2.0

> Allow subclasses of tuple, list, dict as pvalues.
> -
>
> Key: BEAM-2966
> URL: https://issues.apache.org/jira/browse/BEAM-2966
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[Jacob Marble (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173410#comment-16173410
 ] 

Jacob Marble edited comment on BEAM-2500 at 9/20/17 4:04 PM:
-

Hmm, I think there's a bug in my code. Please ignore for now.


was (Author: jmarble):
Hmm, I think there's a bug. Please ignore for now.

> Add support for S3 as a Apache Beam FileSystem
> --
>
> Key: BEAM-2500
> URL: https://issues.apache.org/jira/browse/BEAM-2500
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Luke Cwik
>Priority: Minor
> Attachments: hadoop_fs_patch.patch
>
>
> Note that this is for providing direct integration with S3 as an Apache Beam 
> FileSystem.
> There is already support for using the Hadoop S3 connector by depending on 
> the Hadoop File System module[1], configuring HadoopFileSystemOptions[2] with 
> a S3 configuration[3].
> 1: https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system
> 2: 
> https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.java#L53
> 3: https://wiki.apache.org/hadoop/AmazonS3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[Jacob Marble (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173410#comment-16173410
 ] 

Jacob Marble commented on BEAM-2500:


Hmm, I think there's a bug. Please ignore for now.

> Add support for S3 as a Apache Beam FileSystem
> --
>
> Key: BEAM-2500
> URL: https://issues.apache.org/jira/browse/BEAM-2500
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Luke Cwik
>Priority: Minor
> Attachments: hadoop_fs_patch.patch
>
>
> Note that this is for providing direct integration with S3 as an Apache Beam 
> FileSystem.
> There is already support for using the Hadoop S3 connector by depending on 
> the Hadoop File System module[1], configuring HadoopFileSystemOptions[2] with 
> a S3 configuration[3].
> 1: https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system
> 2: 
> https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.java#L53
> 3: https://wiki.apache.org/hadoop/AmazonS3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[Jacob Marble (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173403#comment-16173403
 ] 

Jacob Marble commented on BEAM-2500:


I see this warning thousands of times when reading from S3:

Sep 20, 2017 8:45:04 AM 
com.amazonaws.services.s3.internal.S3AbortableInputStream close
WARNING: Not all bytes were read from the S3ObjectInputStream, aborting HTTP 
connection. This is likely an error and may result in sub-optimal behavior. 
Request only the bytes you need via a ranged GET or drain the input stream 
after use.

It looks like TextIO requests bytes n through m, but only consumes less than 
m-n bytes, then closes the channel (channel wraps a stream). Am I wrong? Is m-n 
predictably small enough that I should drain the stream at close?

> Add support for S3 as a Apache Beam FileSystem
> --
>
> Key: BEAM-2500
> URL: https://issues.apache.org/jira/browse/BEAM-2500
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Luke Cwik
>Priority: Minor
> Attachments: hadoop_fs_patch.patch
>
>
> Note that this is for providing direct integration with S3 as an Apache Beam 
> FileSystem.
> There is already support for using the Hadoop S3 connector by depending on 
> the Hadoop File System module[1], configuring HadoopFileSystemOptions[2] with 
> a S3 configuration[3].
> 1: https://github.com/apache/beam/tree/master/sdks/java/io/hadoop-file-system
> 2: 
> https://github.com/apache/beam/blob/master/sdks/java/io/hadoop-file-system/src/main/java/org/apache/beam/sdk/io/hdfs/HadoopFileSystemOptions.java#L53
> 3: https://wiki.apache.org/hadoop/AmazonS3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2966) Allow subclasses of tuple, list, dict as pvalues.

[Konstantinos Katsiapis (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173400#comment-16173400
 ] 

Konstantinos Katsiapis commented on BEAM-2966:
--

[~reuvenlax][~robertwb]: Is this fixed and will it be included in the Beam 
2.2.0 release?

> Allow subclasses of tuple, list, dict as pvalues.
> -
>
> Key: BEAM-2966
> URL: https://issues.apache.org/jira/browse/BEAM-2966
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2701) use a custom implementation of java.io.ObjectInputStream

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173349#comment-16173349
 ] 

ASF GitHub Bot commented on BEAM-2701:
--

GitHub user rmannibucau opened a pull request:

https://github.com/apache/beam/pull/3871

[BEAM-2701] ensure objectinputstream uses the right classloader for 
serialization

When the PTransform are not loaded with the app classloader the 
ensureSerializable code can end up through ObjectInputStream on 
vmLatestUserDefinedClassLoader which likely falls back on app classloader 
whereas it should use the TCCL in resolveClass.

This PR ensures:

1. we use the TCCL as expected to deserialize an instance
2. uses the serialized instance classloader contextually in 
ensureSerializable to tolerate cross classloader usage

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmannibucau/incubator-beam 
fix/serialization-classloader

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3871.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3871


commit 0c3d743f50a8c40fb1f064886996941af37fb8c8
Author: Romain Manni-Bucau 
Date:   2017-09-20T15:29:53Z

ensure objectinputstream uses the right classloader for serialization




> use a custom implementation of java.io.ObjectInputStream
> 
>
> Key: BEAM-2701
> URL: https://issues.apache.org/jira/browse/BEAM-2701
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Luke Cwik
>
> java.io.ObjectInputStream should override resolve[Proxy]Class using the TCCL 
> to support any classloader and not fallback into some JVM pitfall using 
> another classloader (default). This will enable beam to use any classloader 
> instead of requiring to run in the JVM using java serialization.
> {code}
> @Override
> protected Class resolveClass(final ObjectStreamClass classDesc) throws 
> IOException, ClassNotFoundException {
> final String n = classDesc.getName();
> final ClassLoader classloader = getClassloader();
> try {
> return Class.forName(n, false, classloader);
> } catch (ClassNotFoundException e) {
> if (n.equals("boolean")) {
> return boolean.class;
> }
> if (n.equals("byte")) {
> return byte.class;
> }
> if (n.equals("char")) {
> return char.class;
> }
> if (n.equals("short")) {
> return short.class;
> }
> if (n.equals("int")) {
> return int.class;
> }
> if (n.equals("long")) {
> return long.class;
> }
> if (n.equals("float")) {
> return float.class;
> }
> if (n.equals("double")) {
> return double.class;
> }
> //Last try - Let runtime try and find it.
> return Class.forName(n, false, null);
> }
> }
> @Override
> protected Class resolveProxyClass(final String[] interfaces) throws 
> IOException, ClassNotFoundException {
> final Class[] cinterfaces = new Class[interfaces.length];
> for (int i = 0; i < interfaces.length; i++) {
> cinterfaces[i] = getClassloader().loadClass(interfaces[i]);
> }
> try {
> return Proxy.getProxyClass(getClassloader(), cinterfaces);
> } catch (IllegalArgumentException e) {
> throw new ClassNotFoundException(null, e);
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3871: [BEAM-2701] ensure objectinputstream uses the right...

[rmannibucau]

GitHub user rmannibucau opened a pull request:

https://github.com/apache/beam/pull/3871

[BEAM-2701] ensure objectinputstream uses the right classloader for 
serialization

When the PTransform are not loaded with the app classloader the 
ensureSerializable code can end up through ObjectInputStream on 
vmLatestUserDefinedClassLoader which likely falls back on app classloader 
whereas it should use the TCCL in resolveClass.

This PR ensures:

1. we use the TCCL as expected to deserialize an instance
2. uses the serialized instance classloader contextually in 
ensureSerializable to tolerate cross classloader usage

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmannibucau/incubator-beam 
fix/serialization-classloader

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3871.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3871


commit 0c3d743f50a8c40fb1f064886996941af37fb8c8
Author: Romain Manni-Bucau 
Date:   2017-09-20T15:29:53Z

ensure objectinputstream uses the right classloader for serialization




---

Build failed in Jenkins: beam_PostCommit_Python_Verify #3173

[Apache Jenkins Server]

See 


--
[...truncated 43.93 KB...]
Collecting crcmod<2.0,>=1.7 (from apache-beam==2.2.0.dev0)
Collecting dill==0.2.6 (from apache-beam==2.2.0.dev0)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
apache-beam==2.2.0.dev0)
Collecting httplib2<0.10,>=0.8 (from apache-beam==2.2.0.dev0)
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.2.0.dev0)
  Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting oauth2client<4.0.0,>=2.0.1 (from apache-beam==2.2.0.dev0)
Collecting protobuf<=3.3.0,>=3.2.0 (from apache-beam==2.2.0.dev0)
  Using cached protobuf-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.2.0.dev0)
Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.2.0.dev0)
  Using cached typing-3.6.2-py2-none-any.whl
Requirement already satisfied: six>=1.5.2 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Requirement already satisfied: futures>=2.2.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1_modules-0.1.4-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1-0.3.5-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Requirement already satisfied: setuptools in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
protobuf<=3.3.0,>=3.2.0->apache-beam==2.2.0.dev0)
Building wheels for collected packages: apache-beam
  Running setup.py bdist_wheel for apache-beam: started
  Running setup.py bdist_wheel for apache-beam: finished with status 'error'
  Complete output from command 

 -u -c "import setuptools, 
tokenize;__file__='/tmp/pip-k1R8e0-build/setup.py';f=getattr(tokenize, 'open', 
open)(__file__);code=f.read().replace('\r\n', 
'\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d 
/tmp/tmpwg9zobpip-wheel- --python-tag cp27:
  
:351:
 UserWarning: Normalizing '2.2.0.dev' to '2.2.0.dev0'
normalized_version,
  running bdist_wheel
  running build
  running build_py
  Traceback (most recent call last):
File "", line 1, in 
File "/tmp/pip-k1R8e0-build/setup.py", line 200, in 
  'test': generate_protos_first(test),
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
  dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
  self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File 
"
 line 204, in run
  self.run_command('build')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
  self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/tmp/pip-k1R8e0-build/setup.py", line 140, in run
  gen_protos.generate_proto_files()
File "gen_protos.py", line 65, in generate_proto_files
  'Not in apache git tree; unable to find proto definitions.')
  RuntimeError: Not in apache git tree; unable to find proto definitions.
  
  
  Failed building wheel for apache-beam
  Running setup.py clean for apache-beam
Failed to build apache-beam
Installing collected packages: avro, crcmod, dill, httplib2, funcsigs, pbr, 
mock, pyasn1, pyasn1-modules, rsa, oauth2client, protobuf, pyyaml, typing, 
apache-beam
  Found existing installation: protobuf 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4833

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4832

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #3172

[Apache Jenkins Server]

See 


--
[...truncated 43.93 KB...]
Collecting crcmod<2.0,>=1.7 (from apache-beam==2.2.0.dev0)
Collecting dill==0.2.6 (from apache-beam==2.2.0.dev0)
Requirement already satisfied: grpcio<2.0,>=1.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
apache-beam==2.2.0.dev0)
Collecting httplib2<0.10,>=0.8 (from apache-beam==2.2.0.dev0)
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.2.0.dev0)
  Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting oauth2client<4.0.0,>=2.0.1 (from apache-beam==2.2.0.dev0)
Collecting protobuf<=3.3.0,>=3.2.0 (from apache-beam==2.2.0.dev0)
  Using cached protobuf-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.2.0.dev0)
Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.2.0.dev0)
  Using cached typing-3.6.2-py2-none-any.whl
Requirement already satisfied: futures>=2.2.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Requirement already satisfied: six>=1.5.2 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1_modules-0.1.4-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1-0.3.5-py2.py3-none-any.whl
Requirement already satisfied: setuptools in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
protobuf<=3.3.0,>=3.2.0->apache-beam==2.2.0.dev0)
Building wheels for collected packages: apache-beam
  Running setup.py bdist_wheel for apache-beam: started
  Running setup.py bdist_wheel for apache-beam: finished with status 'error'
  Complete output from command 

 -u -c "import setuptools, 
tokenize;__file__='/tmp/pip-aoFXJz-build/setup.py';f=getattr(tokenize, 'open', 
open)(__file__);code=f.read().replace('\r\n', 
'\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d 
/tmp/tmpjNgjU6pip-wheel- --python-tag cp27:
  
:351:
 UserWarning: Normalizing '2.2.0.dev' to '2.2.0.dev0'
normalized_version,
  running bdist_wheel
  running build
  running build_py
  Traceback (most recent call last):
File "", line 1, in 
File "/tmp/pip-aoFXJz-build/setup.py", line 200, in 
  'test': generate_protos_first(test),
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
  dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
  self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File 
"
 line 204, in run
  self.run_command('build')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
  self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/tmp/pip-aoFXJz-build/setup.py", line 140, in run
  gen_protos.generate_proto_files()
File "gen_protos.py", line 65, in generate_proto_files
  'Not in apache git tree; unable to find proto definitions.')
  RuntimeError: Not in apache git tree; unable to find proto definitions.
  
  
  Failed building wheel for apache-beam
  Running setup.py clean for apache-beam
Failed to build apache-beam
Installing collected packages: avro, crcmod, dill, httplib2, pbr, funcsigs, 
mock, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pyyaml, typing, 
apache-beam
  Found existing installation: protobuf 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4831

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #3170

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2377) Cross compile flink runner to scala 2.11

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172873#comment-16172873
 ] 

ASF GitHub Bot commented on BEAM-2377:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3255


> Cross compile flink runner to scala 2.11
> 
>
> Key: BEAM-2377
> URL: https://issues.apache.org/jira/browse/BEAM-2377
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Ole Langbehn
>Assignee: Aljoscha Krettek
> Fix For: 2.2.0
>
>
> The flink runner is compiled for flink built against scala 2.10. flink cross 
> compiles its scala artifacts against 2.10 and 2.11.
> In order to make it possible to use beam with the flink runner in scala 2.11 
> projects, it would be nice if you could publish the flink runner for 2.11 
> next to 2.10.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2377) Cross compile flink runner to scala 2.11

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-2377.
--
   Resolution: Fixed
Fix Version/s: 2.2.0

Implemented in ab975317e1aa532053b68ccc105e13afff0c0b1a

> Cross compile flink runner to scala 2.11
> 
>
> Key: BEAM-2377
> URL: https://issues.apache.org/jira/browse/BEAM-2377
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Ole Langbehn
>Assignee: Aljoscha Krettek
> Fix For: 2.2.0
>
>
> The flink runner is compiled for flink built against scala 2.10. flink cross 
> compiles its scala artifacts against 2.10 and 2.11.
> In order to make it possible to use beam with the flink runner in scala 2.11 
> projects, it would be nice if you could publish the flink runner for 2.11 
> next to 2.10.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[1/2] beam git commit: [BEAM-2377] Allow cross compilation (2.10, 2.11) for flink runner

[aljoscha]

Repository: beam
Updated Branches:
  refs/heads/master 64123e9d3 -> a1835c619


[BEAM-2377] Allow cross compilation (2.10,2.11) for flink runner

Flink allows being built against scala 2.11. But the Flink Runner did
not.

This commit alleviates that, as well as allowing for ensuring that
builds work against scala 2.11 dependencies. It introduces a
flink.scala.version mvn property that is set to 2.11 as a default, as well as
a mvn profile that overrides the scala version to 2.10.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ab975317
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ab975317
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ab975317

Branch: refs/heads/master
Commit: ab975317e1aa532053b68ccc105e13afff0c0b1a
Parents: 64123e9
Author: Ole Langbehn 
Authored: Wed May 31 09:54:04 2017 +0200
Committer: Aljoscha Krettek 
Committed: Wed Sep 20 09:36:53 2017 +0200

--
 examples/java/pom.xml   |  2 +-
 examples/java8/pom.xml  |  2 +-
 pom.xml | 16 +++-
 runners/flink/pom.xml   | 14 +++---
 sdks/java/javadoc/pom.xml   |  2 +-
 .../src/main/resources/archetype-resources/pom.xml  |  2 +-
 .../src/main/resources/archetype-resources/pom.xml  |  2 +-
 7 files changed, 27 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ab975317/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index ade4cac..817af33 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -95,7 +95,7 @@
   
 
   org.apache.beam
-  beam-runners-flink_2.10
+  beam-runners-flink_${flink.scala.version}
   runtime
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ab975317/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 585d7b8..f27f6df 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -95,7 +95,7 @@
   
 
   org.apache.beam
-  beam-runners-flink_2.10
+  beam-runners-flink_${flink.scala.version}
   runtime
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ab975317/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 236645c..f112c64 100644
--- a/pom.xml
+++ b/pom.xml
@@ -154,6 +154,7 @@
 1.1.4
 0.10.1.0
 1.4
+2.11
 
 1.5.0.Final
 2.0
@@ -363,6 +364,19 @@
 
   
 
+
+
+  flink-scala-2.10
+  
+
+  flink-scala-2.10
+
+  
+  
+2.10
+  
+
+
   
 
   
@@ -606,7 +620,7 @@
 
   
 org.apache.beam
-beam-runners-flink_2.10
+beam-runners-flink_${flink.scala.version}
 ${project.version}
   
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ab975317/runners/flink/pom.xml
--
diff --git a/runners/flink/pom.xml b/runners/flink/pom.xml
index 0ef1931..5c680c8 100644
--- a/runners/flink/pom.xml
+++ b/runners/flink/pom.xml
@@ -26,7 +26,7 @@
 ../pom.xml
   
 
-  beam-runners-flink_2.10
+  beam-runners-flink_${flink.scala.version}
   Apache Beam :: Runners :: Flink
   jar
 
@@ -165,7 +165,7 @@
 
 
   org.apache.flink
-  flink-clients_2.10
+  flink-clients_${flink.scala.version}
   ${flink.version}
 
 
@@ -189,13 +189,13 @@
 
 
   org.apache.flink
-  flink-runtime_2.10
+  flink-runtime_${flink.scala.version}
   ${flink.version}
 
 
 
   org.apache.flink
-  flink-streaming-java_2.10
+  flink-streaming-java_${flink.scala.version}
   ${flink.version}
 
 
@@ -210,7 +210,7 @@
 
 
   org.apache.flink
-  flink-runtime_2.10
+  flink-runtime_${flink.scala.version}
   ${flink.version}
   test-jar
   test
@@ -336,7 +336,7 @@
 
 
   org.apache.flink
-  flink-streaming-java_2.10
+  flink-streaming-java_${flink.scala.version}
   ${flink.version}
   test
   test-jar
@@ -344,7 +344,7 @@
 
 
   org.apache.flink
-  flink-test-utils_2.10
+  flink-test-utils_${flink.scala.version}
   ${flink.version}
   test
   

http://git-wip-us.apache.org/repos/asf/beam/blob/ab975317/sdks/java/javadoc/pom.xml
--
diff --git a/sdks/java/javadoc/pom.xml 

[2/2] beam git commit: This closes #3255

[aljoscha]

This closes #3255


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a1835c61
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a1835c61
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a1835c61

Branch: refs/heads/master
Commit: a1835c6191a480f88f0dc03f69d2c7c31d7ed624
Parents: 64123e9 ab97531
Author: Aljoscha Krettek 
Authored: Wed Sep 20 09:37:44 2017 +0200
Committer: Aljoscha Krettek 
Committed: Wed Sep 20 09:37:44 2017 +0200

--
 examples/java/pom.xml   |  2 +-
 examples/java8/pom.xml  |  2 +-
 pom.xml | 16 +++-
 runners/flink/pom.xml   | 14 +++---
 sdks/java/javadoc/pom.xml   |  2 +-
 .../src/main/resources/archetype-resources/pom.xml  |  2 +-
 .../src/main/resources/archetype-resources/pom.xml  |  2 +-
 7 files changed, 27 insertions(+), 13 deletions(-)
--

[GitHub] beam pull request #3255: [BEAM-2377] Allow usage of scala 2.11 dependencies ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3255


---

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4830

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4005

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2972) Nexmark: create a query that illustrates a simple custom window merge

[Etienne Chauchot (JIRA)]

Etienne Chauchot created BEAM-2972:
--

 Summary: Nexmark: create a query that illustrates a simple custom 
window merge
 Key: BEAM-2972
 URL: https://issues.apache.org/jira/browse/BEAM-2972
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Etienne Chauchot


To replace complex custom window merge in WinningBids query.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2971) Nexmark: migrate WinningBids to state API in place of custom window merge.

[Etienne Chauchot (JIRA)]

Etienne Chauchot created BEAM-2971:
--

 Summary: Nexmark: migrate WinningBids to state API in place of 
custom window merge.
 Key: BEAM-2971
 URL: https://issues.apache.org/jira/browse/BEAM-2971
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Etienne Chauchot


Custom window merge in WinningBids is expensive (builds a few maps). It would 
be simpler to use state API there.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3870: Marks TikaIO as experimental to allow backward-inco...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3870


---

[2/2] beam git commit: This closes #3870

[jbonofre]

This closes #3870


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/64123e9d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/64123e9d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/64123e9d

Branch: refs/heads/master
Commit: 64123e9d3c1fd37515184d40ec03fadbf0b412fb
Parents: e8a5282 7d7baf7
Author: Jean-Baptiste Onofré 
Authored: Wed Sep 20 08:56:18 2017 +0200
Committer: Jean-Baptiste Onofré 
Committed: Wed Sep 20 08:56:18 2017 +0200

--
 .../tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java  | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)
--

[1/2] beam git commit: Marks TikaIO as experimental to allow backward-incompatible changes

[jbonofre]

Repository: beam
Updated Branches:
  refs/heads/master e8a52827e -> 64123e9d3


Marks TikaIO as experimental to allow backward-incompatible changes


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7d7baf7d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7d7baf7d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7d7baf7d

Branch: refs/heads/master
Commit: 7d7baf7d419c873ecb58545c7feaaea25a862c23
Parents: cfbdb61
Author: Eugene Kirpichov 
Authored: Tue Sep 19 19:30:45 2017 -0700
Committer: Eugene Kirpichov 
Committed: Tue Sep 19 19:30:45 2017 -0700

--
 .../tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java  | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7d7baf7d/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java
--
diff --git 
a/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java 
b/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java
index 5d6eea7..4876dcf 100644
--- a/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java
+++ b/sdks/java/io/tika/src/main/java/org/apache/beam/sdk/io/tika/TikaIO.java
@@ -21,7 +21,7 @@ import static 
com.google.common.base.Preconditions.checkNotNull;
 import com.google.auto.value.AutoValue;
 
 import javax.annotation.Nullable;
-
+import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.io.Read.Bounded;
@@ -53,7 +53,10 @@ import org.apache.tika.metadata.Metadata;
  * // A simple Read of a local PDF file (only runs locally):
  * PCollection content = 
p.apply(TikaInput.from("/local/path/to/file.pdf"));
  * }
+ *
+ * Warning: the API of this IO is likely to change in the next release.
  */
+@Experimental(Experimental.Kind.SOURCE_SINK)
 public class TikaIO {
 
   /**

Build failed in Jenkins: beam_PerformanceTests_JDBC #212

[Apache Jenkins Server]

See 


Changes:

[robertwb] Fix typo in variable name: window_fn --> windowfn

[iemejia] [BEAM-2764] Change document size range to fix flakiness on SolrIO 
tests

[lcwik] Add fn API progress reporting protos

[iemejia] [BEAM-2787] Fix MongoDbIO exception when trying to write empty bundle

[lcwik] Upgrade snappy-java to version 1.1.4

[lcwik] Upgrade slf4j to version 1.7.25

[chamikara] Updates bigtable.version to 1.0.0-pre3.

[lcwik] Add ThrowingBiConsumer to the set of functional interfaces

[altay] Added snippet tags for documentation

[altay] Added concrete example for CoGroupByKey snippet

[jbonofre] Add license-maven-plugin and some default license merges

[relax] BEAM-934 Fixed build by fixing firebug error.

[relax] BEAM-934 Enabled firebug after fixing the bug.

[relax] BEAM-934 Fixed code after review.

[jbonofre] [BEAM-2328] Add TikaIO

[mingmxu] Add DSLs module

[mingmxu] [BEAM-301] Initial skeleton for Beam SQL

[mingmxu] checkstyle and rename package

[mingmxu] redesign BeamSqlExpression to execute Calcite SQL expression.

[mingmxu] Fix inconsistent mapping for SQL FLOAT

[mingmxu] [BEAM-2158] Implement the arithmetic operators

[mingmxu] [BEAM-2161] Add support for String operators

[mingmxu] [BEAM-2079] Support TextIO as SQL source/sink

[mingmxu] [BEAM-2006] window support Add support for aggregation: global, HOP,

[mingmxu] Support common-used aggregation functions in SQL, including:  

[mingmxu] [BEAM-2195] Implement conditional operator (CASE)

[mingmxu] [BEAM-2234] Change return type of buildBeamPipeline to

[mingmxu] [BEAM-2149] Improved kafka table implemention.

[mingmxu] support UDF

[mingmxu] update JavaDoc.

[mingmxu] [BEAM-2255] Implement ORDER BY

[mingmxu] [BEAM-2288] Refine DSL interface as design doc of BEAM-2010: 1. rename

[mingmxu] [BEAM-2292] Add BeamPCollectionTable to create table from

[mingmxu] fix NoSuchFieldException

[mingmxu] [BEAM-2309] Implement VALUES and add support for data type CHAR (to be

[mingmxu] [BEAM-2310] Support encoding/decoding of TIME and new DECIMAL data 
type

[mingmxu] DSL interface for Beam SQL

[mingmxu] [BEAM-2329] Add ABS and SQRT math functions

[mingmxu] upgrade to version 2.1.0-SNAPSHOT

[mingmxu] rename SQL to Sql in class name

[mingmxu] [BEAM-2247] Implement date functions in SQL DSL

[mingmxu] [BEAM-2325] Support Set operator: intersect & except

[mingmxu] Add ROUND function on DSL_SQL branch.

[mingmxu] register table for both BeamSql.simpleQuery and BeamSql.query

[mingmxu] Add NOT operator on DSL_SQL branch (plus some refactoring)

[mingmxu] [BEAM-2444] BeamSql: use java standard exception

[mingmxu] [BEAM-2442] BeamSql surface api test.

[mingmxu] [BEAM-2440] BeamSql: reduce visibility

[mingmxu] Remove unused BeamPipelineCreator class

[mingmxu] [BEAM-2443] apply AutoValue to BeamSqlRecordType

[mingmxu] Update filter/project/aggregation tests to use BeamSql

[mingmxu] Remove UnsupportedOperationVisitor, which is currently just a no-op

[mingmxu] restrict the scope of BeamSqlEnv

[mingmxu] Add ACOS, ASIN, ATAN, COS, COT, DEGREES, RADIANS, SIN, TAN, SIGN, LN,

[mingmxu] [BEAM-2477] BeamAggregationRel should use Combine.perKey instead of

[mingmxu] use static table name PCOLLECTION in BeamSql.simpleQuery.

[mingmxu] Small fixes to make the example run in a runner agnostic way: - Add

[mingmxu] [BEAM-2193] Implement FULL, INNER, and OUTER JOIN: - FULL and INNER

[mingmxu] UDAF support: - Adds an abstract class BeamSqlUdaf for defining 
Calcite

[mingmxu] BeamSql: refactor the MockedBeamSqlTable and related tests

[mingmxu] MockedBeamSqlTable -> MockedBoundedTable

[mingmxu] Test unsupported/invalid cases in DSL tests.

[mingmxu] [BEAM-2550] add UnitTest for JOIN in DSL

[mingmxu] support TUMBLE/HOP/SESSION _START function

[mingmxu] Test queries on unbounded PCollections with BeamSql DSL API. Also add

[mingmxu] [BEAM-2564] add integration test for string functions

[mingmxu] CAST operator supporting numeric, date and timestamp types

[mingmxu] POWER function

[mingmxu] support UDF/UDAF in BeamSql

[mingmxu] upgrade pom to 2.2.0-SNAPSHOT

[mingmxu] [BEAM-2560] Add integration test for arithmetic operators.

[mingmxu] cleanup BeamSqlRow

[mingmxu] proposal for new UDF

[mingmxu] [BEAM-2562] Add integration test for logical operators

[mingmxu] [BEAM-2384] CEIL, FLOOR, TRUNCATE, PI, ATAN2 math functions

[mingmxu] [BEAM-2561] add integration test for date functions

[mingmxu] refactor the datetime test to use ExpressionChecker and fix

[mingmxu] [BEAM-2565] add integration test for conditional functions

[mingmxu] rebased, add RAND/RAND_INTEGER

[mingmxu] [BEAM-2621] BeamSqlRecordType -> BeamSqlRowType

[mingmxu] [BEAM-2563] Add integration test for math operators Misc: 1. no SQRT 
in

[mingmxu] [BEAM-2613] add integration test for comparison operators

[mingmxu] remove README.md and update usages in BeamSqlExample

[mingmxu] update pom.xml 

[jira] [Commented] (BEAM-2826) Need to generate a single XML file when write is performed on small amount of data

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172818#comment-16172818
 ] 

Eugene Kirpichov commented on BEAM-2826:


This will be addressed as part of the FileIO.write() effort. However, what Luke 
suggests above will also work in practice as a workaround.

> Need to generate a single XML file when write is performed on small amount of 
> data
> --
>
> Key: BEAM-2826
> URL: https://issues.apache.org/jira/browse/BEAM-2826
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Affects Versions: 2.0.0
>Reporter: Balajee Venkatesh
>Assignee: Eugene Kirpichov
>
> I'm trying to write an XML file where the source is a text file stored in 
> GCS. The code is running fine but instead of a single XML file, it is 
> generating multiple XML files. (No. of XML files seem to follow total no. of 
> records present in source text file). I have observed this scenario while 
> using 'DataflowRunner'.
> When I run the same code in local then two files get generated. First one 
> contains all the records with proper elements and the second one contains 
> only opening and closing root element.
> As I learnt,it is expected that it may produce multiple files: e.g. if the 
> runner chooses to process your data parallelizing it into 3 tasks 
> ("bundles"), you'll get 3 files. Some of the parts may turn out empty in some 
> cases, but the total data written will always add up to the expected data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)