date:20170731

Build failed in Jenkins: beam_PostCommit_Python_Verify #2840

2017-07-31 Thread Apache Jenkins Server

See 


--
[...truncated 558.55 KB...]
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_multiple_empty_outputs 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.6.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_dict_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.6.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest) 
... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.6.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3690

2017-07-31 Thread Apache Jenkins Server

See

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4494

2017-07-31 Thread Apache Jenkins Server

See

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3689

2017-07-31 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PostCommit_Python_Verify #2839

2017-07-31 Thread Apache Jenkins Server

See 


Changes:

[tgroh] Programmatically Create Beam Jenkins View.

--
[...truncated 576.29 KB...]
test_default_value_singleton_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.6.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_flattened_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_empty_singleton_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_multi_valued_singleton_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok

==
ERROR: test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.transforms.ptransform_test.PTransformTest)
--
Traceback (most recent call last):
  File 
"
 line 236, in test_par_do_with_multiple_outputs_and_using_return
pipeline.run()
  File 
"
 line 96, in run
result = super(TestPipeline, self).run()
  File 
"
 line 319, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 328, in run
return self.runner.run(self)
  File 
"
 line 38, in run
self.result = super(TestDataflowRunner, self).run(pipeline)
  File 
"
 line 272, in run
self.dataflow_client.create_job(self.job), self)
  File 
"
 line 168, in wrapper
return fun(*args, **kwargs)
  File 
"
 line 438, in create_job
self.create_job_description(job)
  File 
"
 line 461, in create_job_description
job.options, file_copy=self._gcs_file_copy)
  File 
"
 line 327, in stage_job_resources
setup_options.requirements_file, requirements_cache_path)
  File 
"
 line 261, in _populate_requirements_cache
processes.check_call(cmd_args)
  File 
"
 line 44, in check_call
return subprocess.check_call(*args, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command 
'['
 '-m', 'pip', 'install', '--download', '/tmp/dataflow-requirements-cache', 
'-r', 'postcommit_requirements.txt', '--no-binary', ':all:']' returned non-zero 
exit

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2749

2017-07-31 Thread Apache Jenkins Server

See

[GitHub] beam pull request #3374: Programmatically create Beam view.

2017-07-31 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3374


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Programmatically Create Beam Jenkins View.

2017-07-31 Thread tgroh

Programmatically Create Beam Jenkins View.

Signed-off-by: Jason Kuster 


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8b60fcb6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8b60fcb6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8b60fcb6

Branch: refs/heads/master
Commit: 8b60fcb6ddaa843e9d30f5abe7b29abc9f73f3ad
Parents: 95a2ead
Author: Jason Kuster 
Authored: Thu Jun 15 16:14:16 2017 -0700
Committer: Thomas Groh 
Committed: Mon Jul 31 17:44:52 2017 -0700

--
 .../jenkins/job_beam_ListView_Create.groovy | 42 
 1 file changed, 42 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/8b60fcb6/.test-infra/jenkins/job_beam_ListView_Create.groovy
--
diff --git a/.test-infra/jenkins/job_beam_ListView_Create.groovy 
b/.test-infra/jenkins/job_beam_ListView_Create.groovy
new file mode 100644
index 000..9c304da
--- /dev/null
+++ b/.test-infra/jenkins/job_beam_ListView_Create.groovy
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// This creates a standard List View for Beam. It filters out executors not 
associated with Beam
+// and pulls in all jobs which start with 'beam_'.
+listView('Beam') {
+description('Jenkins builds for the Beam project.')
+filterBuildQueue()
+filterExecutors()
+jobs {
+regex(/beam_.+/)
+}
+jobFilters {
+all()
+}
+columns {
+status()
+weather()
+name()
+lastSuccess()
+lastFailure()
+lastDuration()
+lastBuildConsole()
+buildButton()
+}
+}
+

[1/2] beam git commit: This closes #3374

2017-07-31 Thread tgroh

Repository: beam
Updated Branches:
  refs/heads/master 95a2eadad -> 870551ccc


This closes #3374


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/870551cc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/870551cc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/870551cc

Branch: refs/heads/master
Commit: 870551ccc20a5b47917a1d8aa3e310c95dfd2394
Parents: 95a2ead 8b60fcb
Author: Thomas Groh 
Authored: Mon Jul 31 17:44:52 2017 -0700
Committer: Thomas Groh 
Committed: Mon Jul 31 17:44:52 2017 -0700

--
 .../jenkins/job_beam_ListView_Create.groovy | 42 
 1 file changed, 42 insertions(+)
--

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2748

2017-07-31 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-2702) Dataflow pipeline stalls after autoscaling

2017-07-31 Thread Flavio Fiszman (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108162#comment-16108162
 ] 

Flavio Fiszman commented on BEAM-2702:
--

Hi Johann,
JIRA issues are primarily used for Beam SDK-related issues. The issue here 
seems to be related to the Dataflow Runner in specific, so a better reporting 
mechanism is to send that information to dataflow-feedb...@google.com .
Thanks!

> Dataflow pipeline stalls after autoscaling
> --
>
> Key: BEAM-2702
> URL: https://issues.apache.org/jira/browse/BEAM-2702
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.0.0
>Reporter: Johann Steinbrecher
>Assignee: Thomas Groh
>
> A 4 step dataflow pipeline (Pubsubio.Read, windowing, message parsing, 
> DatastoreV1.write) stalls as soon as the autoscaling algorithm is increasing 
> the number of workers from 1 to 4. 
> *Expected*:
> Throughput (elements/sec) for each pipeline step increases due to more 
> workers.
> *Actual*:
> Throughput (elements/sec) goes to 0 for all steps. The number of processed 
> elements in the first step equals the number of processed elements in the 
> last step. The number of workers stays high.
> Runner: google-cloud-platform managed dataflow runner
> Sample dataflow job id (log level debug):
> 2017-07-27_14_51_37-4624978117098944513
> Log message after autoscaling:
> Rpc to .. completed with error DEADLINE_EXCEEDED (cause or symptom?)
> autoscaling configuration 
> --autoscalingAlgorithm=THROUGHPUT_BASED 
> --maxNumWorkers=4 
> machine types tested:
> - n1-highmem-2
> - n1-standard-1
> zone: us-east1-d
> sdk version:
> org.apache.beam@2.0.0



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2838

2017-07-31 Thread Apache Jenkins Server

See 


--
[...truncated 558.63 KB...]
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_multiple_empty_outputs 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.5.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest) 
... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.5.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.5.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_dict_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line

[jira] [Commented] (BEAM-2141) beam_PerformanceTests_JDBC have not passed in weeks

2017-07-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108019#comment-16108019
 ] 

ASF GitHub Bot commented on BEAM-2141:
--

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/3668

[BEAM-2141] Updates jenkins job for JDBCIOIT

This is a slightly updated version of Stephen Sisk's 
https://github.com/apache/beam/pull/3604.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam enable_jdbc_it

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3668.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3668


commit d61ca8f8845c4290e3e88dd9da2bd94605ab141b
Author: chamik...@google.com 
Date:   2017-07-31T18:50:46Z

Updates jenkins job for JDBCIOIT.

This is a slightly updated version of Stephen Sisk's 
https://github.com/apache/beam/pull/3604.




> beam_PerformanceTests_JDBC have not passed in weeks
> ---
>
> Key: BEAM-2141
> URL: https://issues.apache.org/jira/browse/BEAM-2141
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Daniel Halperin
>Assignee: Stephen Sisk
> Fix For: Not applicable
>
>
> https://builds.apache.org/job/beam_PerformanceTests_JDBC/
> Disabling them, as no one seems to be maintaining them.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3668: [BEAM-2141] Updates jenkins job for JDBCIOIT

2017-07-31 Thread chamikaramj

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/3668

[BEAM-2141] Updates jenkins job for JDBCIOIT

This is a slightly updated version of Stephen Sisk's 
https://github.com/apache/beam/pull/3604.

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam enable_jdbc_it

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3668.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3668


commit d61ca8f8845c4290e3e88dd9da2bd94605ab141b
Author: chamik...@google.com 
Date:   2017-07-31T18:50:46Z

Updates jenkins job for JDBCIOIT.

This is a slightly updated version of Stephen Sisk's 
https://github.com/apache/beam/pull/3604.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2431) Model Runner interactions in RPC layer for Runner API

2017-07-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107975#comment-16107975
 ] 

ASF GitHub Bot commented on BEAM-2431:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/3667

[BEAM-2431] Check in the job api and prototype of python direct runner

This PR works with a test pipeline but we need some changes in the
direct runner to get rid of the apply loop and split the run and
execute function. But for an MVP version demonstrating the job api this
should be good to go.

R: @kennknowles PTAL

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-2431-beam-job-api

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3667.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3667


commit 7845810a591e7f72493109b755e76bddaf03ee63
Author: Sourabh Bajaj 
Date:   2017-07-31T21:19:30Z

[BEAM-2431] Check in the job api and prototype of python direct runner

This PR works with a test pipeline but we need some changes in the
direct runner to get rid of the apply loop and split the run and
execute function. But for an MVP version demonstrating the job api this
should be good to go.




> Model Runner interactions in RPC layer for Runner API
> -
>
> Key: BEAM-2431
> URL: https://issues.apache.org/jira/browse/BEAM-2431
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api
>Reporter: Kenneth Knowles
>Assignee: Sourabh Bajaj
>  Labels: beam-python-everywhere
>
> The "Runner API" today is actually just a definition of what constitutes a 
> Beam pipeline. It needs to actually be a (very small) API.
> This would allow e.g. a Java-based job launcher to respond to launch requests 
> and state queries from a Python-based adapter.
> The expected API would be something like a distillation of the APIs for 
> PipelineRunner and PipelineResult (which is really "Job") via analyzing how 
> these both look in Java and Python.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3667: [BEAM-2431] Check in the job api and prototype of p...

2017-07-31 Thread sb2nov

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/3667

[BEAM-2431] Check in the job api and prototype of python direct runner

This PR works with a test pipeline but we need some changes in the
direct runner to get rid of the apply loop and split the run and
execute function. But for an MVP version demonstrating the job api this
should be good to go.

R: @kennknowles PTAL

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-2431-beam-job-api

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3667.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3667


commit 7845810a591e7f72493109b755e76bddaf03ee63
Author: Sourabh Bajaj 
Date:   2017-07-31T21:19:30Z

[BEAM-2431] Check in the job api and prototype of python direct runner

This PR works with a test pipeline but we need some changes in the
direct runner to get rid of the apply loop and split the run and
execute function. But for an MVP version demonstrating the job api this
should be good to go.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2704) KafkaIO: NPE without key serializer set

2017-07-31 Thread Chris Pettitt (JIRA)

Chris Pettitt created BEAM-2704:
---

 Summary: KafkaIO: NPE without key serializer set
 Key: BEAM-2704
 URL: https://issues.apache.org/jira/browse/BEAM-2704
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Chris Pettitt
Assignee: Davor Bonaci


The KafkaIO javadoc implies that you do not need to set a Serializer if you 
only want to emit values:

{code}
 * Often you might want to write just values without any keys to Kafka. Use 
{@code values()} to
 * write records with default empty(null) key:
 *
 * {@code
 *  PCollection strings = ...;
 *  strings.apply(KafkaIO.write()
 *  .withBootstrapServers("broker_1:9092,broker_2:9092")
 *  .withTopic("results")
 *  .withValueSerializer(new StringSerializer()) // just need serializer 
for value
 *  .values()
 *);
 * }
{code}

However, if you don't set the key serializer then Kafka blows up when trying to 
instantiate the key serializer (in Kafka 0.10.1, at least). It would be more 
convenient if KafkaIO worked as documented and assigned a null serializer if 
values() is used.  

Relevant stack trace:

{code}
Caused by: java.lang.NullPointerException
at 
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:230)
at 
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:163)
at 
org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter.setup(KafkaIO.java:1582)
at 
org.apache.beam.sdk.io.kafka.KafkaIO$KafkaWriter$DoFnInvoker.invokeSetup(Unknown
 Source)
{code}





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2703) KafkaIO: watermark outside the bounds of BoundedWindow

2017-07-31 Thread Chris Pettitt (JIRA)

Chris Pettitt created BEAM-2703:
---

 Summary: KafkaIO: watermark outside the bounds of BoundedWindow
 Key: BEAM-2703
 URL: https://issues.apache.org/jira/browse/BEAM-2703
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Chris Pettitt
Assignee: Davor Bonaci


KafkaIO appears to use an incorrect lower bound for it's initial watermark with 
respect to BoundedWindow.TIMESTAMP_MIN_VALUE.

KafkaIO's initial watermark:
new Instant(Long.MIN_VALUE) -> -9223372036854775808

BoundedWindow.TIMESTAMP_MIN_VALUE:
new Instant(TimeUnit.MICROSECONDS.toMillis(Long.MIN_VALUE)) -> -9223372036854775

The difference is that the last three digits have been truncated due to the 
micro to millis conversion.

This difference can cause errors in runners that assert that the input 
watermark can never regress as KafkaIO gives a value below the lower bound when 
no messages have been received yet. For consistency it would probably be best 
for it to use BoundedWindow.TIMESTAMP_MIN_VALUE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-2587) Build fails due to python sdk

2017-07-31 Thread Ahmet Altay (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107735#comment-16107735
 ] 

Ahmet Altay edited comment on BEAM-2587 at 7/31/17 7:50 PM:


[~jbonofre] I will suggest moving this out of the 2.1.0 blocker list. The 
reason is, it does not affect end users (i.e. published artifacts will already 
have the correct generated files), it does not affect daily development (i.e. 
works within a git tree), and even out of that conditions it fails based on 
pip/os versions that is not clearly defined yet. It will take some more time to 
reproduce and fix it and does not need to block the release.

However, this is a regression in the sense that at the time of 2.0.0 there was 
no code path for dynamically generating these files.


was (Author: altay):
[~jbonofre] I will suggest moving this out of the 2.1.0 blocker list. The 
reason is, it does not affect end users (i.e. published artifacts will already 
have the correct generated files), it does not affect daily development (i.e. 
works within a git tree), and event out of that conditions it fails based on 
pip/os versions that is not clearly defined yet. It will take some more time to 
reproduce and fix it and does not need to block the release.

However, this is a regression in the sense that at the time of 2.0.0 there was 
code path for dynamically generating these files.

> Build fails due to python sdk
> -
>
> Key: BEAM-2587
> URL: https://issues.apache.org/jira/browse/BEAM-2587
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 2.1.0
>Reporter: Ahmet Altay
>
> Build fails with the following errors when {{mvn clean package}} is used on a 
> clean Ubuntu 16.04 LTS machine with pip 8.x. The issue is resolved when pip 
> is upgraded to pip 9.x
> "RuntimeError: Not in apache git tree; unable to find proto definitions."
> "DistutilsOptionError: can't combine user with prefix, exec_prefix/home, or 
> install_(plat)base"
> We need to understand the issue and maybe add a note about requiring pip 9.x 
> for development. Note that this does not affect end users using prepackaged 
> artifacts from central repositories.
> cc: [~robertwb]
> Script for reproduction:
> {code}
> #!/bin/bash
> set -e
> readonly MACHINE_ID=$(hexdump -n 1 -e '"%x"' /dev/random)
> readonly MACHINE="${USER}-beam-build-${MACHINE_ID}"
> readonly ZONE="us-central1-c"
> # provision building machine
> echo "Provisioning Build Machine (Ubuntu 16.04 LTS)"
> gcloud compute instances create "$MACHINE" \
>   --zone="$ZONE" \
>   --image-project="ubuntu-os-cloud" \
>   --image-family="ubuntu-1604-lts"
> # wait for ssh to be ready
> echo "Waiting for machine to finish booting"
> sleep 30
> # ssh into the machine
> # 1. install dependencies as specified by beam readme
> # 2. download beam source from github
> # 3. build with maven
> echo "Downloading and building Apache Beam (release-2.1.0)"
> gcloud compute ssh "$MACHINE" --zone="$ZONE" << EOF
> sudo apt-get --assume-yes update
> sudo apt-get --assume-yes install \
> openjdk-8-jdk \
> maven \
> python-setuptools \
> python-pip
> wget https://github.com/apache/beam/archive/release-2.1.0.tar.gz
> tar -xzf release-2.1.0.tar.gz
> cd beam-release-2.1.0
> mvn clean package
> EOF
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3688

2017-07-31 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-2587) Build fails due to python sdk

2017-07-31 Thread JIRA


[ 
https://issues.apache.org/jira/browse/BEAM-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107819#comment-16107819
 ] 

Jean-Baptiste Onofré commented on BEAM-2587:


Agree. I removed the fixed version. Thanks !

> Build fails due to python sdk
> -
>
> Key: BEAM-2587
> URL: https://issues.apache.org/jira/browse/BEAM-2587
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 2.1.0
>Reporter: Ahmet Altay
>
> Build fails with the following errors when {{mvn clean package}} is used on a 
> clean Ubuntu 16.04 LTS machine with pip 8.x. The issue is resolved when pip 
> is upgraded to pip 9.x
> "RuntimeError: Not in apache git tree; unable to find proto definitions."
> "DistutilsOptionError: can't combine user with prefix, exec_prefix/home, or 
> install_(plat)base"
> We need to understand the issue and maybe add a note about requiring pip 9.x 
> for development. Note that this does not affect end users using prepackaged 
> artifacts from central repositories.
> cc: [~robertwb]
> Script for reproduction:
> {code}
> #!/bin/bash
> set -e
> readonly MACHINE_ID=$(hexdump -n 1 -e '"%x"' /dev/random)
> readonly MACHINE="${USER}-beam-build-${MACHINE_ID}"
> readonly ZONE="us-central1-c"
> # provision building machine
> echo "Provisioning Build Machine (Ubuntu 16.04 LTS)"
> gcloud compute instances create "$MACHINE" \
>   --zone="$ZONE" \
>   --image-project="ubuntu-os-cloud" \
>   --image-family="ubuntu-1604-lts"
> # wait for ssh to be ready
> echo "Waiting for machine to finish booting"
> sleep 30
> # ssh into the machine
> # 1. install dependencies as specified by beam readme
> # 2. download beam source from github
> # 3. build with maven
> echo "Downloading and building Apache Beam (release-2.1.0)"
> gcloud compute ssh "$MACHINE" --zone="$ZONE" << EOF
> sudo apt-get --assume-yes update
> sudo apt-get --assume-yes install \
> openjdk-8-jdk \
> maven \
> python-setuptools \
> python-pip
> wget https://github.com/apache/beam/archive/release-2.1.0.tar.gz
> tar -xzf release-2.1.0.tar.gz
> cd beam-release-2.1.0
> mvn clean package
> EOF
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2587) Build fails due to python sdk

2017-07-31 Thread JIRA


 [ 
https://issues.apache.org/jira/browse/BEAM-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2587:
---
Fix Version/s: (was: 2.1.0)

> Build fails due to python sdk
> -
>
> Key: BEAM-2587
> URL: https://issues.apache.org/jira/browse/BEAM-2587
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 2.1.0
>Reporter: Ahmet Altay
>
> Build fails with the following errors when {{mvn clean package}} is used on a 
> clean Ubuntu 16.04 LTS machine with pip 8.x. The issue is resolved when pip 
> is upgraded to pip 9.x
> "RuntimeError: Not in apache git tree; unable to find proto definitions."
> "DistutilsOptionError: can't combine user with prefix, exec_prefix/home, or 
> install_(plat)base"
> We need to understand the issue and maybe add a note about requiring pip 9.x 
> for development. Note that this does not affect end users using prepackaged 
> artifacts from central repositories.
> cc: [~robertwb]
> Script for reproduction:
> {code}
> #!/bin/bash
> set -e
> readonly MACHINE_ID=$(hexdump -n 1 -e '"%x"' /dev/random)
> readonly MACHINE="${USER}-beam-build-${MACHINE_ID}"
> readonly ZONE="us-central1-c"
> # provision building machine
> echo "Provisioning Build Machine (Ubuntu 16.04 LTS)"
> gcloud compute instances create "$MACHINE" \
>   --zone="$ZONE" \
>   --image-project="ubuntu-os-cloud" \
>   --image-family="ubuntu-1604-lts"
> # wait for ssh to be ready
> echo "Waiting for machine to finish booting"
> sleep 30
> # ssh into the machine
> # 1. install dependencies as specified by beam readme
> # 2. download beam source from github
> # 3. build with maven
> echo "Downloading and building Apache Beam (release-2.1.0)"
> gcloud compute ssh "$MACHINE" --zone="$ZONE" << EOF
> sudo apt-get --assume-yes update
> sudo apt-get --assume-yes install \
> openjdk-8-jdk \
> maven \
> python-setuptools \
> python-pip
> wget https://github.com/apache/beam/archive/release-2.1.0.tar.gz
> tar -xzf release-2.1.0.tar.gz
> cd beam-release-2.1.0
> mvn clean package
> EOF
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2676) move BeamSqlRow and BeamSqlRowType to sdk/java/core

2017-07-31 Thread Robert Bradshaw (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107771#comment-16107771
 ] 

Robert Bradshaw commented on BEAM-2676:
---

The motivation for this came up because (1) we're currently lacking a Generic 
tuple-like class (which would be useful for joins and a Java port for 
https://github.com/tensorflow/transform , to name a few) and (2) if we have 
such a class, SQL should use it rather than do its own thing if at all 
possible. 

The BeamRow object itself is not at all SQL centric, however BeamRowType is. 
The original proposal was to have BeamRow (I would call it BeamRecord, but 
don't feel that strongly) be a generic type in core, a sibling to KV, along 
with an abstract BeamRowType (which, among other things, vends the Coder from a 
list of component coders). Then one would have a concrete BeamSqlRowType 
subclass of BeamRowType which is instantiated via java.sql.Types. This is the 
most forward compatible state without tying the tuple to SQL. 

(I added an item to mark the entire API as Experimental for now, which I think 
makes a lot of sense.)

> move BeamSqlRow and BeamSqlRowType to sdk/java/core
> ---
>
> Key: BEAM-2676
> URL: https://issues.apache.org/jira/browse/BEAM-2676
> Project: Beam
>  Issue Type: Test
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>  Labels: dsl_sql_merge
>
> BeamSqlRow/BeamSqlRowType is the fundamental of structured data processing in 
> Beam, like joins, simple projections/expansions. It's more visible to move 
> them to sdk-java-core.
> It contains two parts:
> 1). remove SQL word in the name,
> BeamSqlRow --> BeamRow
> BeamSqlRowType --> BeamRowType
> 2). move from package {{org.apache.beam.dsls.sql.schema}} to 
> {{org.apache.beam.sdk.sd}} (sd stands for structure data), in module 
> {{beam-sdks-java-core}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2747

2017-07-31 Thread Apache Jenkins Server

See

[jira] [Commented] (BEAM-2587) Build fails due to python sdk

2017-07-31 Thread Ahmet Altay (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107735#comment-16107735
 ] 

Ahmet Altay commented on BEAM-2587:
---

[~jbonofre] I will suggest moving this out of the 2.1.0 blocker list. The 
reason is, it does not affect end users (i.e. published artifacts will already 
have the correct generated files), it does not affect daily development (i.e. 
works within a git tree), and event out of that conditions it fails based on 
pip/os versions that is not clearly defined yet. It will take some more time to 
reproduce and fix it and does not need to block the release.

However, this is a regression in the sense that at the time of 2.0.0 there was 
code path for dynamically generating these files.

> Build fails due to python sdk
> -
>
> Key: BEAM-2587
> URL: https://issues.apache.org/jira/browse/BEAM-2587
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 2.1.0
>Reporter: Ahmet Altay
> Fix For: 2.1.0
>
>
> Build fails with the following errors when {{mvn clean package}} is used on a 
> clean Ubuntu 16.04 LTS machine with pip 8.x. The issue is resolved when pip 
> is upgraded to pip 9.x
> "RuntimeError: Not in apache git tree; unable to find proto definitions."
> "DistutilsOptionError: can't combine user with prefix, exec_prefix/home, or 
> install_(plat)base"
> We need to understand the issue and maybe add a note about requiring pip 9.x 
> for development. Note that this does not affect end users using prepackaged 
> artifacts from central repositories.
> cc: [~robertwb]
> Script for reproduction:
> {code}
> #!/bin/bash
> set -e
> readonly MACHINE_ID=$(hexdump -n 1 -e '"%x"' /dev/random)
> readonly MACHINE="${USER}-beam-build-${MACHINE_ID}"
> readonly ZONE="us-central1-c"
> # provision building machine
> echo "Provisioning Build Machine (Ubuntu 16.04 LTS)"
> gcloud compute instances create "$MACHINE" \
>   --zone="$ZONE" \
>   --image-project="ubuntu-os-cloud" \
>   --image-family="ubuntu-1604-lts"
> # wait for ssh to be ready
> echo "Waiting for machine to finish booting"
> sleep 30
> # ssh into the machine
> # 1. install dependencies as specified by beam readme
> # 2. download beam source from github
> # 3. build with maven
> echo "Downloading and building Apache Beam (release-2.1.0)"
> gcloud compute ssh "$MACHINE" --zone="$ZONE" << EOF
> sudo apt-get --assume-yes update
> sudo apt-get --assume-yes install \
> openjdk-8-jdk \
> maven \
> python-setuptools \
> python-pip
> wget https://github.com/apache/beam/archive/release-2.1.0.tar.gz
> tar -xzf release-2.1.0.tar.gz
> cd beam-release-2.1.0
> mvn clean package
> EOF
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2702) Dataflow pipeline stalls after autoscaling

2017-07-31 Thread Johann Steinbrecher (JIRA)

Johann Steinbrecher created BEAM-2702:
-

 Summary: Dataflow pipeline stalls after autoscaling
 Key: BEAM-2702
 URL: https://issues.apache.org/jira/browse/BEAM-2702
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Affects Versions: 2.0.0
Reporter: Johann Steinbrecher
Assignee: Thomas Groh


A 4 step dataflow pipeline (Pubsubio.Read, windowing, message parsing, 
DatastoreV1.write) stalls as soon as the autoscaling algorithm is increasing 
the number of workers from 1 to 4. 

*Expected*:
Throughput (elements/sec) for each pipeline step increases due to more workers.

*Actual*:
Throughput (elements/sec) goes to 0 for all steps. The number of processed 
elements in the first step equals the number of processed elements in the last 
step. The number of workers stays high.


Runner: google-cloud-platform managed dataflow runner

Sample dataflow job id (log level debug):
2017-07-27_14_51_37-4624978117098944513

Log message after autoscaling:
Rpc to .. completed with error DEADLINE_EXCEEDED (cause or symptom?)

autoscaling configuration 
--autoscalingAlgorithm=THROUGHPUT_BASED 
--maxNumWorkers=4 

machine types tested:
- n1-highmem-2
- n1-standard-1

zone: us-east1-d

sdk version:
org.apache.beam@2.0.0




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2699) AppliedPTransform is used as a key in hashmaps but PTransform is not hashable/equality-comparable

2017-07-31 Thread Thomas Groh (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107634#comment-16107634
 ] 

Thomas Groh commented on BEAM-2699:
---

The latter claim is incorrect: two applications of the same transform to the 
same input will produce different output (unless the transform's output is a 
subset of it's input; in which case the two applications are indistinguishable 
(same input, same transform, same output), and reusing the same references 
cannot be observed. Additionally, applications within a pipeline are forced to 
have unique names, so no two transforms will ever incorrectly collide.

The former claim is more worrying; failing to find the same application is a 
pretty big problem. Potentially equals/hashCode should be (Name + Pipeline)?

As an aside, PTransforms shouldn't throw an exception if you call equals or 
hashCode on them, just return a more-discriminating response.

> AppliedPTransform is used as a key in hashmaps but PTransform is not 
> hashable/equality-comparable
> -
>
> Key: BEAM-2699
> URL: https://issues.apache.org/jira/browse/BEAM-2699
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Thomas Groh
>
> There's plenty of occurrences in runners-core of Map or BiMap where the key 
> is an AppliedPTransform.
> However, PTransform does not advertise that it is required to implement 
> equals/hashCode, and some transforms can't do it properly anyway - for 
> example, transforms that capture a ValueProvider which is also not 
> hashable/eq-comparable. I'm surprised that things aren't already very broken 
> because of this.
> Fundamentally, I don't see why we should ever compare two PTransform's for 
> equality.
> I looked at the code and wondered "can AppliedPTransform simply be 
> identity-hashable", but right now the answer is no because we can create an 
> AppliedPTransform for the same transform applied to the same thing multiple 
> times.
> Fixing that appears to be not very easy, but definitely possible. Ideally 
> TransformHierarchy.Node would just know its AppliedPTransform, however a Node 
> can be constructed when there's yet no Pipeline. Suppose there's gotta be 
> some way to propagate a Pipeline into Node.finishSpecifying() (which should 
> be called exactly once on the Node, and this should be enforced), and have 
> finishSpecifying() return the AppliedPTransform, and have the caller use that 
> instead of potentially repeatedly calling .toAppliedPTransform() on the same 
> Node.
> [~kenn] is on vacation but perhaps [~tgroh] can help with this meanwhile?
> CC: [~reuvenlax]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2701) use a custom implementation of java.io.ObjectInputStream

2017-07-31 Thread Romain Manni-Bucau (JIRA)

Romain Manni-Bucau created BEAM-2701:


 Summary: use a custom implementation of java.io.ObjectInputStream
 Key: BEAM-2701
 URL: https://issues.apache.org/jira/browse/BEAM-2701
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Romain Manni-Bucau
Assignee: Davor Bonaci


java.io.ObjectInputStream should override resolve[Proxy]Class using the TCCL to 
support any classloader and not fallback into some JVM pitfall using another 
classloader (default). This will enable beam to use any classloader instead of 
requiring to run in the JVM using java serialization.

{code}
@Override
protected Class resolveClass(final ObjectStreamClass classDesc) throws 
IOException, ClassNotFoundException {
final String n = classDesc.getName();
final ClassLoader classloader = getClassloader();
try {
return Class.forName(n, false, classloader);
} catch (ClassNotFoundException e) {

if (n.equals("boolean")) {
return boolean.class;
}
if (n.equals("byte")) {
return byte.class;
}
if (n.equals("char")) {
return char.class;
}
if (n.equals("short")) {
return short.class;
}
if (n.equals("int")) {
return int.class;
}
if (n.equals("long")) {
return long.class;
}
if (n.equals("float")) {
return float.class;
}
if (n.equals("double")) {
return double.class;
}

//Last try - Let runtime try and find it.
return Class.forName(n, false, null);
}
}

@Override
protected Class resolveProxyClass(final String[] interfaces) throws 
IOException, ClassNotFoundException {
final Class[] cinterfaces = new Class[interfaces.length];
for (int i = 0; i < interfaces.length; i++) {
cinterfaces[i] = getClassloader().loadClass(interfaces[i]);
}

try {
return Proxy.getProxyClass(getClassloader(), cinterfaces);
} catch (IllegalArgumentException e) {
throw new ClassNotFoundException(null, e);
}
}
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2837

2017-07-31 Thread Apache Jenkins Server

See 


--
[...truncated 558.63 KB...]
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_multiple_empty_outputs 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.5.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest) 
... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.5.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.5.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3687

2017-07-31 Thread Apache Jenkins Server

See

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2746

2017-07-31 Thread Apache Jenkins Server

See

Build failed in Jenkins: beam_PostCommit_Python_Verify #2836

2017-07-31 Thread Apache Jenkins Server

See 


--
[...truncated 558.16 KB...]
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:135:
 UserWarning: Using fallback coder for typehint: List[Any].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.5.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.5.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-36.2.5.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-3.1.1.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_undeclared_outputs (apache_beam.transforms.ptransform_test.PTransformTest) 
... ok
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded

[jira] [Commented] (BEAM-2441) DSL SQL maven module location and name

2017-07-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107010#comment-16107010
 ] 

ASF GitHub Bot commented on BEAM-2441:
--

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3666

[BEAM-2441] move dsls/sql to sdks/java/extensions/sql

R: @XuMingmin @takidau 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam BEAM-2441

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3666.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3666


commit ba493f85a5a18665cd6ae4eb61e0a86fab1a6c07
Author: James Xu 
Date:   2017-07-30T15:39:37Z

move dsls/sql to sdks/java/extensions/sql

commit c1b5482d3fb13a14926f0ffc23d0810b3105ed24
Author: James Xu 
Date:   2017-07-30T15:58:02Z

rename package org.apache.beam.dsls.sql to 
org.apache.beam.sdk.extensions.sql

commit febd044ae306a28fa3797a1663e54c1d7fbe43ce
Author: James Xu 
Date:   2017-07-31T09:11:53Z

move all implementation classes/packages into impl package




> DSL SQL maven module location and name
> --
>
> Key: BEAM-2441
> URL: https://issues.apache.org/jira/browse/BEAM-2441
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Luke Cwik
>Assignee: James Xu
>  Labels: dsl_sql_merge
>
> The current maven module location is *dsl/sql*, unfortunately this occludes 
> the fact that this is for the Java SDK and also prevents alternative language 
> implementations.
> Some alternative locations could be:
> {code}
> sdks/java/extensions/sql
> sdks/java/dsls/sql
> dsls/sql/java
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3666: [BEAM-2441] move dsls/sql to sdks/java/extensions/s...

2017-07-31 Thread xumingming

GitHub user xumingming opened a pull request:

https://github.com/apache/beam/pull/3666

[BEAM-2441] move dsls/sql to sdks/java/extensions/sql

R: @XuMingmin @takidau 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xumingming/beam BEAM-2441

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3666.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3666


commit ba493f85a5a18665cd6ae4eb61e0a86fab1a6c07
Author: James Xu 
Date:   2017-07-30T15:39:37Z

move dsls/sql to sdks/java/extensions/sql

commit c1b5482d3fb13a14926f0ffc23d0810b3105ed24
Author: James Xu 
Date:   2017-07-30T15:58:02Z

rename package org.apache.beam.dsls.sql to 
org.apache.beam.sdk.extensions.sql

commit febd044ae306a28fa3797a1663e54c1d7fbe43ce
Author: James Xu 
Date:   2017-07-31T09:11:53Z

move all implementation classes/packages into impl package




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2530) Make Beam compatible with Java 9

2017-07-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106996#comment-16106996
 ] 

ASF GitHub Bot commented on BEAM-2530:
--

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/3665

[BEAM-2530] Upgrade maven plugins to the latest versions

This is part of the ongoing work to support java 9

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-2530-java9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3665.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3665


commit ea97c73227f9323947b6082e0abdeac149732eb0
Author: Ismaël Mejía 
Date:   2017-07-06T15:53:36Z

[BEAM-2530] Upgrade maven plugins to the latest versions

This is part of the ongoing work to support java 9




> Make Beam compatible with Java 9
> 
>
> Key: BEAM-2530
> URL: https://issues.apache.org/jira/browse/BEAM-2530
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Affects Versions: Not applicable
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: Not applicable
>
>
> Java 9 seems to be finally been released this year, this is a JIRA to keep 
> track of the needed changes to support Beam on Java 9.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3665: [BEAM-2530] Upgrade maven plugins to the latest ver...

2017-07-31 Thread iemejia

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/3665

[BEAM-2530] Upgrade maven plugins to the latest versions

This is part of the ongoing work to support java 9

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-2530-java9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3665.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3665


commit ea97c73227f9323947b6082e0abdeac149732eb0
Author: IsmaÃ«l MejÃa 
Date:   2017-07-06T15:53:36Z

[BEAM-2530] Upgrade maven plugins to the latest versions

This is part of the ongoing work to support java 9




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2530) Make Beam compatible with Java 9

2017-07-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106991#comment-16106991
 ] 

ASF GitHub Bot commented on BEAM-2530:
--

Github user iemejia closed the pull request at:

https://github.com/apache/beam/pull/3664


> Make Beam compatible with Java 9
> 
>
> Key: BEAM-2530
> URL: https://issues.apache.org/jira/browse/BEAM-2530
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Affects Versions: Not applicable
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: Not applicable
>
>
> Java 9 seems to be finally been released this year, this is a JIRA to keep 
> track of the needed changes to support Beam on Java 9.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3664: [BEAM-2530] Upgrade maven enforcer and javadoc plug...

2017-07-31 Thread iemejia

Github user iemejia closed the pull request at:

https://github.com/apache/beam/pull/3664


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3664: [BEAM-2530] Upgrade maven enforcer and javadoc plug...

2017-07-31 Thread iemejia

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/3664

[BEAM-2530] Upgrade maven enforcer and javadoc plugins to support Java 9

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-2530-java9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3664.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3664


commit af91ebf2b18acd96ba6e71b8bcdb74f3c0e22e94
Author: IsmaÃ«l MejÃa 
Date:   2017-07-06T15:53:36Z

[BEAM-2530] Upgrade maven enforcer and javadoc plugins to support Java 9




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2530) Make Beam compatible with Java 9

2017-07-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/BEAM-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106987#comment-16106987
 ] 

ASF GitHub Bot commented on BEAM-2530:
--

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/3664

[BEAM-2530] Upgrade maven enforcer and javadoc plugins to support Java 9

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-2530-java9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3664.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3664


commit af91ebf2b18acd96ba6e71b8bcdb74f3c0e22e94
Author: Ismaël Mejía 
Date:   2017-07-06T15:53:36Z

[BEAM-2530] Upgrade maven enforcer and javadoc plugins to support Java 9




> Make Beam compatible with Java 9
> 
>
> Key: BEAM-2530
> URL: https://issues.apache.org/jira/browse/BEAM-2530
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Affects Versions: Not applicable
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: Not applicable
>
>
> Java 9 seems to be finally been released this year, this is a JIRA to keep 
> track of the needed changes to support Beam on Java 9.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3653: Make maven-enforcer-plugin version unique

2017-07-31 Thread iemejia

Github user iemejia closed the pull request at:

https://github.com/apache/beam/pull/3653


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3686

2017-07-31 Thread Apache Jenkins Server

See

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2745

2017-07-31 Thread Apache Jenkins Server

See

43 matches

Mail list logo