[jira] [Commented] (BEAM-4088) ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 does not pass in gradle

[Etienne Chauchot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16473864#comment-16473864
 ] 

Etienne Chauchot commented on BEAM-4088:


Thanks [~tgroh] for deactivating this flaky test while I was on holidays. FYI I 
had a PR ([https://github.com/apache/beam/pull/5280)] to add some logs to 
figure out why it is (often) failing.

> ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 
> does not pass in gradle
> -
>
> Key: BEAM-4088
> URL: https://issues.apache.org/jira/browse/BEAM-4088
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> This is a new test being added to ensure threads don't leak. The failure 
> seems to indicate that threads do leak.
> This test fails using gradle but previously passed using maven
> PR: https://github.com/apache/beam/pull/4965
> Presubmit Failure: 
>  * https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/4059/
>  * 
> https://scans.gradle.com/s/grha56432j3t2/tests/jqhvlvf72f7pg-ipde5etqqejoa?openStackTraces=WzBd



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4088) ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 does not pass in gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4088?focusedWorklogId=101608&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101608
 ]

ASF GitHub Bot logged work on BEAM-4088:


Author: ASF GitHub Bot
Created on: 14/May/18 07:40
Start Date: 14/May/18 07:40
Worklog Time Spent: 10m 
  Work Description: echauchot commented on issue #5300: [BEAM-4088] 
Suppress a test in ExecutorServiceParallelExecutorTest
URL: https://github.com/apache/beam/pull/5300#issuecomment-388725158
 
 
   Sorry @tgroh I was off, thanks for that ! 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101608)
Time Spent: 2h 40m  (was: 2.5h)

> ExecutorServiceParallelExecutorTest#ensureMetricsThreadDoesntLeak in PR #4965 
> does not pass in gradle
> -
>
> Key: BEAM-4088
> URL: https://issues.apache.org/jira/browse/BEAM-4088
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> This is a new test being added to ensure threads don't leak. The failure 
> seems to indicate that threads do leak.
> This test fails using gradle but previously passed using maven
> PR: https://github.com/apache/beam/pull/4965
> Presubmit Failure: 
>  * https://builds.apache.org/job/beam_PreCommit_Java_GradleBuild/4059/
>  * 
> https://scans.gradle.com/s/grha56432j3t2/tests/jqhvlvf72f7pg-ipde5etqqejoa?openStackTraces=WzBd



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1624

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision bb319dbfaa81c27d7ba6b0a465994aa656143100 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bb319dbfaa81c27d7ba6b0a465994aa656143100
Commit message: "Merge pull request #5277: [BEAM-3515] Portable translation of 
SplittableProcessKeyed"
 > git rev-list --no-walk bb319dbfaa81c27d7ba6b0a465994aa656143100 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins6613856327013411637.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ekirpic...@gmail.com

[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=101635&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101635
 ]

ASF GitHub Bot logged work on BEAM-4059:


Author: ASF GitHub Bot
Created on: 14/May/18 08:59
Start Date: 14/May/18 08:59
Worklog Time Spent: 10m 
  Work Description: echauchot commented on a change in pull request #5193: 
[BEAM-4059] Reduce number of ValidatesRunner tests and reorganize them for 
better parallelization
URL: https://github.com/apache/beam/pull/5193#discussion_r187879854
 
 

 ##
 File path: 
sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/MetricsTest.java
 ##
 @@ -69,216 +74,286 @@ private static MetricQueryResults 
queryTestMetrics(PipelineResult result) {
 .build());
   }
 
-  @Rule
-  public final transient TestPipeline pipeline = TestPipeline.create();
-
-  @Rule
-  public final transient ExpectedException thrown = ExpectedException.none();
-
-  @After
-  public void tearDown() {
-MetricsEnvironment.setCurrentContainer(null);
-  }
-
-  @Test
-  public void testDistributionWithoutContainer() {
-assertNull(MetricsEnvironment.getCurrentContainer());
-// Should not fail even though there is no metrics container.
-Metrics.distribution(NS, NAME).update(5L);
-  }
-
-  @Test
-  public void testCounterWithoutContainer() {
-assertNull(MetricsEnvironment.getCurrentContainer());
-// Should not fail even though there is no metrics container.
-Counter counter = Metrics.counter(NS, NAME);
-counter.inc();
-counter.inc(5L);
-counter.dec();
-counter.dec(5L);
-  }
-
-  @Test
-  public void testCounterWithEmptyName() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.counter(NS, "");
-  }
-
-  @Test
-  public void testCounterWithEmptyNamespace() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.counter("", NAME);
+  /** Shared test helpers and setup/teardown. */
+  public abstract static class SharedTestBase {
+@Rule
+public final transient ExpectedException thrown = ExpectedException.none();
+
+@Rule
+public final transient TestPipeline pipeline = TestPipeline.create();
+
+@After
+public void tearDown() {
+  MetricsEnvironment.setCurrentContainer(null);
+}
+
+protected PipelineResult runPipelineWithMetrics() {
+  final Counter count = Metrics.counter(MetricsTest.class, "count");
+  final TupleTag output1 = new TupleTag(){};
+  final TupleTag output2 = new TupleTag(){};
+  pipeline
+  .apply(Create.of(5, 8, 13))
+  .apply("MyStep1", ParDo.of(new DoFn() {
+Distribution bundleDist = Metrics.distribution(MetricsTest.class, 
"bundle");
+
+@StartBundle
+public void startBundle() {
+  bundleDist.update(10L);
+}
+
+@SuppressWarnings("unused")
+@ProcessElement
+public void processElement(ProcessContext c) {
+  Distribution values = Metrics.distribution(MetricsTest.class, 
"input");
+  count.inc();
+  values.update(c.element());
+
+  c.output(c.element());
+  c.output(c.element());
+}
+
+@DoFn.FinishBundle
+public void finishBundle() {
+  bundleDist.update(40L);
+}
+  }))
+  .apply("MyStep2", ParDo
+  .of(new DoFn() {
+@SuppressWarnings("unused")
+@ProcessElement
+public void processElement(ProcessContext c) {
+  Distribution values = 
Metrics.distribution(MetricsTest.class, "input");
+  Gauge gauge = Metrics.gauge(MetricsTest.class, "my-gauge");
+  Integer element = c.element();
+  count.inc();
+  values.update(element);
+  gauge.set(12L);
+  c.output(element);
+  c.output(output2, element);
+}
+  })
+  .withOutputTags(output1, TupleTagList.of(output2)));
+  PipelineResult result = pipeline.run();
+
+  result.waitUntilFinish();
+  return result;
+}
   }
 
-  @Test
-  public void testDistributionWithEmptyName() {
-thrown.expect(IllegalArgumentException.class);
-Metrics.distribution(NS, "");
+  /** Tests validating basic metric scenarios. */
+  @RunWith(JUnit4.class)
+  public static class BasicTests extends SharedTestBase {
+@Test
+public void testDistributionWithoutContainer() {
+  assertNull(MetricsEnvironment.getCurrentContainer());
+  // Should not fail even though there is no metrics container.
+  Metrics.distribution(NS, NAME).update(5L);
+}
+@Test
+public void testCounterWithoutContainer() {
+  assertNull(MetricsEnvironment.getCurrentContainer());
+  // Should n

[jira] [Work logged] (BEAM-4065) Performance Tests Results Analysis and Regression Detection

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4065?focusedWorklogId=101639&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101639
 ]

ASF GitHub Bot logged work on BEAM-4065:


Author: ASF GitHub Bot
Created on: 14/May/18 09:07
Start Date: 14/May/18 09:07
Worklog Time Spent: 10m 
  Work Description: szewi commented on issue #5180: [BEAM-4065] Basic 
performance tests analysis added.
URL: https://github.com/apache/beam/pull/5180#issuecomment-388748573
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101639)
Time Spent: 7h 40m  (was: 7.5h)

> Performance Tests Results Analysis and Regression Detection
> ---
>
> Key: BEAM-4065
> URL: https://issues.apache.org/jira/browse/BEAM-4065
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kamil Szewczyk
>Assignee: Kamil Szewczyk
>Priority: Major
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> Performance tests are running on Jenkins on regular basis and results are 
> pushed to BigQuery. However there is no automatic regression detection or 
> daily reports with results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4065) Performance Tests Results Analysis and Regression Detection

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4065?focusedWorklogId=101640&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101640
 ]

ASF GitHub Bot logged work on BEAM-4065:


Author: ASF GitHub Bot
Created on: 14/May/18 09:13
Start Date: 14/May/18 09:13
Worklog Time Spent: 10m 
  Work Description: szewi commented on issue #5180: [BEAM-4065] Basic 
performance tests analysis added.
URL: https://github.com/apache/beam/pull/5180#issuecomment-388750307
 
 
   Run Performance Tests Analysis


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101640)
Time Spent: 7h 50m  (was: 7h 40m)

> Performance Tests Results Analysis and Regression Detection
> ---
>
> Key: BEAM-4065
> URL: https://issues.apache.org/jira/browse/BEAM-4065
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kamil Szewczyk
>Assignee: Kamil Szewczyk
>Priority: Major
>  Time Spent: 7h 50m
>  Remaining Estimate: 0h
>
> Performance tests are running on Jenkins on regular basis and results are 
> pushed to BigQuery. However there is no automatic regression detection or 
> daily reports with results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4958

[Apache Jenkins Server]

See 


--
[...truncated 1.01 MB...]
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/

[jira] [Work logged] (BEAM-4065) Performance Tests Results Analysis and Regression Detection

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4065?focusedWorklogId=101654&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101654
 ]

ASF GitHub Bot logged work on BEAM-4065:


Author: ASF GitHub Bot
Created on: 14/May/18 10:12
Start Date: 14/May/18 10:12
Worklog Time Spent: 10m 
  Work Description: szewi commented on a change in pull request #5180: 
[BEAM-4065] Basic performance tests analysis added.
URL: https://github.com/apache/beam/pull/5180#discussion_r187900175
 
 

 ##
 File path: .test-infra/jenkins/verify_performance_test_results_test.py
 ##
 @@ -0,0 +1,62 @@
+#!/usr/bin/env python
+#
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+#   This script performs testing of scenarios from 
verify_performance_test_results.py
+#
+
+import unittest, mock
+from mock import patch
+from verify_performance_test_results import create_report
+
+class VerifyAnalysisScript(unittest.TestCase):
+"""Tests for `verify_performance_test_results.py`."""
+
+@patch('verify_performance_test_results.count_queries', return_value=0)
+def test_create_daily_report_when_no_data_was_uploaded(self, *args):
+"""Testing creating report when no data was uploaded. Expected: Error 
message"""
+output_message = create_report(["test_bq_table"], "test", False)
+assert "no tests results uploaded in recent 24h." in output_message
+
+@patch('verify_performance_test_results.count_queries', return_value=1)
+@patch('verify_performance_test_results.get_average_from', return_value=10)
+@patch('verify_performance_test_results.get_stddev_from', return_value=10)
+def test_create_daily_report_when_single_entry_was_uploaded(self, *args):
+"""Testing stddev value when single data entry was uploaded. Expected: 
0"""
+output_message = create_report(["test_bq_table"], "test", False)
+assert ", stddev 0.00" in output_message
+
+@patch('verify_performance_test_results.count_queries', side_effect=[1, 0])
+@patch('verify_performance_test_results.get_average_from', return_value=10)
+@patch('verify_performance_test_results.get_stddev_from', return_value=10)
+def test_create_daily_report_when_no_historical_data_was_uploaded(self, 
*args):
+"""Testing output when no historical data is available. Expected: no 
message."""
+output_message = create_report(["test_bq_table"], "test", False)
+self.assertEqual(output_message, "")
+
+@patch('verify_performance_test_results.count_queries', side_effect=[5, 5])
 
 Review comment:
   Ok, those tests take no longer than a second, so I will add them as a part 
of Performance Analysis job to be executed before running 
verify_performance_test_results.py script.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101654)
Time Spent: 8h  (was: 7h 50m)

> Performance Tests Results Analysis and Regression Detection
> ---
>
> Key: BEAM-4065
> URL: https://issues.apache.org/jira/browse/BEAM-4065
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kamil Szewczyk
>Assignee: Kamil Szewczyk
>Priority: Major
>  Time Spent: 8h
>  Remaining Estimate: 0h
>
> Performance tests are running on Jenkins on regular basis and results are 
> pushed to BigQuery. However there is no automatic regression detection or 
> daily reports with results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4065) Performance Tests Results Analysis and Regression Detection

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4065?focusedWorklogId=101662&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101662
 ]

ASF GitHub Bot logged work on BEAM-4065:


Author: ASF GitHub Bot
Created on: 14/May/18 10:31
Start Date: 14/May/18 10:31
Worklog Time Spent: 10m 
  Work Description: szewi commented on a change in pull request #5180: 
[BEAM-4065] Basic performance tests analysis added.
URL: https://github.com/apache/beam/pull/5180#discussion_r187904743
 
 

 ##
 File path: .test-infra/jenkins/verify_performance_test_results.py
 ##
 @@ -0,0 +1,252 @@
+#!/usr/bin/env python
+#
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+#   This script performs basic analytic of performance tests results.
+#   It operates in two modes:
+#   --mode=report - In this mode script iterates over list of BigQuery tables 
and
+#   analyses the data. This mode is intended to be run on a regulars basis, 
e.g. daily.
+#   Report will contain average tests execution time of given metric, its 
comparison with
+#   with average calculated from historical data, recent standard deviation 
and standard
+#   deviation calculated based on historical data.
+#   --mode=validation - In this mode script will analyse single BigQuery table 
and check
+#   recent results.
+#
+#   Other parameters are described in script. Notification is optional 
parameter.
+#   --send_notification - if present, script will send notification to slack 
channel.
+#   Requires setting env variable SLACK_WEBOOK_URL which value could be 
obtained by
+#   creating incoming webhook on Slack.
+#
+#   This script is intended to be used only by Jenkins.
+#   Example script usage:
+#   verify_performance_test_results.py \
+# --bqtable='["beam_performance.avroioit_hdfs_pkb_results", \
+# "beam_performance.textioit_pkb_results"]' \
+# --metric="run_time" --mode=report --send_notification
+#
+
+import argparse, time, calendar, json, re, os, requests
+from google.cloud import bigquery
+
+### TIME SETTINGS ###
+TIME_PATTERN = '%d-%m-%Y_%H-%M-%S'
+NOW = int(time.time())
+# First analysis time interval definition - 24h before
+TIME_POINT_1 = NOW - 1 * 86400
+# Second analysis time interval definition - week before
+TIME_POINT_2 = NOW - 7 * 86400
+##
+
+SLACK_USER = os.getenv('SLACK_USER', "jenkins-beam")
+SLACK_WEBHOOK_URL = os.getenv('SLACK_WEBHOOK_URL')
+SLACK_CHANNEL = os.getenv('SLACK_CHANNEL', "beam-testing")
+
+def submit_big_query_job(sql_command, return_type):
+query_job = client.query(sql_command)
 
 Review comment:
   Retry is implement in big query client python library. Default setting is 6 
retries. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101662)
Time Spent: 8h 10m  (was: 8h)

> Performance Tests Results Analysis and Regression Detection
> ---
>
> Key: BEAM-4065
> URL: https://issues.apache.org/jira/browse/BEAM-4065
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kamil Szewczyk
>Assignee: Kamil Szewczyk
>Priority: Major
>  Time Spent: 8h 10m
>  Remaining Estimate: 0h
>
> Performance tests are running on Jenkins on regular basis and results are 
> pushed to BigQuery. However there is no automatic regression detection or 
> daily reports with results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #228

[Apache Jenkins Server]

See 


--
[...truncated 1.49 MB...]
mvn compile exec:java -q   
-Dexec.mainClass=org.apache.beam.examples.WordCount   
-Dexec.args="--runner=DataflowRunner
--project=apache-beam-testing
--gcpTempLocation=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp

--output=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts
--inputFile=gs://apache-beam-samples/shakespeare/*" 
-Pdataflow-runner
Using maven /home/jenkins/tools/maven/apache-maven-3.5.2
May 14, 2018 11:06:24 AM 
org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory
 create
INFO: No stagingLocation provided, falling back to gcpTempLocation
May 14, 2018 11:06:25 AM org.apache.beam.runners.dataflow.DataflowRunner 
fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from 
the classpath: will stage 109 files. Enable logging at DEBUG level to see which 
files will be staged.
May 14, 2018 11:06:25 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing 
implications related to Google Compute Engine usage and other Google Cloud 
Services.
May 14, 2018 11:06:25 AM org.apache.beam.runners.dataflow.util.PackageUtil 
stageClasspathElements
INFO: Uploading 109 files from PipelineOptions.filesToStage to staging location 
to prepare for execution.
May 14, 2018 11:06:26 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/tmp/groovy-generated-6531596002284788303-tmpdir/word-count-beam/target/classes 
to 
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/classes-1U19THXLahlQ3PU4jmxCtA.jar
May 14, 2018 11:06:26 AM org.apache.beam.runners.dataflow.util.PackageUtil 
stageClasspathElements
INFO: Staging files complete: 108 files cached, 1 files newly uploaded
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ReadLines/Read as step s1
May 14, 2018 11:06:27 AM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern gs://apache-beam-samples/shakespeare/* matched 43 files with 
total size 5284696
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WordCount.CountWords/ParDo(ExtractWords) as step s2
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WordCount.CountWords/Count.PerElement/Init/Map as step s3
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey as step 
s4
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues
 as step s5
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding MapElements/Map as step s6
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign as step s7
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles 
as step s8
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten as step 
s9
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten as step 
s10
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum as step s11
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections as 
step s12
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)
 as step s13
May 14, 2018 11:06:27 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView as 
step s14

[jira] [Work logged] (BEAM-4004) Futurize and fix python 2 compatibility for testing subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4004?focusedWorklogId=101688&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101688
 ]

ASF GitHub Bot logged work on BEAM-4004:


Author: ASF GitHub Bot
Created on: 14/May/18 11:25
Start Date: 14/May/18 11:25
Worklog Time Spent: 10m 
  Work Description: Fematich opened a new pull request #5352: [BEAM-4004] 
Futurize testing subpackage
URL: https://github.com/apache/beam/pull/5352
 
 
   This pull request prepares the testing subpackage for Python 3 support. This 
PR is part of a series in which all subpackages will be updated using the same 
approach.
   This approach has been documented 
[here](https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE/edit#heading=h.5vrnrgpkrk89)
 and the first pull request in the series (Futurize coders subpackage) 
demonstrating this approach can be found at #5053.
   
   R: @aaltay @tvalentyn @RobbeSneyders


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101688)
Time Spent: 10m
Remaining Estimate: 0h

> Futurize and fix python 2 compatibility for testing subpackage
> --
>
> Key: BEAM-4004
> URL: https://issues.apache.org/jira/browse/BEAM-4004
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4065) Performance Tests Results Analysis and Regression Detection

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4065?focusedWorklogId=101691&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101691
 ]

ASF GitHub Bot logged work on BEAM-4065:


Author: ASF GitHub Bot
Created on: 14/May/18 11:29
Start Date: 14/May/18 11:29
Worklog Time Spent: 10m 
  Work Description: szewi commented on a change in pull request #5180: 
[BEAM-4065] Basic performance tests analysis added.
URL: https://github.com/apache/beam/pull/5180#discussion_r187917973
 
 

 ##
 File path: .test-infra/jenkins/verify_performance_test_results.py
 ##
 @@ -0,0 +1,252 @@
+#!/usr/bin/env python
+#
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+#   This script performs basic analytic of performance tests results.
+#   It operates in two modes:
+#   --mode=report - In this mode script iterates over list of BigQuery tables 
and
+#   analyses the data. This mode is intended to be run on a regulars basis, 
e.g. daily.
+#   Report will contain average tests execution time of given metric, its 
comparison with
+#   with average calculated from historical data, recent standard deviation 
and standard
+#   deviation calculated based on historical data.
+#   --mode=validation - In this mode script will analyse single BigQuery table 
and check
+#   recent results.
+#
+#   Other parameters are described in script. Notification is optional 
parameter.
+#   --send_notification - if present, script will send notification to slack 
channel.
+#   Requires setting env variable SLACK_WEBOOK_URL which value could be 
obtained by
+#   creating incoming webhook on Slack.
+#
+#   This script is intended to be used only by Jenkins.
+#   Example script usage:
+#   verify_performance_test_results.py \
+# --bqtable='["beam_performance.avroioit_hdfs_pkb_results", \
+# "beam_performance.textioit_pkb_results"]' \
+# --metric="run_time" --mode=report --send_notification
+#
+
+import argparse, time, calendar, json, re, os, requests
+from google.cloud import bigquery
+
+### TIME SETTINGS ###
+TIME_PATTERN = '%d-%m-%Y_%H-%M-%S'
+NOW = int(time.time())
+# First analysis time interval definition - 24h before
+TIME_POINT_1 = NOW - 1 * 86400
+# Second analysis time interval definition - week before
+TIME_POINT_2 = NOW - 7 * 86400
+##
+
+SLACK_USER = os.getenv('SLACK_USER', "jenkins-beam")
+SLACK_WEBHOOK_URL = os.getenv('SLACK_WEBHOOK_URL')
+SLACK_CHANNEL = os.getenv('SLACK_CHANNEL', "beam-testing")
+
+def submit_big_query_job(sql_command, return_type):
+query_job = client.query(sql_command)
+results = query_job.result()
+if return_type == "list":
+# Queries that have multiple elements in output displayed as 
query_result
+result_list = []
+for row in results:
+result_list.append(row.query_result)
+return result_list
+elif return_type == "value":
+# All queries must have single element in output displayed as 
query_result
+for row in results:
 
 Review comment:
   Yes, currently my script is only returning single values as a result of 
queries, but in current implementation is possible to return list of values. 
The method `get_records_from` is using it, but I am never using this 
method(yet?). Should I remove it and simplify this logic?



This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101691)
Time Spent: 8h 20m  (was: 8h 10m)

> Performance Tests Results Analysis and Regression Detection
> ---
>
> Key: BEAM-4065
> URL: https://issues.apache.org/jira/browse/BEAM-4065
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Repo

[jira] [Work logged] (BEAM-4065) Performance Tests Results Analysis and Regression Detection

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4065?focusedWorklogId=101693&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101693
 ]

ASF GitHub Bot logged work on BEAM-4065:


Author: ASF GitHub Bot
Created on: 14/May/18 11:30
Start Date: 14/May/18 11:30
Worklog Time Spent: 10m 
  Work Description: szewi commented on a change in pull request #5180: 
[BEAM-4065] Basic performance tests analysis added.
URL: https://github.com/apache/beam/pull/5180#discussion_r187918302
 
 

 ##
 File path: .test-infra/jenkins/verify_performance_test_results.py
 ##
 @@ -0,0 +1,252 @@
+#!/usr/bin/env python
+#
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+#   This script performs basic analytic of performance tests results.
+#   It operates in two modes:
+#   --mode=report - In this mode script iterates over list of BigQuery tables 
and
+#   analyses the data. This mode is intended to be run on a regulars basis, 
e.g. daily.
+#   Report will contain average tests execution time of given metric, its 
comparison with
+#   with average calculated from historical data, recent standard deviation 
and standard
+#   deviation calculated based on historical data.
+#   --mode=validation - In this mode script will analyse single BigQuery table 
and check
+#   recent results.
+#
+#   Other parameters are described in script. Notification is optional 
parameter.
+#   --send_notification - if present, script will send notification to slack 
channel.
+#   Requires setting env variable SLACK_WEBOOK_URL which value could be 
obtained by
+#   creating incoming webhook on Slack.
+#
+#   This script is intended to be used only by Jenkins.
+#   Example script usage:
+#   verify_performance_test_results.py \
+# --bqtable='["beam_performance.avroioit_hdfs_pkb_results", \
+# "beam_performance.textioit_pkb_results"]' \
+# --metric="run_time" --mode=report --send_notification
+#
+
+import argparse, time, calendar, json, re, os, requests
+from google.cloud import bigquery
+
+### TIME SETTINGS ###
+TIME_PATTERN = '%d-%m-%Y_%H-%M-%S'
+NOW = int(time.time())
+# First analysis time interval definition - 24h before
+TIME_POINT_1 = NOW - 1 * 86400
+# Second analysis time interval definition - week before
+TIME_POINT_2 = NOW - 7 * 86400
+##
+
+SLACK_USER = os.getenv('SLACK_USER', "jenkins-beam")
+SLACK_WEBHOOK_URL = os.getenv('SLACK_WEBHOOK_URL')
+SLACK_CHANNEL = os.getenv('SLACK_CHANNEL', "beam-testing")
+
+def submit_big_query_job(sql_command, return_type):
+query_job = client.query(sql_command)
+results = query_job.result()
+if return_type == "list":
+# Queries that have multiple elements in output displayed as 
query_result
+result_list = []
+for row in results:
+result_list.append(row.query_result)
+return result_list
+elif return_type == "value":
+# All queries must have single element in output displayed as 
query_result
+for row in results:
 
 Review comment:
   I will do that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101693)
Time Spent: 8.5h  (was: 8h 20m)

> Performance Tests Results Analysis and Regression Detection
> ---
>
> Key: BEAM-4065
> URL: https://issues.apache.org/jira/browse/BEAM-4065
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kamil Szewczyk
>Assignee: Kamil Szewczyk
>Priority: Major
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> Performance tests are running on Jenkins on regular basis and results are 
> pushed to BigQuery. However there is no automat

[jira] [Assigned] (BEAM-4003) Futurize and fix python 2 compatibility for runners subpackage

[Matthias Feys (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Feys reassigned BEAM-4003:
---

Assignee: Matthias Feys  (was: Ahmet Altay)

> Futurize and fix python 2 compatibility for runners subpackage
> --
>
> Key: BEAM-4003
> URL: https://issues.apache.org/jira/browse/BEAM-4003
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (bb319db -> 7a3f688)

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from bb319db  Merge pull request #5277: [BEAM-3515] Portable translation of 
SplittableProcessKeyed
 add aeae5c0  [BEAM-3813] Refactor constructors to use only S3Options 
instead of each param
 add 3d169fc  [BEAM-3813] Update aws-java-sdk version to version 1.11.319
 add 6473b91  [BEAM-3813] Add support for SSE-C in S3 Filesystem
 add 01d3bd7  [BEAM-3813] Add support for SSE-S3 in S3 Filesystem
 add 3e834c6  [BEAM-3813] Add support for SSE-KMS in S3 Filesystem
 add 5fb21de  [BEAM-3813] Fix error-prone and other warnings for the AWS 
module
 new 7a3f688  Merge pull request #5244 from iemejia/BEAM-3813-s3-sse

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/io/amazon-web-services/build.gradle  |   4 +-
 sdks/java/io/amazon-web-services/pom.xml   |  15 +-
 .../apache/beam/sdk/io/aws/options/AwsModule.java  |  19 +-
 .../apache/beam/sdk/io/aws/options/S3Options.java  |  39 ++-
 .../apache/beam/sdk/io/aws/s3/S3FileSystem.java| 128 +-
 .../beam/sdk/io/aws/s3/S3FileSystemRegistrar.java  |   2 +-
 .../io/aws/s3/S3ReadableSeekableByteChannel.java   |  15 +-
 .../beam/sdk/io/aws/s3/S3WritableByteChannel.java  |  52 +++-
 .../beam/sdk/io/aws/options/AwsModuleTest.java |   2 +-
 .../beam/sdk/io/aws/s3/MatchResultMatcher.java |   4 +-
 .../beam/sdk/io/aws/s3/S3FileSystemTest.java   | 262 +++--
 .../beam/sdk/io/aws/s3/S3ResourceIdTest.java   |  41 ++--
 .../org/apache/beam/sdk/io/aws/s3/S3TestUtils.java |  82 +++
 .../sdk/io/aws/s3/S3WritableByteChannelTest.java   | 102 ++--
 14 files changed, 561 insertions(+), 206 deletions(-)
 create mode 100644 
sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/s3/S3TestUtils.java

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[beam] 01/01: Merge pull request #5244 from iemejia/BEAM-3813-s3-sse

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7a3f6885c69649d0428269bfc4b86ab130de99e8
Merge: bb319db 5fb21de
Author: Jean-Baptiste Onofré 
AuthorDate: Mon May 14 14:03:50 2018 +0200

Merge pull request #5244 from iemejia/BEAM-3813-s3-sse

[BEAM-3813] Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)

 sdks/java/io/amazon-web-services/build.gradle  |   4 +-
 sdks/java/io/amazon-web-services/pom.xml   |  15 +-
 .../apache/beam/sdk/io/aws/options/AwsModule.java  |  19 +-
 .../apache/beam/sdk/io/aws/options/S3Options.java  |  39 ++-
 .../apache/beam/sdk/io/aws/s3/S3FileSystem.java| 128 +-
 .../beam/sdk/io/aws/s3/S3FileSystemRegistrar.java  |   2 +-
 .../io/aws/s3/S3ReadableSeekableByteChannel.java   |  15 +-
 .../beam/sdk/io/aws/s3/S3WritableByteChannel.java  |  52 +++-
 .../beam/sdk/io/aws/options/AwsModuleTest.java |   2 +-
 .../beam/sdk/io/aws/s3/MatchResultMatcher.java |   4 +-
 .../beam/sdk/io/aws/s3/S3FileSystemTest.java   | 262 +++--
 .../beam/sdk/io/aws/s3/S3ResourceIdTest.java   |  41 ++--
 .../org/apache/beam/sdk/io/aws/s3/S3TestUtils.java |  82 +++
 .../sdk/io/aws/s3/S3WritableByteChannelTest.java   | 102 ++--
 14 files changed, 561 insertions(+), 206 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[jira] [Work logged] (BEAM-3813) Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3813?focusedWorklogId=101702&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101702
 ]

ASF GitHub Bot logged work on BEAM-3813:


Author: ASF GitHub Bot
Created on: 14/May/18 12:03
Start Date: 14/May/18 12:03
Worklog Time Spent: 10m 
  Work Description: jbonofre closed pull request #5244: [BEAM-3813] Support 
encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)
URL: https://github.com/apache/beam/pull/5244
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/io/amazon-web-services/build.gradle 
b/sdks/java/io/amazon-web-services/build.gradle
index 10d016037f3..53b674bdb6f 100644
--- a/sdks/java/io/amazon-web-services/build.gradle
+++ b/sdks/java/io/amazon-web-services/build.gradle
@@ -22,7 +22,7 @@ applyJavaNature()
 description = "Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services"
 ext.summary = "IO library to read and write Amazon Web Services services from 
Beam."
 
-def aws_java_sdk_version = "1.11.255"
+def aws_java_sdk_version = "1.11.319"
 
 dependencies {
   compile library.java.guava
@@ -34,7 +34,9 @@ dependencies {
   shadow library.java.jackson_databind
   shadow library.java.findbugs_jsr305
   shadow library.java.slf4j_api
+  runtime 'commons-codec:commons-codec:1.9'
   runtime "org.apache.httpcomponents:httpclient:4.5.2"
+  testCompile project(path: ":beam-runners-direct-java", configuration: 
"shadow")
   shadowTest library.java.guava_testlib
   shadowTest library.java.hamcrest_core
   shadowTest library.java.mockito_core
diff --git a/sdks/java/io/amazon-web-services/pom.xml 
b/sdks/java/io/amazon-web-services/pom.xml
index f76d5a7e23a..d76f5394200 100644
--- a/sdks/java/io/amazon-web-services/pom.xml
+++ b/sdks/java/io/amazon-web-services/pom.xml
@@ -33,7 +33,7 @@
   jar
 
   
-1.11.255
+1.11.319
   
 
   
@@ -77,6 +77,13 @@
   ${aws-java-sdk.version}
 
 
+
+  commons-codec
+  commons-codec
+  1.9
+  runtime
+
+
 
   com.fasterxml.jackson.core
   jackson-core
@@ -142,6 +149,12 @@
   test
 
 
+
+  org.apache.beam
+  beam-runners-direct-java
+  test
+
+
 
   org.hamcrest
   hamcrest-core
diff --git 
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
 
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
index 228ab5c8b85..0c7ce5b292c 100644
--- 
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
+++ 
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
@@ -47,7 +47,6 @@
 import java.io.IOException;
 import java.lang.reflect.Field;
 import java.util.Map;
-import java.util.Set;
 
 /**
  * A Jackson {@link Module} that registers a {@link JsonSerializer} and {@link 
JsonDeserializer}
@@ -75,7 +74,8 @@ public AwsModule() {
 
   }
 
-  static class AWSCredentialsProviderDeserializer extends 
JsonDeserializer {
+  private static class AWSCredentialsProviderDeserializer
+  extends JsonDeserializer {
 
 @Override
 public AWSCredentialsProvider deserialize(
@@ -123,15 +123,14 @@ public AWSCredentialsProvider 
deserializeWithType(JsonParser jsonParser,
   }
 
   static class AWSCredentialsProviderSerializer extends 
JsonSerializer {
-
 // These providers are singletons, so don't require any serialization, 
other than type.
-private static final Set SINGLETON_CREDENTIAL_PROVIDERS = 
ImmutableSet.of(
-DefaultAWSCredentialsProviderChain.class,
-EnvironmentVariableCredentialsProvider.class,
-SystemPropertiesCredentialsProvider.class,
-ProfileCredentialsProvider.class,
-EC2ContainerCredentialsProviderWrapper.class
-);
+private static final ImmutableSet SINGLETON_CREDENTIAL_PROVIDERS =
+ImmutableSet.of(
+DefaultAWSCredentialsProviderChain.class,
+EnvironmentVariableCredentialsProvider.class,
+SystemPropertiesCredentialsProvider.class,
+ProfileCredentialsProvider.class,
+EC2ContainerCredentialsProviderWrapper.class);
 
 @Override
 public void serialize(AWSCredentialsProvider credentialsProvider, 
JsonGenerator jsonGenerator,
diff --git 
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/S3Options.java
 
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/S3Options.java
index 253272898a1..22ab91ce39f 100644
--- 
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/S3Options.java
+++

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1625

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-3813] Refactor constructors to use only S3Options instead of 
each

[iemejia] [BEAM-3813] Update aws-java-sdk version to version 1.11.319

[iemejia] [BEAM-3813] Add support for SSE-C in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-S3 in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-KMS in S3 Filesystem

[iemejia] [BEAM-3813] Fix error-prone and other warnings for the AWS module

--
Started by GitHub push by jbonofre
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7a3f6885c69649d0428269bfc4b86ab130de99e8 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7a3f6885c69649d0428269bfc4b86ab130de99e8
Commit message: "Merge pull request #5244 from iemejia/BEAM-3813-s3-sse"
 > git rev-list --no-walk bb319dbfaa81c27d7ba6b0a465994aa656143100 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins3831701452755375900.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ekirpic...@gmail.com

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #174

[Apache Jenkins Server]

See 


--
[...truncated 54.77 KB...]
> Task :beam-model-pipeline:processTestResources NO-SOURCE
> Task :beam-model-pipeline:testClasses UP-TO-DATE
> Task :beam-model-pipeline:packageTests
> Task :beam-model-job-management:generateProto
> Task :beam-model-fn-execution:generateProto
> Task :beam-model-pipeline:install
> Task :beam-model-job-management:compileJava FROM-CACHE
> Task :beam-model-job-management:classes
> Task :beam-model-fn-execution:compileJava FROM-CACHE
> Task :beam-model-fn-execution:classes

> Task :beam-sdks-python:setupVirtualenv
Requirement not upgraded as not directly required: virtualenv>=1.11.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (15.2.0)
Requirement not upgraded as not directly required: py>=1.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from tox==3.0.0) (1.5.3)
Collecting grpcio>=1.3.5 (from grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/0d/54/b647a6323be6526be27b2c90bb042769f1a7a6e59bd1a5f2eeb795bfece4/grpcio-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting protobuf>=3.2.0 (from grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/9d/61/54c3a9cfde6ffe0ca6a1786ddb8874263f4ca32e7693ad383bd8cf935015/protobuf-3.5.2.post1-cp27-cp27mu-manylinux1_x86_64.whl
Requirement not upgraded as not directly required: enum34>=1.0.4 in 
/usr/local/lib/python2.7/dist-packages (from 
grpcio>=1.3.5->grpcio-tools==1.3.5) (1.1.6)
Collecting futures>=2.2.0 (from grpcio>=1.3.5->grpcio-tools==1.3.5)
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement not upgraded as not directly required: setuptools in 
/usr/local/lib/python2.7/dist-packages (from 
protobuf>=3.2.0->grpcio-tools==1.3.5) (39.0.1)
Installing collected packages: protobuf, futures, grpcio, grpcio-tools
Could not install packages due to an EnvironmentError: [Errno 13] Permission 
denied: 
'/usr/local/lib/python2.7/dist-packages/protobuf-3.5.2.post1-py2.7-nspkg.pth'
Consider using the `--user` option or check the permissions.


> Task :beam-sdks-python:setupVirtualenv FAILED
> Task :beam-model-job-management:shadowJar
> Task :beam-model-fn-execution:shadowJar
> Task :beam-sdks-java-core:shadowJar

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 5.0.
See 
https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 49s
132 actionable tasks: 125 executed, 5 from cache, 2 up-to-date

Publishing build scan...
https://gradle.com/s/hraqxwjub6xm2


STDERR: 
FAILURE: Build failed with an exception.

* Where:
Build file 
'
 line: 36

* What went wrong:
Execution failed for task ':beam-sdks-python:setupVirtualenv'.
> Process 'command 'sh'' finished with non-zero exit value 1

* Try:
Run with --info or --debug option to get more log output. Run with --scan to 
get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
':beam-sdks-python:setupVirtualenv'.
at 
org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:103)
at 
org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:73)
at 
org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
at 
org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
at 
org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
at 
org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
at 
org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
at 
org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
at 
org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
at 
org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
at 
org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
at 
org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
at 
org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecu

[jira] [Resolved] (BEAM-3813) Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-3813.

   Resolution: Fixed
Fix Version/s: 2.5.0

> Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)
> ---
>
> Key: BEAM-3813
> URL: https://issues.apache.org/jira/browse/BEAM-3813
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> We should enable AWS S3 users to use encryption when reading or writing by 
> provide encryption keys or using server side encryption via an algorithm, or 
> a key management system (KMS)..
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #571

[Apache Jenkins Server]

See 


--
[...truncated 91.97 KB...]
:beam-runners-google-cloud-dataflow-java:classes (Thread[Task worker for ':' 
Thread 13,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for 
':',5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:compileTestJava' is 
c1c6fd31b046bb0f3afb97608f211252
Skipping task ':beam-sdks-java-io-google-cloud-platform:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 5,5,main]) completed. Took 0.071 secs.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 5,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-google-cloud-platform:testClasses' as it has 
no actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 5,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 5,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
e6e2998e3880c5ecbd3a90abb9ac0b26
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 5,5,main]) completed. Took 0.034 secs.

> Task :beam-runners-google-cloud-dataflow-java:shadowJar UP-TO-DATE
Build cache key for task ':beam-runners-google-cloud-dataflow-java:shadowJar' 
is 7c7090f6d645b1d045f1db05641b3b6e
Caching disabled for task ':beam-runners-google-cloud-dataflow-java:shadowJar': 
Caching has not been enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for 
':',5,main]) completed. Took 0.047 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':',5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:compileTestJava' is 
cd13737fab393925d812180591d707ac
Skipping task ':beam-runners-google-cloud-dataflow-java:compileTestJava' as it 
is up-to-date.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':',5,main]) completed. Took 0.071 secs.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':',5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:testClasses UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:testClasses' as it has 
no actions.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':',5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 13,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar' is 
b54a40487a14b8a8d8ac76b2680fdcde
Caching disabled for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowTestJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 13,5,main]) completed. Took 0.04 secs.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
13,5,main]) started.

> Task :beam-sdks-java-io-jdbc:compileTestJava UP-TO-DATE
Build cache key for task ':beam-sdks-java-io-jdbc:compileTestJava' is 
e68668bd6eb3b90466497a2ad63c419d
Skipping task ':beam-sdks-java-io-jdbc:compileTestJava' as it is up-to-date.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
13,5,main]) completed. Took 0.062 secs.
:beam-sdks-java-io-jdbc:testClasses (Thread[Task worker for ':' Thread 
13,5,main]) started.

> Task :beam-sdks-java-io-jdbc:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-jdbc:testClasses' as it has no actions.
:beam-sdks-java-io-jdbc:testClasses (Thread[Task worker for ':' Thread 
13,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-jdbc:integrationTest (Thread[Task worker for ':' Thread 
13,5,main]) started.
Gradle Test Executor 1 started executing tests.

> Task :beam-sdks-java-io-jdbc:integrationTest
Build cache key for task ':beam-sdks-java-io-jdbc:integrationTest' is 
57a8f75fb5eebab41301e

[jira] [Work logged] (BEAM-4004) Futurize and fix python 2 compatibility for testing subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4004?focusedWorklogId=101703&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101703
 ]

ASF GitHub Bot logged work on BEAM-4004:


Author: ASF GitHub Bot
Created on: 14/May/18 12:07
Start Date: 14/May/18 12:07
Worklog Time Spent: 10m 
  Work Description: Fematich commented on issue #5352: [BEAM-4004] Futurize 
testing subpackage
URL: https://github.com/apache/beam/pull/5352#issuecomment-388794163
 
 
   run python precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101703)
Time Spent: 20m  (was: 10m)

> Futurize and fix python 2 compatibility for testing subpackage
> --
>
> Key: BEAM-4004
> URL: https://issues.apache.org/jira/browse/BEAM-4004
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT_HDFS #167

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #169

[Apache Jenkins Server]

See 


--
[...truncated 456.10 KB...]
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:129)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.202.118.20:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.202.118.20:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.g

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #168

[Apache Jenkins Server]

See 


--
[...truncated 886.24 KB...]
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.crea

[jira] [Work logged] (BEAM-4065) Performance Tests Results Analysis and Regression Detection

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4065?focusedWorklogId=101709&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101709
 ]

ASF GitHub Bot logged work on BEAM-4065:


Author: ASF GitHub Bot
Created on: 14/May/18 12:21
Start Date: 14/May/18 12:21
Worklog Time Spent: 10m 
  Work Description: szewi commented on a change in pull request #5180: 
[BEAM-4065] Basic performance tests analysis added.
URL: https://github.com/apache/beam/pull/5180#discussion_r187932632
 
 

 ##
 File path: .test-infra/jenkins/verify_performance_test_results.py
 ##
 @@ -0,0 +1,252 @@
+#!/usr/bin/env python
+#
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+#   This script performs basic analytic of performance tests results.
+#   It operates in two modes:
+#   --mode=report - In this mode script iterates over list of BigQuery tables 
and
+#   analyses the data. This mode is intended to be run on a regulars basis, 
e.g. daily.
+#   Report will contain average tests execution time of given metric, its 
comparison with
+#   with average calculated from historical data, recent standard deviation 
and standard
+#   deviation calculated based on historical data.
+#   --mode=validation - In this mode script will analyse single BigQuery table 
and check
+#   recent results.
+#
+#   Other parameters are described in script. Notification is optional 
parameter.
+#   --send_notification - if present, script will send notification to slack 
channel.
+#   Requires setting env variable SLACK_WEBOOK_URL which value could be 
obtained by
+#   creating incoming webhook on Slack.
+#
+#   This script is intended to be used only by Jenkins.
+#   Example script usage:
+#   verify_performance_test_results.py \
+# --bqtable='["beam_performance.avroioit_hdfs_pkb_results", \
+# "beam_performance.textioit_pkb_results"]' \
+# --metric="run_time" --mode=report --send_notification
+#
+
+import argparse, time, calendar, json, re, os, requests
+from google.cloud import bigquery
+
+### TIME SETTINGS ###
+TIME_PATTERN = '%d-%m-%Y_%H-%M-%S'
+NOW = int(time.time())
+# First analysis time interval definition - 24h before
+TIME_POINT_1 = NOW - 1 * 86400
+# Second analysis time interval definition - week before
+TIME_POINT_2 = NOW - 7 * 86400
+##
+
+SLACK_USER = os.getenv('SLACK_USER', "jenkins-beam")
+SLACK_WEBHOOK_URL = os.getenv('SLACK_WEBHOOK_URL')
+SLACK_CHANNEL = os.getenv('SLACK_CHANNEL', "beam-testing")
+
+def submit_big_query_job(sql_command, return_type):
+query_job = client.query(sql_command)
+results = query_job.result()
+if return_type == "list":
+# Queries that have multiple elements in output displayed as 
query_result
+result_list = []
+for row in results:
+result_list.append(row.query_result)
+return result_list
+elif return_type == "value":
+# All queries must have single element in output displayed as 
query_result
+for row in results:
+return (row.query_result)
+else:
+print("This type is not supported")
+return None
+
+def count_queries(table_name, time_start, time_stop, metric):
+# This function checks how many data was inserted in time interval.
+sql_command = 'select count(*) as query_result from {} where TIMESTAMP > 
{} and TIMESTAMP < {} and METRIC=\'{}\''.format(
+table_name,
+time_start,
+time_stop,
+metric
+)
+count = submit_big_query_job(sql_command, "value")
+print("Number of records inserted into {} between {} - {}: {}".format(
+table_name,
+time.strftime(TIME_PATTERN, time.gmtime(time_start)),
+time.strftime(TIME_PATTERN, time.gmtime(time_stop)),
+count))
+return count
+
+def get_average_from(table_name, time_start, time_stop, metric):
+# This function return average value of the provided metric in time 
interval.
+sql_command = 'select avg(value) as query_result from {} where TIMESTAMP > 
{} and TIMESTAMP < {} and METRIC=\'{}\''.format(
+table_nam

Build failed in Jenkins: beam_PostCommit_Python_Verify #4959

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-3813] Refactor constructors to use only S3Options instead of 
each

[iemejia] [BEAM-3813] Update aws-java-sdk version to version 1.11.319

[iemejia] [BEAM-3813] Add support for SSE-C in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-S3 in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-KMS in S3 Filesystem

[iemejia] [BEAM-3813] Fix error-prone and other warnings for the AWS module

--
[...truncated 1.01 MB...]
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/dir

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #381

[Apache Jenkins Server]

See 


--
[...truncated 18.92 MB...]
May 14, 2018 12:41:58 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <116033 bytes, hash YicQi0XMUH-KJwYxjyyCQQ> to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testreportfailures-jenkins-0514124151-b843e91d/output/results/staging/pipeline-YicQi0XMUH-KJwYxjyyCQQ.pb

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 14, 2018 12:41:59 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-14_05_41_58-13251467843170674289?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-05-14_05_41_58-13251467843170674289

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 14, 2018 12:41:59 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-14_05_41_58-13251467843170674289
May 14, 2018 12:41:59 PM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-05-14_05_41_58-13251467843170674289 with 0 
expected assertions.
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:41:58.572Z: Autoscaling is enabled for job 
2018-05-14_05_41_58-13251467843170674289. The number of workers will be between 
1 and 1000.
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:41:58.610Z: Autoscaling was automatically enabled for 
job 2018-05-14_05_41_58-13251467843170674289.
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:42:01.068Z: Checking required Cloud APIs are enabled.
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:42:01.184Z: Checking permissions granted to controller 
Service Account.
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:42:05.680Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:42:06.097Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:42:06.342Z: Expanding GroupByKey operations into 
optimizable parts.
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:42:06.387Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:42:06.633Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:42:06.675Z: Elided trivial flatten 
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:42:06.721Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:42:06.765Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:42:06.810Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 14, 2018 12:42:09 PM 
org.apache.beam.runners.data

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #382

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-3813] Refactor constructors to use only S3Options instead of 
each

[iemejia] [BEAM-3813] Update aws-java-sdk version to version 1.11.319

[iemejia] [BEAM-3813] Add support for SSE-C in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-S3 in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-KMS in S3 Filesystem

[iemejia] [BEAM-3813] Fix error-prone and other warnings for the AWS module

--
[...truncated 19.68 MB...]
May 14, 2018 12:46:12 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <116033 bytes, hash fuPr29J5A196DzTl7xYwjg> to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testreportfailures-jenkins-0514124605-529f82a7/output/results/staging/pipeline-fuPr29J5A196DzTl7xYwjg.pb

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 14, 2018 12:46:13 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-14_05_46_12-16185307352620605622?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-05-14_05_46_12-16185307352620605622

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 14, 2018 12:46:13 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-14_05_46_12-16185307352620605622
May 14, 2018 12:46:13 PM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-05-14_05_46_12-16185307352620605622 with 0 
expected assertions.
May 14, 2018 12:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:46:12.703Z: Autoscaling is enabled for job 
2018-05-14_05_46_12-16185307352620605622. The number of workers will be between 
1 and 1000.
May 14, 2018 12:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:46:12.747Z: Autoscaling was automatically enabled for 
job 2018-05-14_05_46_12-16185307352620605622.
May 14, 2018 12:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:46:15.061Z: Checking required Cloud APIs are enabled.
May 14, 2018 12:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:46:15.243Z: Checking permissions granted to controller 
Service Account.
May 14, 2018 12:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:46:19.823Z: Worker configuration: n1-standard-1 in 
us-central1-c.
May 14, 2018 12:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:46:20.241Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 14, 2018 12:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:46:20.498Z: Expanding GroupByKey operations into 
optimizable parts.
May 14, 2018 12:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:46:20.545Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 14, 2018 12:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:46:20.805Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 14, 2018 12:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:46:20.853Z: Elided trivial flatten 
May 14, 2018 12:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:46:20.902Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 14, 2018 12:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:46:20.938Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 14, 2018 12:46:24 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T12:46:20.975Z: Fusing consumer SpannerIO.Write/Write 
mut

[jira] [Work logged] (BEAM-4005) Futurize and fix python 2 compatibility for tools subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4005?focusedWorklogId=101713&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101713
 ]

ASF GitHub Bot logged work on BEAM-4005:


Author: ASF GitHub Bot
Created on: 14/May/18 12:55
Start Date: 14/May/18 12:55
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5343: 
[BEAM-4005] Futurize tools subpackage
URL: https://github.com/apache/beam/pull/5343#discussion_r187941931
 
 

 ##
 File path: sdks/python/apache_beam/tools/distribution_counter_microbenchmark.py
 ##
 @@ -23,11 +23,16 @@
   python -m apache_beam.tools.distribution_counter_microbenchmark
 """
 
+from __future__ import absolute_import
+from __future__ import division
 from __future__ import print_function
 
 import random
 import sys
 import time
+from builtins import range
+
+from past.utils import old_div
 
 Review comment:
   Fixed in new commit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101713)
Time Spent: 40m  (was: 0.5h)

> Futurize and fix python 2 compatibility for tools subpackage
> 
>
> Key: BEAM-4005
> URL: https://issues.apache.org/jira/browse/BEAM-4005
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4065) Performance Tests Results Analysis and Regression Detection

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4065?focusedWorklogId=101724&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101724
 ]

ASF GitHub Bot logged work on BEAM-4065:


Author: ASF GitHub Bot
Created on: 14/May/18 13:33
Start Date: 14/May/18 13:33
Worklog Time Spent: 10m 
  Work Description: szewi commented on a change in pull request #5180: 
[BEAM-4065] Basic performance tests analysis added.
URL: https://github.com/apache/beam/pull/5180#discussion_r187953700
 
 

 ##
 File path: .test-infra/jenkins/verify_performance_test_results.py
 ##
 @@ -0,0 +1,252 @@
+#!/usr/bin/env python
+#
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+#   This script performs basic analytic of performance tests results.
+#   It operates in two modes:
+#   --mode=report - In this mode script iterates over list of BigQuery tables 
and
+#   analyses the data. This mode is intended to be run on a regulars basis, 
e.g. daily.
+#   Report will contain average tests execution time of given metric, its 
comparison with
+#   with average calculated from historical data, recent standard deviation 
and standard
+#   deviation calculated based on historical data.
+#   --mode=validation - In this mode script will analyse single BigQuery table 
and check
+#   recent results.
+#
+#   Other parameters are described in script. Notification is optional 
parameter.
+#   --send_notification - if present, script will send notification to slack 
channel.
+#   Requires setting env variable SLACK_WEBOOK_URL which value could be 
obtained by
+#   creating incoming webhook on Slack.
+#
+#   This script is intended to be used only by Jenkins.
+#   Example script usage:
+#   verify_performance_test_results.py \
+# --bqtable='["beam_performance.avroioit_hdfs_pkb_results", \
+# "beam_performance.textioit_pkb_results"]' \
+# --metric="run_time" --mode=report --send_notification
+#
+
+import argparse, time, calendar, json, re, os, requests
+from google.cloud import bigquery
+
+### TIME SETTINGS ###
+TIME_PATTERN = '%d-%m-%Y_%H-%M-%S'
+NOW = int(time.time())
+# First analysis time interval definition - 24h before
+TIME_POINT_1 = NOW - 1 * 86400
+# Second analysis time interval definition - week before
+TIME_POINT_2 = NOW - 7 * 86400
+##
+
+SLACK_USER = os.getenv('SLACK_USER', "jenkins-beam")
+SLACK_WEBHOOK_URL = os.getenv('SLACK_WEBHOOK_URL')
+SLACK_CHANNEL = os.getenv('SLACK_CHANNEL', "beam-testing")
+
+def submit_big_query_job(sql_command, return_type):
+query_job = client.query(sql_command)
+results = query_job.result()
+if return_type == "list":
+# Queries that have multiple elements in output displayed as 
query_result
+result_list = []
+for row in results:
+result_list.append(row.query_result)
+return result_list
+elif return_type == "value":
+# All queries must have single element in output displayed as 
query_result
+for row in results:
+return (row.query_result)
+else:
+print("This type is not supported")
+return None
+
+def count_queries(table_name, time_start, time_stop, metric):
 
 Review comment:
   Ok, I have added pydoc to all functions, except 
validate_single_performance_test as it is not used yet and its implementation 
may change in recent future. I will get back to working on this later this 
week. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101724)
Time Spent: 8h 50m  (was: 8h 40m)

> Performance Tests Results Analysis and Regression Detection
> ---
>
> Key: BEAM-4065
> URL: https://issues.apache.org/jira/browse/BEAM-4065
>   

[jira] [Work logged] (BEAM-4065) Performance Tests Results Analysis and Regression Detection

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4065?focusedWorklogId=101731&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101731
 ]

ASF GitHub Bot logged work on BEAM-4065:


Author: ASF GitHub Bot
Created on: 14/May/18 13:48
Start Date: 14/May/18 13:48
Worklog Time Spent: 10m 
  Work Description: szewi commented on issue #5180: [BEAM-4065] Basic 
performance tests analysis added.
URL: https://github.com/apache/beam/pull/5180#issuecomment-388822754
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101731)
Time Spent: 9h  (was: 8h 50m)

> Performance Tests Results Analysis and Regression Detection
> ---
>
> Key: BEAM-4065
> URL: https://issues.apache.org/jira/browse/BEAM-4065
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kamil Szewczyk
>Assignee: Kamil Szewczyk
>Priority: Major
>  Time Spent: 9h
>  Remaining Estimate: 0h
>
> Performance tests are running on Jenkins on regular basis and results are 
> pushed to BigQuery. However there is no automatic regression detection or 
> daily reports with results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3634) [SQL] Refactor BeamRelNodes into PTransforms

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3634?focusedWorklogId=101733&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101733
 ]

ASF GitHub Bot logged work on BEAM-3634:


Author: ASF GitHub Bot
Created on: 14/May/18 13:54
Start Date: 14/May/18 13:54
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #4705: [BEAM-3634] 
Refactor BeamRelNode to return a PTransform
URL: https://github.com/apache/beam/pull/4705#issuecomment-388824982
 
 
   @apilloud it occurred to me that this PR of mine probably is a source of 
some of the UI issues that you noticed. Before this PR I think the graph layout 
would have been more "top-to-bottom" and less "nested boxes".
   
   I think a middle ground would be best
   
- having the explicit PTransforms at each node is nice for getting the 
composite structure
- but the recursion into sub-nodes needs to happen outside of that 
PTransform's `apply` method to get top-to-bottom layout


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101733)
Time Spent: 2h 20m  (was: 2h 10m)

> [SQL] Refactor BeamRelNodes into PTransforms
> 
>
> Key: BEAM-3634
> URL: https://issues.apache.org/jira/browse/BEAM-3634
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> BeamRelNode exposes PCollection buildBeamPipeline() which builds 
> a pipeline when parsing.
> It feels like it should instead implement a 
> PTransform, PCollection> which would 
> receive a prepared PCollection, and apply sub-expressions instead of manually 
> invoking expression evaluation to get the input.
> And maybe consider building it lazily.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4065) Performance Tests Results Analysis and Regression Detection

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4065?focusedWorklogId=101734&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101734
 ]

ASF GitHub Bot logged work on BEAM-4065:


Author: ASF GitHub Bot
Created on: 14/May/18 13:54
Start Date: 14/May/18 13:54
Worklog Time Spent: 10m 
  Work Description: szewi commented on issue #5180: [BEAM-4065] Basic 
performance tests analysis added.
URL: https://github.com/apache/beam/pull/5180#issuecomment-388824992
 
 
   Run Performance Tests Analysis


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101734)
Time Spent: 9h 10m  (was: 9h)

> Performance Tests Results Analysis and Regression Detection
> ---
>
> Key: BEAM-4065
> URL: https://issues.apache.org/jira/browse/BEAM-4065
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Kamil Szewczyk
>Assignee: Kamil Szewczyk
>Priority: Major
>  Time Spent: 9h 10m
>  Remaining Estimate: 0h
>
> Performance tests are running on Jenkins on regular basis and results are 
> pushed to BigQuery. However there is no automatic regression detection or 
> daily reports with results.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4257) Add error reason and table destination to BigQueryIO streaming failed inserts

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4257?focusedWorklogId=101743&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101743
 ]

ASF GitHub Bot logged work on BEAM-4257:


Author: ASF GitHub Bot
Created on: 14/May/18 14:23
Start Date: 14/May/18 14:23
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5341: [BEAM-4257] 
Increases BigQuery streaming error information
URL: https://github.com/apache/beam/pull/5341#issuecomment-388835105
 
 
   FYI @akedin 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101743)
Time Spent: 20m  (was: 10m)

> Add error reason and table destination to BigQueryIO streaming failed inserts
> -
>
> Key: BEAM-4257
> URL: https://issues.apache.org/jira/browse/BEAM-4257
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Carlos Alonso
>Assignee: Carlos Alonso
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> When using `BigQueryIO.Write` and getting `WriteResult.getFailedInserts()` we 
> get a `PCollection` which is fine, but in order to properly work on 
> the errors downstream having extended information such as the `InsertError` 
> fields and the `TableReference` it was routed to would be really valuable.
>  
> My suggestion is to create a new object that contains all that information 
> and return a `PCollection` of those instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4283) Export nexmark execution times to bigQuery

[Etienne Chauchot (JIRA)]

Etienne Chauchot created BEAM-4283:
--

 Summary: Export nexmark execution times to bigQuery
 Key: BEAM-4283
 URL: https://issues.apache.org/jira/browse/BEAM-4283
 Project: Beam
  Issue Type: Sub-task
  Components: examples-nexmark
Reporter: Etienne Chauchot
Assignee: Etienne Chauchot


Nexmark only outputs the results collection to bigQuery and prints in the 
console the execution times. To supervise Nexmark execution times, we need to 
store them as well per runner/query/mode



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=101746&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101746
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 14/May/18 14:35
Start Date: 14/May/18 14:35
Worklog Time Spent: 10m 
  Work Description: jbonofre commented on issue #1729: [BEAM-1240] Create 
RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#issuecomment-388839507
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101746)
Time Spent: 4.5h  (was: 4h 20m)

> Create RabbitMqIO
> -
>
> Key: BEAM-1240
> URL: https://issues.apache.org/jira/browse/BEAM-1240
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2850) Add Nexmark to Beam Continuous Integration

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2850?focusedWorklogId=101747&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101747
 ]

ASF GitHub Bot logged work on BEAM-2850:


Author: ASF GitHub Bot
Created on: 14/May/18 14:37
Start Date: 14/May/18 14:37
Worklog Time Spent: 10m 
  Work Description: echauchot commented on issue #4976: [BEAM-2850] and 
[BEAM-4225] Add Nexmark PostCommit runs for spark, flink and direct runner and 
export to Bigquery
URL: https://github.com/apache/beam/pull/4976#issuecomment-388840362
 
 
   @lgajowy Nexmark configures the bigQueryIO that way: 
   
   - it creates the table if it does not exist 
   - it appends rows to it. 
   - Actually, while digging into it I saw that only the results PCollection 
was written to bigquery. I think I need to change that because in our case we 
don't care about the results but only about the execution times. I opened a 
ticket (https://issues.apache.org/jira/browse/BEAM-4283) to support writing the 
execution times. I also need to change the command lines in this PR once I have 
the feature available.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101747)
Time Spent: 1h 50m  (was: 1h 40m)

> Add Nexmark to Beam Continuous Integration
> --
>
> Key: BEAM-2850
> URL: https://issues.apache.org/jira/browse/BEAM-2850
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, testing
>Reporter: Ismaël Mejía
>Assignee: Etienne Chauchot
>Priority: Minor
>  Labels: nexmark
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4189) Write sql.Array type to SQL with JdbcIO

[Boris Kjaer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boris Kjaer resolved BEAM-4189.
---
   Resolution: Not A Problem
Fix Version/s: 0.6.0

Okay, getConnection() exposes the connection and createArrayOf() can be called 
on that. 

> Write sql.Array type to SQL with JdbcIO
> ---
>
> Key: BEAM-4189
> URL: https://issues.apache.org/jira/browse/BEAM-4189
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-jdbc
>Reporter: Boris Kjaer
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 0.6.0
>
>
> java.sql.array are normally created with Connection.createArrayOf 
> ([https://docs.oracle.com/javase/tutorial/jdbc/basics/array.html).]
> However, for JdbcIO.Write the connection is private, so it is not possible to 
> serialise an input array, before setting it with PreparedStatement.setArray.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4189) Write sql.Array type to SQL with JdbcIO

[Boris Kjaer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boris Kjaer closed BEAM-4189.
-

Resolved

> Write sql.Array type to SQL with JdbcIO
> ---
>
> Key: BEAM-4189
> URL: https://issues.apache.org/jira/browse/BEAM-4189
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-jdbc
>Reporter: Boris Kjaer
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 0.6.0
>
>
> java.sql.array are normally created with Connection.createArrayOf 
> ([https://docs.oracle.com/javase/tutorial/jdbc/basics/array.html).]
> However, for JdbcIO.Write the connection is private, so it is not possible to 
> serialise an input array, before setting it with PreparedStatement.setArray.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1626

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7a3f6885c69649d0428269bfc4b86ab130de99e8 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7a3f6885c69649d0428269bfc4b86ab130de99e8
Commit message: "Merge pull request #5244 from iemejia/BEAM-3813-s3-sse"
 > git rev-list --no-walk 7a3f6885c69649d0428269bfc4b86ab130de99e8 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins341287752886555062.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ekirpic...@gmail.com

[jira] [Work logged] (BEAM-2850) Add Nexmark to Beam Continuous Integration

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2850?focusedWorklogId=101753&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101753
 ]

ASF GitHub Bot logged work on BEAM-2850:


Author: ASF GitHub Bot
Created on: 14/May/18 15:01
Start Date: 14/May/18 15:01
Worklog Time Spent: 10m 
  Work Description: echauchot commented on issue #4976: [BEAM-2850] and 
[BEAM-4225] Add Nexmark PostCommit runs for spark, flink and direct runner and 
export to Bigquery
URL: https://github.com/apache/beam/pull/4976#issuecomment-388840362
 
 
   @lgajowy Nexmark configures the bigQueryIO that way: 
   
   - it creates the table if it does not exist 
   - it appends rows to it. 
   Actually, while digging into it I saw that only the results PCollection was 
written to bigquery. I think I need to change that because in our case we don't 
care about the results but only about the execution times. I opened a ticket 
(https://issues.apache.org/jira/browse/BEAM-4283) to support writing the 
execution times. I also need to change the command lines in this PR once I have 
the feature available.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101753)
Time Spent: 2h  (was: 1h 50m)

> Add Nexmark to Beam Continuous Integration
> --
>
> Key: BEAM-2850
> URL: https://issues.apache.org/jira/browse/BEAM-2850
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, testing
>Reporter: Ismaël Mejía
>Assignee: Etienne Chauchot
>Priority: Minor
>  Labels: nexmark
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4960

[Apache Jenkins Server]

See 


--
[...truncated 1.01 MB...]
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/

[beam] branch master updated (7a3f688 -> 7beece0)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7a3f688  Merge pull request #5244 from iemejia/BEAM-3813-s3-sse
 add d80cc5a  Fix error-prone warnings for HCatalogIO
 new 7beece0  Merge pull request #5354: Fix error-prone warnings for 
HCatalogIO

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../src/test/java/org/apache/beam/sdk/io/hcatalog/HCatalogIOIT.java | 2 +-
 .../src/test/java/org/apache/beam/sdk/io/hcatalog/HCatalogIOTest.java   | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #5354: Fix error-prone warnings for HCatalogIO

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7beece05ba01b349c88a40f7caa1d0346d69f9e8
Merge: 7a3f688 d80cc5a
Author: Kenn Knowles 
AuthorDate: Mon May 14 09:14:42 2018 -0700

Merge pull request #5354: Fix error-prone warnings for HCatalogIO

 .../src/test/java/org/apache/beam/sdk/io/hcatalog/HCatalogIOIT.java | 2 +-
 .../src/test/java/org/apache/beam/sdk/io/hcatalog/HCatalogIOTest.java   | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1627

[Apache Jenkins Server]

See 


Changes:

[iemejia] Fix error-prone warnings for HCatalogIO

--
Started by GitHub push by kennknowles
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7beece05ba01b349c88a40f7caa1d0346d69f9e8 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7beece05ba01b349c88a40f7caa1d0346d69f9e8
Commit message: "Merge pull request #5354: Fix error-prone warnings for 
HCatalogIO"
 > git rev-list --no-walk 7a3f6885c69649d0428269bfc4b86ab130de99e8 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins2675337492932730609.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ekirpic...@gmail.com

Build failed in Jenkins: beam_PostCommit_Python_Verify #4961

[Apache Jenkins Server]

See 


Changes:

[iemejia] Fix error-prone warnings for HCatalogIO

--
[...truncated 1.01 MB...]
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apac

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #383

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3042) Add tracking of bytes read / time spent when reading side inputs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3042?focusedWorklogId=101773&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101773
 ]

ASF GitHub Bot logged work on BEAM-3042:


Author: ASF GitHub Bot
Created on: 14/May/18 17:17
Start Date: 14/May/18 17:17
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5309: [BEAM-3042] Adding 
time tracking of batch side inputs [low priority]
URL: https://github.com/apache/beam/pull/5309#issuecomment-388894509
 
 
   Gating on a new flag. I've added a commit for that. How does it all look?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101773)
Time Spent: 6h 20m  (was: 6h 10m)

> Add tracking of bytes read / time spent when reading side inputs
> 
>
> Key: BEAM-3042
> URL: https://issues.apache.org/jira/browse/BEAM-3042
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> It is difficult for Dataflow users to understand how modifying a pipeline or 
> data set can affect how much inter-transform IO is used in their job. The 
> intent of this feature request is to help users understand how side inputs 
> behave when they are consumed.
> This will allow users to understand how much time and how much data their 
> pipeline uses to read/write to inter-transform IO. Users will also be able to 
> modify their pipelines and understand how their changes affect these IO 
> metrics.
> For further information, please review the internal Google doc 
> go/insights-transform-io-design-doc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3879) Automate preparation/checking of releases

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3879?focusedWorklogId=101777&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101777
 ]

ASF GitHub Bot logged work on BEAM-3879:


Author: ASF GitHub Bot
Created on: 14/May/18 17:22
Start Date: 14/May/18 17:22
Worklog Time Spent: 10m 
  Work Description: alanmyrvold closed pull request #4896: [BEAM-3879] 
Automate checking of release steps from the release guide
URL: https://github.com/apache/beam/pull/4896
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/release/check_release.sh b/release/check_release.sh
new file mode 100755
index 000..a56b1e1ad6e
--- /dev/null
+++ b/release/check_release.sh
@@ -0,0 +1,91 @@
+#!/bin/bash
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+# This script checks that the release instructions in 
https://beam.apache.org/contribute/release-guide
+# have been followed.
+
+echo 'Check preparation of a new release for Apache Beam.'
+echo ' '
+
+# Load the functions in release_helper.sh.
+. $(dirname "${BASH_SOURCE[0]}")/release_helper.sh 
+
+# Load previous answers, to avoid re-prompting.
+ANSWER_FILE=~/.prepare_beam_release_answers.txt
+load_previous_answers
+
+# Check the needed software is installed.
+check_software gpg
+check_software git
+check_software mvn
+check_software svn
+check_software gpg-agent
+check_software python
+
+check_gpg_key
+check_access_to_nexus
+
+ensure_yes release_proposed "Has a release been proposed to the @dev list" "
+  Deciding to release and selecting a Release Manager is the first step of
+  the release process. This is a consensus-based decision of the entire
+  community. Anybody can propose a release on the dev@ mailing list, giving
+  a solid argument and nominating a committer as the Release Manager
+  (including themselves). There’s no formal process, no vote requirements,
+  and no timing requirements. Any objections should be resolved by consensus
+  before starting the release."
+
+ensure_set beam_version "What version number will be this release"
+check_beam_version_in_jira "${beam_version}" "current"
+beam_version_id=$found_beam_version_id
+
+ensure_yes website_setup "Have you set up access to the beam website?" "
+  You need to prepare access to the beam website to push changes there"
+
+set_next_version
+ensure_yes next_version_looks_ok "Will the next version (after 
${beam_version}) be version ${next_version}" "
+  When contributors resolve an issue in JIRA, they are tagging it with a
+  release that will contain their changes. With the release currently underway,
+  new issues should be resolved against a subsequent future release."
+
+check_beam_version_in_jira "${next_version}" "next"
+
+check_no_unresolved_issues
+
+release_page="https://issues.apache.org/jira/projects/BEAM/versions/${beam_version_id}";
+ensure_yes release_notes_reviewed "Have you reviewed and edited the release 
notes?" "
+  JIRA automatically generates Release Notes based on the Fix Version field 
applied to issues.
+  Release Notes are intended for Beam users (not Beam committers/contributors).
+  You should ensure that Release Notes are informative and useful.
+  The release notes are linked from ${release_page}"
+
+ensure_yes release_build_works "Have you run a release build with mvn 
-Prelease clean install?" "
+  Before creating a release branch, ensure that the release build works and 
javadoc in sdks/java/javadoc
+  looks ok"
+
+check_release_branch_created
+
+check_python_version master "${next_version}.dev"
+check_python_version "release-${beam_version}" "${beam_version}"
+
+check_java_version master beam-master-MMDD
+check_java_version "release-${beam_version}" "beam-${beam_version}"
+
+cleanup
+
+echo ""
+echo "Script complete, but there are more steps at 
https://beam.apache.org/contribute/release-guide";
+echo "To start with a clean state, rm ${ANSWER_FILE} befo

[jira] [Work logged] (BEAM-3813) Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3813?focusedWorklogId=101781&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101781
 ]

ASF GitHub Bot logged work on BEAM-3813:


Author: ASF GitHub Bot
Created on: 14/May/18 17:25
Start Date: 14/May/18 17:25
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5244: 
[BEAM-3813] Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)
URL: https://github.com/apache/beam/pull/5244#discussion_r188033699
 
 

 ##
 File path: 
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/S3Options.java
 ##
 @@ -33,13 +37,44 @@
 
   @Description(
   "Size of S3 upload chunks; max upload object size is this value 
multiplied by 1;"
-  + "default is 64MB, or 5MB in memory-constrained environments")
-  @Nullable
+  + "default is 64MB, or 5MB in memory-constrained environments. Must 
be at least 5MB.")
+  @Default.InstanceFactory(S3UploadBufferSizeBytesFactory.class)
   Integer getS3UploadBufferSizeBytes();
+
   void setS3UploadBufferSizeBytes(Integer value);
 
   @Description("Thread pool size, limiting max concurrent S3 operations")
   @Default.Integer(50)
   int getS3ThreadPoolSize();
   void setS3ThreadPoolSize(int value);
+
+  @Description("Algorithm for SSE-S3 encryption, e.g. AES256.")
+  @Nullable
+  String getSSEAlgorithm();
+  void setSSEAlgorithm(String value);
+
+  @Description("SSE key for SSE-C encryption, e.g. a base64 encoded key and 
the algorithm.")
+  @Nullable
+  SSECustomerKey getSSECustomerKey();
+  void setSSECustomerKey(SSECustomerKey value);
+
+  @Description("KMS key id for SSE-KMS encryption, e.g. \"arn:aws:kms:...\".")
+  @Nullable
+  SSEAwsKeyManagementParams getSSEAwsKeyManagementParams();
 
 Review comment:
   Ditto on SSEAwsKeyManagementParams and Jackson serialization.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101781)
Time Spent: 1h 40m  (was: 1.5h)

> Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)
> ---
>
> Key: BEAM-3813
> URL: https://issues.apache.org/jira/browse/BEAM-3813
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> We should enable AWS S3 users to use encryption when reading or writing by 
> provide encryption keys or using server side encryption via an algorithm, or 
> a key management system (KMS)..
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3813) Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3813?focusedWorklogId=101780&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101780
 ]

ASF GitHub Bot logged work on BEAM-3813:


Author: ASF GitHub Bot
Created on: 14/May/18 17:25
Start Date: 14/May/18 17:25
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5244: 
[BEAM-3813] Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)
URL: https://github.com/apache/beam/pull/5244#discussion_r188033549
 
 

 ##
 File path: 
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/S3Options.java
 ##
 @@ -33,13 +37,44 @@
 
   @Description(
   "Size of S3 upload chunks; max upload object size is this value 
multiplied by 1;"
-  + "default is 64MB, or 5MB in memory-constrained environments")
-  @Nullable
+  + "default is 64MB, or 5MB in memory-constrained environments. Must 
be at least 5MB.")
+  @Default.InstanceFactory(S3UploadBufferSizeBytesFactory.class)
   Integer getS3UploadBufferSizeBytes();
+
   void setS3UploadBufferSizeBytes(Integer value);
 
   @Description("Thread pool size, limiting max concurrent S3 operations")
   @Default.Integer(50)
   int getS3ThreadPoolSize();
   void setS3ThreadPoolSize(int value);
+
+  @Description("Algorithm for SSE-S3 encryption, e.g. AES256.")
+  @Nullable
+  String getSSEAlgorithm();
+  void setSSEAlgorithm(String value);
+
+  @Description("SSE key for SSE-C encryption, e.g. a base64 encoded key and 
the algorithm.")
+  @Nullable
+  SSECustomerKey getSSECustomerKey();
 
 Review comment:
   Since this is a custom object type, how does it get serialized/deserialized 
with Jackson?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101780)
Time Spent: 1.5h  (was: 1h 20m)

> Support encryption for S3FileSystem (SSE-S3, SSE-C and SSE-KMS)
> ---
>
> Key: BEAM-3813
> URL: https://issues.apache.org/jira/browse/BEAM-3813
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> We should enable AWS S3 users to use encryption when reading or writing by 
> provide encryption keys or using server side encryption via an algorithm, or 
> a key management system (KMS)..
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4284) PipelineTranslationTest.testProtoAgainstRehydrated() over-validates translated proto structure

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-4284:
--

 Summary: PipelineTranslationTest.testProtoAgainstRehydrated() 
over-validates translated proto structure
 Key: BEAM-4284
 URL: https://issues.apache.org/jira/browse/BEAM-4284
 Project: Beam
  Issue Type: Bug
  Components: runner-core
Reporter: Scott Wegner
Assignee: Kenneth Knowles


PipelineTranslationTest.testProtoAgainstRehydrated() validates round-tripping 
of a Pipeline object to and form proto, verifying each of the rehydrated 
fields. From conversations with [~lcwik], the validation logic is 
over-specified. There is no requirement that equivalent objects are serialized 
using the same ID. So iterating to find the set of unique IDs and validating 
the count is incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4279) The Fn Harness doesn't properly handle Flattens which consume a single input multiple times

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4279?focusedWorklogId=101800&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101800
 ]

ASF GitHub Bot logged work on BEAM-4279:


Author: ASF GitHub Bot
Created on: 14/May/18 17:55
Start Date: 14/May/18 17:55
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5342: [BEAM-4279] Handle 
Multiple copies of a single input in the Fn Harness
URL: https://github.com/apache/beam/pull/5342#issuecomment-388906262
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101800)
Time Spent: 0.5h  (was: 20m)

> The Fn Harness doesn't properly handle Flattens which consume a single input 
> multiple times
> ---
>
> Key: BEAM-4279
> URL: https://issues.apache.org/jira/browse/BEAM-4279
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Thomas Groh
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> For a {{Flatten}} which consumes a {{PCollection}} n times, each input 
> element should be present {{n}} times in the output {{PCollection}}. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #169

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-3813] Refactor constructors to use only S3Options instead of 
each

[iemejia] [BEAM-3813] Update aws-java-sdk version to version 1.11.319

[iemejia] [BEAM-3813] Add support for SSE-C in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-S3 in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-KMS in S3 Filesystem

[iemejia] [BEAM-3813] Fix error-prone and other warnings for the AWS module

[iemejia] Fix error-prone warnings for HCatalogIO

--
[...truncated 55.98 KB...]
Warning: Bad option "-grammar_encoding=UTF-8" will be ignored.
Reading from file 

 . . .
Note: UNICODE_INPUT option is specified. Please make sure you create the 
parser/lexer using a Reader with the correct character encoding.
Warning: Lookahead adequacy checking not being performed since option LOOKAHEAD 
is more than 1.  Set option FORCE_LA_CHECK to true to force checking.
File "TokenMgrError.java" does not exist.  Will create one.
File "ParseException.java" does not exist.  Will create one.
File "Token.java" does not exist.  Will create one.
File "SimpleCharStream.java" does not exist.  Will create one.
Parser generated with 0 errors and 1 warnings.

> Task :beam-model-job-management:extractIncludeProto
> Task :beam-model-fn-execution:extractIncludeProto
> Task :beam-model-job-management:generateProto
> Task :beam-model-pipeline:jar
> Task :beam-model-fn-execution:generateProto
> Task :beam-model-pipeline:extractIncludeTestProto
> Task :beam-model-pipeline:extractTestProto
> Task :beam-model-pipeline:generateTestProto NO-SOURCE
> Task :beam-model-pipeline:compileTestJava NO-SOURCE
> Task :beam-model-pipeline:processTestResources NO-SOURCE
> Task :beam-model-pipeline:testClasses UP-TO-DATE
> Task :beam-model-pipeline:packageTests
> Task :beam-model-pipeline:install
> Task :beam-model-fn-execution:compileJava FROM-CACHE
> Task :beam-model-fn-execution:classes
> Task :beam-model-job-management:compileJava FROM-CACHE
> Task :beam-model-job-management:classes
> Task :beam-sdks-python:setupVirtualenv FAILED
> Task :beam-model-job-management:shadowJar
> Task :beam-model-fn-execution:shadowJar
> Task :beam-sdks-java-core:shadowJar

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 5.0.
See 
https://docs.gradle.org/4.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 1m 18s
131 actionable tasks: 124 executed, 5 from cache, 2 up-to-date

Publishing build scan...
https://gradle.com/s/aniorbj6aaq6i


STDERR: 
FAILURE: Build failed with an exception.

* Where:
Build file 
'
 line: 36

* What went wrong:
Execution failed for task ':beam-sdks-python:setupVirtualenv'.
> Process 'command 'sh'' finished with non-zero exit value 1

* Try:
Run with --info or --debug option to get more log output. Run with --scan to 
get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task 
':beam-sdks-python:setupVirtualenv'.
at 
org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:103)
at 
org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:73)
at 
org.gradle.api.internal.tasks.execution.OutputDirectoryCreatingTaskExecuter.execute(OutputDirectoryCreatingTaskExecuter.java:51)
at 
org.gradle.api.internal.tasks.execution.SkipCachedTaskExecuter.execute(SkipCachedTaskExecuter.java:105)
at 
org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:59)
at 
org.gradle.api.internal.tasks.execution.ResolveTaskOutputCachingStateExecuter.execute(ResolveTaskOutputCachingStateExecuter.java:54)
at 
org.gradle.api.internal.tasks.execution.ResolveBuildCacheKeyExecuter.execute(ResolveBuildCacheKeyExecuter.java:66)
at 
org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:59)
at 
org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:101)
at 
org.gradle.api.internal.tasks.execution.FinalizeInputFilePropertiesTaskExecuter.execute(FinalizeInputFilePropertiesTaskExecuter.java:44)
at 
org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:91)
at 
org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:62)
at 
o

Build failed in Jenkins: beam_PerformanceTests_JDBC #572

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-3813] Refactor constructors to use only S3Options instead of 
each

[iemejia] [BEAM-3813] Update aws-java-sdk version to version 1.11.319

[iemejia] [BEAM-3813] Add support for SSE-C in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-S3 in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-KMS in S3 Filesystem

[iemejia] [BEAM-3813] Fix error-prone and other warnings for the AWS module

[iemejia] Fix error-prone warnings for HCatalogIO

--
[...truncated 97.07 KB...]
SLF4J: Found binding in 
[jar:
SLF4J: Found binding in 
[jar:file:/home/jenkins/.m2/repository/org/slf4j/slf4j-jdk14/1.7.25/slf4j-jdk14-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory]

Gradle Test Executor 1 finished executing tests.

> Task :beam-sdks-java-io-jdbc:integrationTest

org.apache.beam.sdk.io.jdbc.JdbcIOIT > classMethod FAILED
org.postgresql.util.PSQLException: The connection attempt failed.
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:272)
at 
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:215)
at org.postgresql.Driver.makeConnection(Driver.java:404)
at org.postgresql.Driver.connect(Driver.java:272)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:86)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:71)
at 
org.apache.beam.sdk.io.common.DatabaseTestHelper.createTable(DatabaseTestHelper.java:46)
at org.apache.beam.sdk.io.jdbc.JdbcIOIT.setup(JdbcIOIT.java:85)

Caused by:
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.(PGStream.java:61)
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144)
... 10 more

org.apache.beam.sdk.io.jdbc.JdbcIOIT > classMethod FAILED
org.postgresql.util.PSQLException: The connection attempt failed.
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:272)
at 
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:215)
at org.postgresql.Driver.makeConnection(Driver.java:404)
at org.postgresql.Driver.connect(Driver.java:272)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:86)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:71)
at 
org.apache.beam.sdk.io.common.DatabaseTestHelper.deleteTable(DatabaseTestHelper.java:57)
at org.apache.beam.sdk.io.jdbc.JdbcIOIT.tearDown(JdbcIOIT.java:90)

Caused by:
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.(PGStream.java:61)
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:144)
... 10 more

2 tests completed, 2 failed
Finished generating test XML results (0.013 secs) into: 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT_HDFS #175

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT_HDFS #169

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-4285) Flink batch state request handler

[Ben Sidhom (JIRA)]

Ben Sidhom created BEAM-4285:


 Summary: Flink batch state request handler
 Key: BEAM-4285
 URL: https://issues.apache.org/jira/browse/BEAM-4285
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Reporter: Ben Sidhom


In order to support side inputs Flink needs a state service request handler. As 
in the non-portable we can start by handling batch side inputs by Flink 
broadcast variables.

[https://github.com/bsidhom/beam/blob/41de3bce60f1ebc9211f299612a20d8e561f9b6f/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkBatchStateRequestHandler.java]
 or 
[https://github.com/bsidhom/beam/blob/41de3bce60f1ebc9211f299612a20d8e561f9b6f/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkBatchStateRequestHandler.java]
 can be used as a starting point. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #168

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-3813] Refactor constructors to use only S3Options instead of 
each

[iemejia] [BEAM-3813] Update aws-java-sdk version to version 1.11.319

[iemejia] [BEAM-3813] Add support for SSE-C in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-S3 in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-KMS in S3 Filesystem

[iemejia] [BEAM-3813] Fix error-prone and other warnings for the AWS module

[iemejia] Fix error-prone warnings for HCatalogIO

--
[...truncated 605.56 KB...]
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:129)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException):
 Cannot create 
file/TEXTIO_IT__1526321041492/.temp-beam-2018-05-14_18-04-01-0/6e4cd141-ad68-4728-9d0d-96fb99db85e4.
 Name node is in safe mode.
The reported blocks 0 needs additional 31 blocks to reach the threshold 
0.9990 of total blocks 31.
The number of live datanodes 0 has reached the minimum number 0. Safe mode 
will be turned off automatically once the thresholds have been reached.
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1327)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2447)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2335)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:623)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.

[jira] [Commented] (BEAM-2898) Flink supports chaining/fusion of single-SDK stages

[Ben Sidhom (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474569#comment-16474569
 ] 

Ben Sidhom commented on BEAM-2898:
--

I'm going to mark this as fixed since we have fusion and executable stage 
representation. Fusion is exercised by the portable translator.

> Flink supports chaining/fusion of single-SDK stages
> ---
>
> Key: BEAM-2898
> URL: https://issues.apache.org/jira/browse/BEAM-2898
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Ben Sidhom
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> The Fn API supports fused stages, which avoids unnecessarily round-tripping 
> the data over the Fn API between stages. The Flink runner should use that 
> capability for better performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2898) Flink supports chaining/fusion of single-SDK stages

[Ben Sidhom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Sidhom closed BEAM-2898.

   Resolution: Fixed
Fix Version/s: Not applicable

> Flink supports chaining/fusion of single-SDK stages
> ---
>
> Key: BEAM-2898
> URL: https://issues.apache.org/jira/browse/BEAM-2898
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Ben Sidhom
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> The Fn API supports fused stages, which avoids unnecessarily round-tripping 
> the data over the Fn API between stages. The Flink runner should use that 
> capability for better performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=101811&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101811
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 14/May/18 18:16
Start Date: 14/May/18 18:16
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5348: 
[BEAM-3326] Add a Direct Job Bundle Factory
URL: https://github.com/apache/beam/pull/5348#discussion_r188046059
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectJobBundleFactory.java
 ##
 @@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
+import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import org.apache.beam.runners.fnexecution.control.RemoteBundle;
+import org.apache.beam.runners.fnexecution.control.RemoteOutputReceiver;
+import org.apache.beam.runners.fnexecution.control.SdkHarnessClient;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.ActiveBundle;
+import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.sdk.util.WindowedValue;
+
+/** A {@link JobBundleFactory} for the ReferenceRunner. */
+class DirectJobBundleFactory implements JobBundleFactory {
+  private final EnvironmentFactory environmentFactory;
+
+  private final GrpcFnServer dataService;
+  private final GrpcFnServer stateService;
+
+  private final ConcurrentMap> 
stageBundleFactories =
+  new ConcurrentHashMap<>();
+  private final ConcurrentMap environments =
+  new ConcurrentHashMap<>();
+
+  DirectJobBundleFactory(
+  EnvironmentFactory environmentFactory,
+  GrpcFnServer dataService,
+  GrpcFnServer stateService) {
+this.environmentFactory = environmentFactory;
+this.dataService = dataService;
+this.stateService = stateService;
+  }
+
+  @Override
+  public  StageBundleFactory forStage(ExecutableStage executableStage) {
+return (StageBundleFactory)
+stageBundleFactories.computeIfAbsent(executableStage, 
this::createBundleFactory);
+  }
+
+  private final AtomicLong idgen = new AtomicLong();
+
+  private  StageBundleFactory createBundleFactory(ExecutableStage stage) 
{
+RemoteEnvironment remoteEnv =
+environments.computeIfAbsent(
+stage.getEnvironment(),
+env -> {
+  try {
+return environmentFactory.createEnvironment(env);
+  } catch (Exception e) {
+throw new RuntimeException(e);
+  }
+});
+SdkHarnessClient sdkHarnessClient =
+SdkHarnessClient.usingFnApiClient(
+remoteEnv.getIns

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=101812&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101812
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 14/May/18 18:16
Start Date: 14/May/18 18:16
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5348: 
[BEAM-3326] Add a Direct Job Bundle Factory
URL: https://github.com/apache/beam/pull/5348#discussion_r188044479
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectJobBundleFactory.java
 ##
 @@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
+import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import org.apache.beam.runners.fnexecution.control.RemoteBundle;
+import org.apache.beam.runners.fnexecution.control.RemoteOutputReceiver;
+import org.apache.beam.runners.fnexecution.control.SdkHarnessClient;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.ActiveBundle;
+import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.sdk.util.WindowedValue;
+
+/** A {@link JobBundleFactory} for the ReferenceRunner. */
+class DirectJobBundleFactory implements JobBundleFactory {
+  private final EnvironmentFactory environmentFactory;
+
+  private final GrpcFnServer dataService;
+  private final GrpcFnServer stateService;
+
+  private final ConcurrentMap> 
stageBundleFactories =
+  new ConcurrentHashMap<>();
+  private final ConcurrentMap environments =
+  new ConcurrentHashMap<>();
+
+  DirectJobBundleFactory(
+  EnvironmentFactory environmentFactory,
+  GrpcFnServer dataService,
+  GrpcFnServer stateService) {
+this.environmentFactory = environmentFactory;
+this.dataService = dataService;
+this.stateService = stateService;
+  }
+
+  @Override
+  public  StageBundleFactory forStage(ExecutableStage executableStage) {
+return (StageBundleFactory)
+stageBundleFactories.computeIfAbsent(executableStage, 
this::createBundleFactory);
+  }
+
+  private final AtomicLong idgen = new AtomicLong();
 
 Review comment:
   Use 
https://github.com/apache/beam/blob/master/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/IdGenerator.java


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 1018

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=101810&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101810
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 14/May/18 18:16
Start Date: 14/May/18 18:16
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5348: 
[BEAM-3326] Add a Direct Job Bundle Factory
URL: https://github.com/apache/beam/pull/5348#discussion_r188048988
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectJobBundleFactory.java
 ##
 @@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
+import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import org.apache.beam.runners.fnexecution.control.RemoteBundle;
+import org.apache.beam.runners.fnexecution.control.RemoteOutputReceiver;
+import org.apache.beam.runners.fnexecution.control.SdkHarnessClient;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.ActiveBundle;
+import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.sdk.util.WindowedValue;
+
+/** A {@link JobBundleFactory} for the ReferenceRunner. */
+class DirectJobBundleFactory implements JobBundleFactory {
 
 Review comment:
   Are you planning to have NeedsRunner/ValidatesRunner cover this and the 
StageBundleFactory testing?
   
   If so, it might be worthwhile to add error handling tests since 
NeedsRunner/ValidatesRunner typically don't handle edge cases.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101810)
Time Spent: 8h 40m  (was: 8.5h)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=101813&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101813
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 14/May/18 18:17
Start Date: 14/May/18 18:17
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5348: 
[BEAM-3326] Add a Direct Job Bundle Factory
URL: https://github.com/apache/beam/pull/5348#discussion_r188044479
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectJobBundleFactory.java
 ##
 @@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
+import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import org.apache.beam.runners.fnexecution.control.RemoteBundle;
+import org.apache.beam.runners.fnexecution.control.RemoteOutputReceiver;
+import org.apache.beam.runners.fnexecution.control.SdkHarnessClient;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.ActiveBundle;
+import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.sdk.util.WindowedValue;
+
+/** A {@link JobBundleFactory} for the ReferenceRunner. */
+class DirectJobBundleFactory implements JobBundleFactory {
+  private final EnvironmentFactory environmentFactory;
+
+  private final GrpcFnServer dataService;
+  private final GrpcFnServer stateService;
+
+  private final ConcurrentMap> 
stageBundleFactories =
+  new ConcurrentHashMap<>();
+  private final ConcurrentMap environments =
+  new ConcurrentHashMap<>();
+
+  DirectJobBundleFactory(
+  EnvironmentFactory environmentFactory,
+  GrpcFnServer dataService,
+  GrpcFnServer stateService) {
+this.environmentFactory = environmentFactory;
+this.dataService = dataService;
+this.stateService = stateService;
+  }
+
+  @Override
+  public  StageBundleFactory forStage(ExecutableStage executableStage) {
+return (StageBundleFactory)
+stageBundleFactories.computeIfAbsent(executableStage, 
this::createBundleFactory);
+  }
+
+  private final AtomicLong idgen = new AtomicLong();
 
 Review comment:
   Use 
https://github.com/apache/beam/blob/master/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/IdGenerator.java
   
   Move it to a shared location if needed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking

[jira] [Work logged] (BEAM-4269) Implement Assign Windows in the Java SDK Harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4269?focusedWorklogId=101814&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101814
 ]

ASF GitHub Bot logged work on BEAM-4269:


Author: ASF GitHub Bot
Created on: 14/May/18 18:19
Start Date: 14/May/18 18:19
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5329: [BEAM-4269, 
BEAM-3970] Implement AssignWindows in the Java SDK harness
URL: https://github.com/apache/beam/pull/5329#issuecomment-388913849
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101814)
Time Spent: 2h 40m  (was: 2.5h)

> Implement Assign Windows in the Java SDK Harness
> 
>
> Key: BEAM-4269
> URL: https://issues.apache.org/jira/browse/BEAM-4269
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> This allows execution of Java WindowFns over the Fn API



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_MongoDBIO_IT #170

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4269) Implement Assign Windows in the Java SDK Harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4269?focusedWorklogId=101816&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101816
 ]

ASF GitHub Bot logged work on BEAM-4269:


Author: ASF GitHub Bot
Created on: 14/May/18 18:22
Start Date: 14/May/18 18:22
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5329: 
[BEAM-4269, BEAM-3970] Implement AssignWindows in the Java SDK harness
URL: https://github.com/apache/beam/pull/5329#discussion_r188050919
 
 

 ##
 File path: 
sdks/java/harness/src/test/java/org/apache/beam/fn/harness/AssignWindowsRunnerTest.java
 ##
 @@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.fn.harness;
+
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.ListMultimap;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import 
org.apache.beam.fn.harness.AssignWindowsRunner.AssignWindowsMapFnFactory;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
+import org.apache.beam.model.pipeline.v1.RunnerApi.WindowIntoPayload;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.SdkComponents;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.fn.function.ThrowingFunction;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import 
org.apache.beam.sdk.transforms.windowing.IntervalWindow.IntervalWindowCoder;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.PartitioningWindowFn;
+import org.apache.beam.sdk.transforms.windowing.Sessions;
+import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.transforms.windowing.WindowMappingFn;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link org.apache.beam.fn.harness.AssignWindowsRunner}. */
+@RunWith(JUnit4.class)
+public class AssignWindowsRunnerTest implements Serializable {
+  @Rule public transient ExpectedException thrown = ExpectedException.none();
+
+  @Test
+  public void singleInputSingleOutputSucceeds() throws Exception {
+FixedWindows windowFn = FixedWindows.of(Duration.standardMinutes(10L));
+
+AssignWindowsRunner runner = 
AssignWindowsRunner.create(windowFn);
+
+assertThat(
+runner.assignWindows(WindowedValue.valueInGlobalWindow(1)),
+equalTo(
+WindowedValue.of(
+1,
+BoundedWindow.TIMESTAMP_MIN_VALUE,
+windowFn.assignWindow(BoundedWindow.TIMESTAMP_MIN_VALUE),
+PaneInfo.NO_FIRING)));
+assertThat(
+runner.assignWindows(
+WindowedValue.of(
+2,
+new Instant(-10L),
+new IntervalWindow(new Instant(-12L), 
Duration.standardMinutes(3L)),
+PaneInfo.ON_TIME_AND_ONLY_FIRING)),
+equalTo(
+WindowedValue.of(
+2,
+new Instant(-10L),
+windowFn.assig

[jira] [Work logged] (BEAM-4269) Implement Assign Windows in the Java SDK Harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4269?focusedWorklogId=101823&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101823
 ]

ASF GitHub Bot logged work on BEAM-4269:


Author: ASF GitHub Bot
Created on: 14/May/18 18:26
Start Date: 14/May/18 18:26
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5329: 
[BEAM-4269, BEAM-3970] Implement AssignWindows in the Java SDK harness
URL: https://github.com/apache/beam/pull/5329#discussion_r188052147
 
 

 ##
 File path: 
sdks/java/harness/src/test/java/org/apache/beam/fn/harness/AssignWindowsRunnerTest.java
 ##
 @@ -0,0 +1,363 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.fn.harness;
+
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.ListMultimap;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import 
org.apache.beam.fn.harness.AssignWindowsRunner.AssignWindowsMapFnFactory;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
+import org.apache.beam.model.pipeline.v1.RunnerApi.WindowIntoPayload;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.SdkComponents;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.fn.function.ThrowingFunction;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import 
org.apache.beam.sdk.transforms.windowing.IntervalWindow.IntervalWindowCoder;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.transforms.windowing.PartitioningWindowFn;
+import org.apache.beam.sdk.transforms.windowing.Sessions;
+import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.transforms.windowing.WindowMappingFn;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Tests for {@link org.apache.beam.fn.harness.AssignWindowsRunner}. */
+@RunWith(JUnit4.class)
+public class AssignWindowsRunnerTest implements Serializable {
+  @Rule public transient ExpectedException thrown = ExpectedException.none();
+
+  @Test
+  public void singleInputSingleOutputSucceeds() throws Exception {
+FixedWindows windowFn = FixedWindows.of(Duration.standardMinutes(10L));
+
+AssignWindowsRunner runner = 
AssignWindowsRunner.create(windowFn);
+
+assertThat(
+runner.assignWindows(WindowedValue.valueInGlobalWindow(1)),
+equalTo(
+WindowedValue.of(
+1,
+BoundedWindow.TIMESTAMP_MIN_VALUE,
+windowFn.assignWindow(BoundedWindow.TIMESTAMP_MIN_VALUE),
+PaneInfo.NO_FIRING)));
+assertThat(
+runner.assignWindows(
+WindowedValue.of(
+2,
+new Instant(-10L),
+new IntervalWindow(new Instant(-12L), 
Duration.standardMinutes(3L)),
+PaneInfo.ON_TIME_AND_ONLY_FIRING)),
+equalTo(
+WindowedValue.of(
+2,
+new Instant(-10L),
+windowFn.assignWi

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1628

[Apache Jenkins Server]

See 


--
GitHub pull request #5356 of commit 39a8a62a269d5e71b5b3f9905d605b603aa45503, 
no merge conflicts.
Setting status of 39a8a62a269d5e71b5b3f9905d605b603aa45503 to PENDING with url 
https://builds.apache.org/job/beam_PostCommit_Python_ValidatesRunner_Dataflow/1628/
 and message: 'Build started sha1 is merged.'
Using context: Jenkins: Google Cloud Dataflow Runner Python ValidatesRunner 
Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5356/*:refs/remotes/origin/pr/5356/*
 > git rev-parse refs/remotes/origin/pr/5356/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5356/merge^{commit} # timeout=10
Checking out Revision 7b1f2fc22146fad8aa1866e9ed0d787b2e8bd25e 
(refs/remotes/origin/pr/5356/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7b1f2fc22146fad8aa1866e9ed0d787b2e8bd25e
Commit message: "Merge 39a8a62a269d5e71b5b3f9905d605b603aa45503 into 
7beece05ba01b349c88a40f7caa1d0346d69f9e8"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins7302823271699665837.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ekirpic...@gmail.com

[jira] [Work logged] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4204?focusedWorklogId=101835&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101835
 ]

ASF GitHub Bot logged work on BEAM-4204:


Author: ASF GitHub Bot
Created on: 14/May/18 18:49
Start Date: 14/May/18 18:49
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #5301: [BEAM-4204] 
Splitting ULR in portable runner stub and job service
URL: https://github.com/apache/beam/pull/5301#issuecomment-388923206
 
 
   @robertwb This PR changes ULR a lot. Can you please take a look to see if it 
makes sense.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101835)
Time Spent: 2h 10m  (was: 2h)

> Python: PortableRunner - p.run() via given JobService
> -
>
> Key: BEAM-4204
> URL: https://issues.apache.org/jira/browse/BEAM-4204
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Like BEAM-4071 but for Python. Is this fully encompassed by 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/universal_local_runner.py]
>  ? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4277) Adding unshaded jars to poms for Dataflow.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4277?focusedWorklogId=101834&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101834
 ]

ASF GitHub Bot logged work on BEAM-4277:


Author: ASF GitHub Bot
Created on: 14/May/18 18:49
Start Date: 14/May/18 18:49
Worklog Time Spent: 10m 
  Work Description: youngoli commented on a change in pull request #5340: 
[BEAM-4277] Adding unshaded jars used by Dataflow to poms.
URL: https://github.com/apache/beam/pull/5340#discussion_r188059143
 
 

 ##
 File path: runners/direct-java/pom.xml
 ##
 @@ -125,6 +125,23 @@
 
   
 
+
 
 Review comment:
   I believe that it's used by internal tests, yes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101834)
Time Spent: 40m  (was: 0.5h)

> Adding unshaded jars to poms for Dataflow.
> --
>
> Key: BEAM-4277
> URL: https://issues.apache.org/jira/browse/BEAM-4277
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Dataflow needs some unshaded jars to be generated. Gradle already supports 
> this but the pom files are still in use in some places as of filing this bug, 
> and need to be updated.
> The relevant poms are:
> runners/core-construction-java/pom.xml
> runners/direct-java/pom.xml
> runners/local-java/pom.xml
> sdks/java/extensions/protobuf/pom.xml



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1629

[Apache Jenkins Server]

See 


--
GitHub pull request #5251 of commit 5061362ff0473be448ab872c9c279aa498f93b5a, 
no merge conflicts.
Setting status of 5061362ff0473be448ab872c9c279aa498f93b5a to PENDING with url 
https://builds.apache.org/job/beam_PostCommit_Python_ValidatesRunner_Dataflow/1629/
 and message: 'Build started sha1 is merged.'
Using context: Jenkins: Google Cloud Dataflow Runner Python ValidatesRunner 
Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5251/*:refs/remotes/origin/pr/5251/*
 > git rev-parse refs/remotes/origin/pr/5251/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5251/merge^{commit} # timeout=10
Checking out Revision aaffdfeca468802f0d6727475640b0c4378183ab 
(refs/remotes/origin/pr/5251/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f aaffdfeca468802f0d6727475640b0c4378183ab
Commit message: "Merge 5061362ff0473be448ab872c9c279aa498f93b5a into 
7beece05ba01b349c88a40f7caa1d0346d69f9e8"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins8062028371775682428.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ekirpic...@gmail.com

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=101843&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101843
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 14/May/18 19:19
Start Date: 14/May/18 19:19
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #5251: [BEAM-3883] Refactor 
and clean dependency.py to make it reusable with artifact service
URL: https://github.com/apache/beam/pull/5251#issuecomment-388931882
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101843)
Time Spent: 7h 40m  (was: 7.5h)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4962

[Apache Jenkins Server]

See 


--
[...truncated 1.14 MB...]
  "component_encodings": [
{
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0).output"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s17"
}, 
"user_name": 
"write/Write/WriteImpl/FinalizeWrite/_UnpickledSideInput(PreFinalize.out.0)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s21", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_finalize_write"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {
  "side0": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "SideInput-s18"
  }, 
  "side1": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "SideInput-s19"
  }, 
  "side2": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "SideInput-s20"
  }
}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/Write/WriteImpl/FinalizeWrite.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s7"
}, 
"serialized_fn": "", 
"user_name": "write/Write/WriteImpl/FinalizeWrite/FinalizeWrite"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2018-05-14_12_06_39-1090332721983279184]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-14_12_06_39-1090332721983279184?project=apache-beam-testing
root: INFO: Job 2018-05-14_12_

[jira] [Work logged] (BEAM-4236) Side inputs do not work on Dataflow

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4236?focusedWorklogId=101856&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101856
 ]

ASF GitHub Bot logged work on BEAM-4236:


Author: ASF GitHub Bot
Created on: 14/May/18 19:40
Start Date: 14/May/18 19:40
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5302: [BEAM-4236, 
BEAM-2927] Make Python SDK side inputs work with non well known coders and also 
work with Dataflow
URL: https://github.com/apache/beam/pull/5302#issuecomment-388937819
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101856)
Time Spent: 5.5h  (was: 5h 20m)

> Side inputs do not work on Dataflow
> ---
>
> Key: BEAM-4236
> URL: https://issues.apache.org/jira/browse/BEAM-4236
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Affects Versions: Not applicable
>Reporter: Charles Chen
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> Dataflow side inputs do not work in streaming / batch FnApi mode at head.  
> The pipeline stalls indefinitely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4236) Side inputs do not work on Dataflow

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4236?focusedWorklogId=101859&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101859
 ]

ASF GitHub Bot logged work on BEAM-4236:


Author: ASF GitHub Bot
Created on: 14/May/18 19:40
Start Date: 14/May/18 19:40
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5302: [BEAM-4236, 
BEAM-2927] Make Python SDK side inputs work with non well known coders and also 
work with Dataflow
URL: https://github.com/apache/beam/pull/5302#issuecomment-388076213
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101859)
Time Spent: 6h  (was: 5h 50m)

> Side inputs do not work on Dataflow
> ---
>
> Key: BEAM-4236
> URL: https://issues.apache.org/jira/browse/BEAM-4236
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Affects Versions: Not applicable
>Reporter: Charles Chen
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> Dataflow side inputs do not work in streaming / batch FnApi mode at head.  
> The pipeline stalls indefinitely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1630

[Apache Jenkins Server]

See 


--
GitHub pull request #5302 of commit 41b1b1c49ff338fc1019eb86013daaa7c455bbeb, 
no merge conflicts.
Setting status of 41b1b1c49ff338fc1019eb86013daaa7c455bbeb to PENDING with url 
https://builds.apache.org/job/beam_PostCommit_Python_ValidatesRunner_Dataflow/1630/
 and message: 'Build started sha1 is merged.'
Using context: Jenkins: Google Cloud Dataflow Runner Python ValidatesRunner 
Tests
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/5302/*:refs/remotes/origin/pr/5302/*
 > git rev-parse refs/remotes/origin/pr/5302/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/5302/merge^{commit} # timeout=10
Checking out Revision 179430a1292b3bf1c065c42f96f1adbfe6c1e593 
(refs/remotes/origin/pr/5302/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 179430a1292b3bf1c065c42f96f1adbfe6c1e593
Commit message: "Merge 41b1b1c49ff338fc1019eb86013daaa7c455bbeb into 
7beece05ba01b349c88a40f7caa1d0346d69f9e8"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins4912940448936535297.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ekirpic...@gmail.com

[jira] [Work logged] (BEAM-4236) Side inputs do not work on Dataflow

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4236?focusedWorklogId=101855&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101855
 ]

ASF GitHub Bot logged work on BEAM-4236:


Author: ASF GitHub Bot
Created on: 14/May/18 19:40
Start Date: 14/May/18 19:40
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5302: [BEAM-4236, 
BEAM-2927] Make Python SDK side inputs work with non well known coders and also 
work with Dataflow
URL: https://github.com/apache/beam/pull/5302#issuecomment-388937792
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101855)
Time Spent: 5h 20m  (was: 5h 10m)

> Side inputs do not work on Dataflow
> ---
>
> Key: BEAM-4236
> URL: https://issues.apache.org/jira/browse/BEAM-4236
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Affects Versions: Not applicable
>Reporter: Charles Chen
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> Dataflow side inputs do not work in streaming / batch FnApi mode at head.  
> The pipeline stalls indefinitely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4236) Side inputs do not work on Dataflow

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4236?focusedWorklogId=101857&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101857
 ]

ASF GitHub Bot logged work on BEAM-4236:


Author: ASF GitHub Bot
Created on: 14/May/18 19:40
Start Date: 14/May/18 19:40
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5302: [BEAM-4236, 
BEAM-2927] Make Python SDK side inputs work with non well known coders and also 
work with Dataflow
URL: https://github.com/apache/beam/pull/5302#issuecomment-388080435
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101857)
Time Spent: 5h 40m  (was: 5.5h)

> Side inputs do not work on Dataflow
> ---
>
> Key: BEAM-4236
> URL: https://issues.apache.org/jira/browse/BEAM-4236
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Affects Versions: Not applicable
>Reporter: Charles Chen
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> Dataflow side inputs do not work in streaming / batch FnApi mode at head.  
> The pipeline stalls indefinitely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4236) Side inputs do not work on Dataflow

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4236?focusedWorklogId=101858&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101858
 ]

ASF GitHub Bot logged work on BEAM-4236:


Author: ASF GitHub Bot
Created on: 14/May/18 19:40
Start Date: 14/May/18 19:40
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5302: [BEAM-4236, 
BEAM-2927] Make Python SDK side inputs work with non well known coders and also 
work with Dataflow
URL: https://github.com/apache/beam/pull/5302#issuecomment-388134438
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101858)
Time Spent: 5h 50m  (was: 5h 40m)

> Side inputs do not work on Dataflow
> ---
>
> Key: BEAM-4236
> URL: https://issues.apache.org/jira/browse/BEAM-4236
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Affects Versions: Not applicable
>Reporter: Charles Chen
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> Dataflow side inputs do not work in streaming / batch FnApi mode at head.  
> The pipeline stalls indefinitely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4279) The Fn Harness doesn't properly handle Flattens which consume a single input multiple times

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4279?focusedWorklogId=101860&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101860
 ]

ASF GitHub Bot logged work on BEAM-4279:


Author: ASF GitHub Bot
Created on: 14/May/18 19:41
Start Date: 14/May/18 19:41
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5342: [BEAM-4279] Handle 
Multiple copies of a single input in the Fn Harness
URL: https://github.com/apache/beam/pull/5342
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
index 6551570433a..13f58519e0c 100644
--- 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
+++ 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/ProcessBundleHandler.java
@@ -19,10 +19,13 @@
 package org.apache.beam.fn.harness.control;
 
 import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ArrayListMultimap;
 import com.google.common.collect.HashMultimap;
 import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ListMultimap;
 import com.google.common.collect.Lists;
 import com.google.common.collect.Multimap;
+import com.google.common.collect.SetMultimap;
 import com.google.common.collect.Sets;
 import com.google.protobuf.Message;
 import com.google.protobuf.TextFormat;
@@ -43,6 +46,7 @@
 import org.apache.beam.fn.harness.state.BeamFnStateClient;
 import org.apache.beam.fn.harness.state.BeamFnStateGrpcClientCache;
 import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleDescriptor;
 import org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleRequest;
 import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
 import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest.Builder;
@@ -51,6 +55,8 @@
 import org.apache.beam.model.pipeline.v1.RunnerApi;
 import org.apache.beam.model.pipeline.v1.RunnerApi.Coder;
 import org.apache.beam.model.pipeline.v1.RunnerApi.PCollection;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
+import org.apache.beam.model.pipeline.v1.RunnerApi.WindowingStrategy;
 import org.apache.beam.runners.core.construction.PTransformTranslation;
 import org.apache.beam.sdk.fn.data.FnDataReceiver;
 import org.apache.beam.sdk.fn.function.ThrowingRunnable;
@@ -132,13 +138,14 @@ public ProcessBundleHandler(
   private void createRunnerAndConsumersForPTransformRecursively(
   BeamFnStateClient beamFnStateClient,
   String pTransformId,
-  RunnerApi.PTransform pTransform,
+  PTransform pTransform,
   Supplier processBundleInstructionId,
-  BeamFnApi.ProcessBundleDescriptor processBundleDescriptor,
-  Multimap pCollectionIdsToConsumingPTransforms,
-  Multimap>> 
pCollectionIdsToConsumers,
+  ProcessBundleDescriptor processBundleDescriptor,
+  SetMultimap pCollectionIdsToConsumingPTransforms,
+  ListMultimap>> 
pCollectionIdsToConsumers,
   Consumer addStartFunction,
-  Consumer addFinishFunction) throws IOException {
+  Consumer addFinishFunction)
+  throws IOException {
 
 // Recursively ensure that all consumers of the output PCollection have 
been created.
 // Since we are creating the consumers first, we know that the we are 
building the DAG
@@ -202,10 +209,9 @@ private void 
createRunnerAndConsumersForPTransformRecursively(
 BeamFnApi.ProcessBundleDescriptor bundleDescriptor =
 (BeamFnApi.ProcessBundleDescriptor) fnApiRegistry.apply(bundleId);
 
-Multimap pCollectionIdsToConsumingPTransforms = 
HashMultimap.create();
-Multimap>> pCollectionIdsToConsumers =
-HashMultimap.create();
+SetMultimap pCollectionIdsToConsumingPTransforms = 
HashMultimap.create();
+ListMultimap>> 
pCollectionIdsToConsumers =
+ArrayListMultimap.create();
 List startFunctions = new ArrayList<>();
 List finishFunctions = new ArrayList<>();
 
@@ -339,11 +345,11 @@ public Object createRunnerForPTransform(
 BeamFnDataClient beamFnDataClient,
 BeamFnStateClient beamFnStateClient,
 String pTransformId,
-RunnerApi.PTransform pTransform,
+PTransform pTransform,
 Supplier processBundleInstructionId,
 Map pCollections,
 Map coders,
-Map windowingStrategies,
+Map windowingStrategies,
 Multimap>> 
pCollectionIdsToConsumers,
 Consumer addStartFunction,
  

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1631

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add a FlattenRunnerTest for a multi-consumed input

[tgroh] Use ListMultimap when creating Transform Runners

--
Started by GitHub push by lukecwik
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision f19a7172a4bcee30101ed7012055f5b5460e9a3a (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f19a7172a4bcee30101ed7012055f5b5460e9a3a
Commit message: "[BEAM-4279] Handle Multiple copies of a single input in the Fn 
Harness"
 > git rev-list --no-walk 7beece05ba01b349c88a40f7caa1d0346d69f9e8 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesRunner_Dataflow] $ /bin/bash -xe 
/tmp/jenkins8414068634803319707.sh
+ cd src
+ bash sdks/python/run_validatesrunner.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# INFRA does not install virtualenv
pip install virtualenv --user
Requirement already satisfied: virtualenv in /usr/lib/python2.7/dist-packages 
(15.0.1)

# Virtualenv for the rest of the script to run setup & e2e tests
${LOCAL_PATH}/virtualenv sdks/python
sdks/python/run_validatesrunner.sh: line 38: 
/home/jenkins/.local/bin//virtualenv: No such file or directory
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user ekirpic...@gmail.com

[jira] [Assigned] (BEAM-4279) The Fn Harness doesn't properly handle Flattens which consume a single input multiple times

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4279:
---

Assignee: Thomas Groh  (was: Luke Cwik)

> The Fn Harness doesn't properly handle Flattens which consume a single input 
> multiple times
> ---
>
> Key: BEAM-4279
> URL: https://issues.apache.org/jira/browse/BEAM-4279
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> For a {{Flatten}} which consumes a {{PCollection}} n times, each input 
> element should be present {{n}} times in the output {{PCollection}}. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4279) The Fn Harness doesn't properly handle Flattens which consume a single input multiple times

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4279.
-
   Resolution: Fixed
Fix Version/s: 2.5.0

> The Fn Harness doesn't properly handle Flattens which consume a single input 
> multiple times
> ---
>
> Key: BEAM-4279
> URL: https://issues.apache.org/jira/browse/BEAM-4279
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Thomas Groh
>Assignee: Luke Cwik
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> For a {{Flatten}} which consumes a {{PCollection}} n times, each input 
> element should be present {{n}} times in the output {{PCollection}}. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #150

[Apache Jenkins Server]

See 


--
[...truncated 11.75 KB...]
Initialized native services in: /home/jenkins/.gradle/native
Using 16 worker leases.
Starting Build
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 
'
Included projects: [root project 'beam', project ':beam-examples-java', project 
':beam-local-artifact-service-java', project ':beam-model-fn-execution', 
project ':beam-model-job-management', project ':beam-model-pipeline', project 
':beam-runners-apex', project ':beam-runners-core-construction-java', project 
':beam-runners-core-java', project ':beam-runners-direct-java', project 
':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', 
project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', 
project ':beam-runners-gearpump', project 
':beam-runners-google-cloud-dataflow-java', project 
':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', 
project ':beam-runners-reference-java', project 
':beam-runners-reference-job-server', project ':beam-runners-spark', project 
':beam-sdks-go', project ':beam-sdks-go-container', project 
':beam-sdks-go-examples', project ':beam-sdks-java-build-tools', project 
':beam-sdks-java-container', project ':beam-sdks-java-core', project 
':beam-sdks-java-extensions-google-cloud-platform-core', project 
':beam-sdks-java-extensions-join-library', project 
':beam-sdks-java-extensions-json-jackson', project 
':beam-sdks-java-extensions-protobuf', project 
':beam-sdks-java-extensions-sketching', project 
':beam-sdks-java-extensions-sorter', project ':beam-sdks-java-extensions-sql', 
project ':beam-sdks-java-fn-execution', project ':beam-sdks-java-harness', 
project ':beam-sdks-java-io-amazon-web-services', project 
':beam-sdks-java-io-amqp', project ':beam-sdks-java-io-cassandra', project 
':beam-sdks-java-io-common', project ':beam-sdks-java-io-elasticsearch', 
project ':beam-sdks-java-io-elasticsearch-tests-2', project 
':beam-sdks-java-io-elasticsearch-tests-5', project 
':beam-sdks-java-io-elasticsearch-tests-common', project 
':beam-sdks-java-io-file-based-io-tests', project 
':beam-sdks-java-io-google-cloud-platform', project 
':beam-sdks-java-io-hadoop-common', project 
':beam-sdks-java-io-hadoop-file-system', project 
':beam-sdks-java-io-hadoop-input-format', project ':beam-sdks-java-io-hbase', 
project ':beam-sdks-java-io-hcatalog', project ':beam-sdks-java-io-jdbc', 
project ':beam-sdks-java-io-jms', project ':beam-sdks-java-io-kafka', project 
':beam-sdks-java-io-kinesis', project ':beam-sdks-java-io-mongodb', project 
':beam-sdks-java-io-mqtt', project ':beam-sdks-java-io-redis', project 
':beam-sdks-java-io-solr', project ':beam-sdks-java-io-tika', project 
':beam-sdks-java-io-xml', project ':beam-sdks-java-javadoc', project 
':beam-sdks-java-maven-archetypes-examples', project 
':beam-sdks-java-maven-archetypes-starter', project ':beam-sdks-java-nexmark', 
project ':beam-sdks-python', project ':beam-sdks-python-container', project 
':release']
Parallel execution with configuration on demand is an incubating feature.

> Configure project :
Evaluating root project 'beam' using build file 
'
Applying build_rules.gradle to beam
Offline dependencies root configured at 'offline-repository'
createPerformanceTestHarness with default configuration for project beam
Adding 47 .gitignore exclusions to Apache Rat

> Configure project :beam-sdks-python
Evaluating project ':beam-sdks-python' using build file 
'
Compiling build file 
'
 using SubsetScriptTransformer.
Compiling build file 
'
 using BuildScriptTransformer.
Selected primary task ':beam-sdks-python:setupVirtualenv' from project 
:beam-sdks-python
All projects evaluated.
Tasks to be executed: [task ':beam-sdks-python:setupVirtualenv']
VisTEG: Dependency graph written into 

:beam-sdks-python:setupVirtualenv (Thread[Task worker for ':' Thread 
12,5,main]) started.

> Task :beam-sdks-python:setup

[jira] [Work logged] (BEAM-4236) Side inputs do not work on Dataflow

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4236?focusedWorklogId=101864&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101864
 ]

ASF GitHub Bot logged work on BEAM-4236:


Author: ASF GitHub Bot
Created on: 14/May/18 19:46
Start Date: 14/May/18 19:46
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #5302: [BEAM-4236, 
BEAM-2927] Make Python SDK side inputs work with non well known coders and also 
work with Dataflow
URL: https://github.com/apache/beam/pull/5302#issuecomment-388939376
 
 
   The refcounts should only be needed for the old DirectRunner. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101864)
Time Spent: 6h 10m  (was: 6h)

> Side inputs do not work on Dataflow
> ---
>
> Key: BEAM-4236
> URL: https://issues.apache.org/jira/browse/BEAM-4236
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Affects Versions: Not applicable
>Reporter: Charles Chen
>Assignee: Luke Cwik
>Priority: Blocker
> Fix For: 2.5.0
>
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> Dataflow side inputs do not work in streaming / batch FnApi mode at head.  
> The pipeline stalls indefinitely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101865&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101865
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 14/May/18 19:47
Start Date: 14/May/18 19:47
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5347: [BEAM-4201][SQL] Add 
integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#issuecomment-388939779
 
 
   fixed typos, added global windowing


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101865)
Time Spent: 1.5h  (was: 1h 20m)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4963

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add a FlattenRunnerTest for a multi-consumed input

[tgroh] Use ListMultimap when creating Transform Runners

--
[...truncated 1.01 MB...]
copying apache_beam/runners/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct

[jira] [Work logged] (BEAM-4204) Python: PortableRunner - p.run() via given JobService

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4204?focusedWorklogId=101869&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101869
 ]

ASF GitHub Bot logged work on BEAM-4204:


Author: ASF GitHub Bot
Created on: 14/May/18 20:18
Start Date: 14/May/18 20:18
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #5301: [BEAM-4204] 
Splitting ULR in portable runner stub and job service
URL: https://github.com/apache/beam/pull/5301#issuecomment-388948757
 
 
   This split makes a lot of sense to me. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101869)
Time Spent: 2h 20m  (was: 2h 10m)

> Python: PortableRunner - p.run() via given JobService
> -
>
> Key: BEAM-4204
> URL: https://issues.apache.org/jira/browse/BEAM-4204
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Like BEAM-4071 but for Python. Is this fully encompassed by 
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/universal_local_runner.py]
>  ? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101870&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101870
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 14/May/18 20:19
Start Date: 14/May/18 20:19
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5347: [BEAM-4201][SQL] Add 
integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#issuecomment-388949153
 
 
   run python precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101870)
Time Spent: 1h 40m  (was: 1.5h)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101871&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101871
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 14/May/18 20:19
Start Date: 14/May/18 20:19
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5347: [BEAM-4201][SQL] Add 
integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#issuecomment-388949153
 
 
   run python precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101871)
Time Spent: 1h 50m  (was: 1h 40m)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4284) PipelineTranslationTest.testProtoAgainstRehydrated() over-validates translated proto structure

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474751#comment-16474751
 ] 

Kenneth Knowles commented on BEAM-4284:
---

Back in the day, I sunk an excessive amount of time looking for appropriate 
ways to validate isomorphism of planar DAGs. That would be the gold standard of 
pipeline translation testing. I tried to err on the side of not overspecifying, 
but with the caveat that without the above it is pretty underspecified. I may 
have struck the wrong balance of testing the spec vs the shortcut of testing 
the current implementation. I'll try to find time to take a look and loosen it 
up.

> PipelineTranslationTest.testProtoAgainstRehydrated() over-validates 
> translated proto structure
> --
>
> Key: BEAM-4284
> URL: https://issues.apache.org/jira/browse/BEAM-4284
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Scott Wegner
>Assignee: Kenneth Knowles
>Priority: Minor
>
> PipelineTranslationTest.testProtoAgainstRehydrated() validates round-tripping 
> of a Pipeline object to and form proto, verifying each of the rehydrated 
> fields. From conversations with [~lcwik], the validation logic is 
> over-specified. There is no requirement that equivalent objects are 
> serialized using the same ID. So iterating to find the set of unique IDs and 
> validating the count is incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5324: Use preinstalled virtualenv on Jenkins nodes.

[jaku]

This is an automated email from the ASF dual-hosted git repository.

jaku pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 1c8f6a216e314c2e8d22412590e32939c09d25c0
Merge: f19a717 42fd4c0
Author: jasonkuster 
AuthorDate: Mon May 14 13:30:44 2018 -0700

Merge pull request #5324: Use preinstalled virtualenv on Jenkins nodes.

Use preinstalled virtualenv on Jenkins nodes.

 sdks/python/run_validatesrunner.sh | 9 +
 1 file changed, 1 insertion(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (f19a717 -> 1c8f6a2)

[jaku]

This is an automated email from the ASF dual-hosted git repository.

jaku pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f19a717  [BEAM-4279] Handle Multiple copies of a single input in the 
Fn Harness
 add 42fd4c0  Use preinstalled virtualenv on Jenkins nodes.
 new 1c8f6a2  Merge pull request #5324: Use preinstalled virtualenv on 
Jenkins nodes.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/run_validatesrunner.sh | 9 +
 1 file changed, 1 insertion(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Work logged] (BEAM-4280) DirectStreamObserver for outbound channel can block indefinitely if invoked from inbound channel thread causing deadlock

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4280?focusedWorklogId=101872&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101872
 ]

ASF GitHub Bot logged work on BEAM-4280:


Author: ASF GitHub Bot
Created on: 14/May/18 20:34
Start Date: 14/May/18 20:34
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5345: 
[BEAM-4280] Prevent DirectStreamObserver from blocking indefinitely if invoked 
from inbound channel thread
URL: https://github.com/apache/beam/pull/5345#discussion_r188089152
 
 

 ##
 File path: 
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/stream/DirectStreamObserver.java
 ##
 @@ -34,21 +38,66 @@
  */
 @ThreadSafe
 public final class DirectStreamObserver implements StreamObserver {
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(DirectStreamObserver.class);
+  private static final int DEFAULT_MAX_MESSAGES_BEFORE_CHECK = 100;
+
   private final Phaser phaser;
   private final CallStreamObserver outboundObserver;
+  private final int maxMessagesBeforeCheck;
+
+  private int numberOfMessagesBeforeReadyCheck;
 
   public DirectStreamObserver(
   Phaser phaser,
   CallStreamObserver outboundObserver) {
+this(phaser, outboundObserver, DEFAULT_MAX_MESSAGES_BEFORE_CHECK);
+  }
+
+  DirectStreamObserver(
+  Phaser phaser,
+  CallStreamObserver outboundObserver,
+  int maxMessagesBeforeCheck) {
 this.phaser = phaser;
 this.outboundObserver = outboundObserver;
+this.maxMessagesBeforeCheck = maxMessagesBeforeCheck;
   }
 
   @Override
   public void onNext(T value) {
-int phase = phaser.getPhase();
-if (!outboundObserver.isReady()) {
-  phaser.awaitAdvance(phase);
+numberOfMessagesBeforeReadyCheck += 1;
+if (numberOfMessagesBeforeReadyCheck >= maxMessagesBeforeCheck) {
+  numberOfMessagesBeforeReadyCheck = 0;
+  int waitTime = 1;
+  int totalTimeWaited = 0;
+  int phase = phaser.getPhase();
+  while (!outboundObserver.isReady()) {
+try {
+  phaser.awaitAdvanceInterruptibly(phase, waitTime, TimeUnit.SECONDS);
+} catch (TimeoutException e) {
+  totalTimeWaited += waitTime;
+  waitTime = waitTime * 2;
+} catch (InterruptedException e) {
+  Thread.currentThread().interrupt();
+  throw new RuntimeException(e);
+}
+  }
+  if (totalTimeWaited > 0) {
+// If the phase didn't change, this means that the installed onReady 
callback had not
+// been invoked.
+if (phase == phaser.getPhase()) {
+  LOGGER.info(
+  "Output channel stalled for {}s, outbound thread {}. gRPC 
requires that outbound "
 
 Review comment:
   Updated the comment and referred to the JIRA.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101872)
Time Spent: 1h 10m  (was: 1h)

> DirectStreamObserver for outbound channel can block indefinitely if invoked 
> from inbound channel thread causing deadlock
> 
>
> Key: BEAM-4280
> URL: https://issues.apache.org/jira/browse/BEAM-4280
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> gRPC docs say that: 
>         // Note: the onReadyHandler's invocation is serialized on the same 
> thread pool as the incoming StreamObserver's
>         // onNext(), onError(), and onComplete() handlers. Blocking the 
> onReadyHandler will prevent additional messages
>         // from being processed by the incoming StreamObserver. The 
> onReadyHandler must return in a timely manor or else
>         // message processing throughput will suffer.
> Looking at the stack, it i because one of the gRPC threads is blocked waiting 
> for the channel to become ready, preventing for that same thread to mark it 
> as ready:
> "grpc-default-executor-0" #12 daemon prio=5 os_prio=0 tid=0x7fcea88ee800 
> nid=0x3cc8a waiting on condition [0x7fce4b9f8000]
>    java.lang.Thread.State: WAITING (parking)
>         at (C/C++) 0x7fcead7519f2 (Unknown Source)
>         at (C/C++) 0x7fceac8b8f11 (Unknown Source)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x000740e39c48> 

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101876&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101876
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 14/May/18 20:47
Start Date: 14/May/18 20:47
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5347: [BEAM-4201][SQL] Add 
integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#issuecomment-388957365
 
 
   run python precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101876)
Time Spent: 2h  (was: 1h 50m)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4269) Implement Assign Windows in the Java SDK Harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4269?focusedWorklogId=101877&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101877
 ]

ASF GitHub Bot logged work on BEAM-4269:


Author: ASF GitHub Bot
Created on: 14/May/18 20:49
Start Date: 14/May/18 20:49
Worklog Time Spent: 10m 
  Work Description: tgroh commented on issue #5329: [BEAM-4269, BEAM-3970] 
Implement AssignWindows in the Java SDK harness
URL: https://github.com/apache/beam/pull/5329#issuecomment-388957890
 
 
   Fixed. We hadn't been exploding input windowed values either, so fixed that 
as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101877)
Time Spent: 3h 10m  (was: 3h)

> Implement Assign Windows in the Java SDK Harness
> 
>
> Key: BEAM-4269
> URL: https://issues.apache.org/jira/browse/BEAM-4269
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-harness
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> This allows execution of Java WindowFns over the Fn API



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4127) Flink portable runner translates streaming pipelines by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4127?focusedWorklogId=101880&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101880
 ]

ASF GitHub Bot logged work on BEAM-4127:


Author: ASF GitHub Bot
Created on: 14/May/18 20:53
Start Date: 14/May/18 20:53
Worklog Time Spent: 10m 
  Work Description: tweise opened a new pull request #5358: [BEAM-4127] 
Flink streaming translation for portable pipeline.
URL: https://github.com/apache/beam/pull/5358
 
 
   This is the translator code from the prototype, minus executable stage 
translation (which I'm going to work an as follow-up).
   
   R: @bsidhom @lukecwik 
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `./gradlew build` to make sure basic checks pass. A more thorough 
check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101880)
Time Spent: 10m
Remaining Estimate: 0h

> Flink portable runner translates streaming pipelines by proto
> -
>
> Key: BEAM-4127
> URL: https://issues.apache.org/jira/browse/BEAM-4127
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Thomas Weise
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)