Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #3125

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-1491) HDFSFileSource should be able to read the HADOOP_CONF_DIR(YARN_CONF_DIR) environmen variable

[yangping wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yangping wu updated BEAM-1491:
--
Description: 
Currently, if we want to read file store on HDFS, we will do it as follow:
{code} PCollection> resultCollection = 
p.apply(HDFSFileSource.readFrom(
"hdfs://hadoopserver:8020/tmp/data.txt",
TextInputFormat.class, LongWritable.class, Text.class));
{code}
As we have seen above, we must be set {{hdfs://hadoopserver:8020}} in the file 
path, and we cann't set any variables when read file, because in 
[HDFSFileSource.java|https://github.com/apache/beam/blob/master/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java#L310]
 we initialize {{job}} instance as follow:
{code}
this.job = Job.getInstance();
{code}
we should initialize {{job}} instance by configure:
{code}
this.job = Job.getInstance(conf);
{code}
where {{conf}} is instance of {{Configuration}}, and we initialize {{conf}} by 
reading {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen variable,then we can 
read HDFS file like this:
{code} PCollection> resultCollection = 
p.apply(HDFSFileSource.readFrom(
"/tmp/data.txt",
TextInputFormat.class, LongWritable.class, Text.class));
{code}
note we don't specify {{hdfs://hadoopserver:8020}} prefix, because the program 
read it from  {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen.

  was:
Currently, if we want to read file store on HDFS, we will do it as follow:
{code} PCollection> resultCollection = 
p.apply(HDFSFileSource.readFrom(
"hdfs://hadoopserver:8020/tmp/data.txt",
TextInputFormat.class, LongWritable.class, Text.class));
{code}
As we have seen above, we must be set {{hdfs://hadoopserver:8020}} in the file 
path, and we cann't set any variables when read file, because in 
[HDFSFileSource.java|https://github.com/apache/beam/blob/master/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java#L310]
 we initialize {{job}} instance as follow:
{code}
this.job = Job.getInstance();
{code}
we should initialize {{job}} instance by configure:
{code}
this.job = Job.getInstance(conf);
{code}
where {{conf}} is instance of {{Configuration}}, and we initialize {{conf}} by 
reading {{HADOOP_CONF}}({{YARN_CONF}}) environmen variable,then we can read 
HDFS file like this:
{code} PCollection> resultCollection = 
p.apply(HDFSFileSource.readFrom(
"/tmp/data.txt",
TextInputFormat.class, LongWritable.class, Text.class));
{code}
note we don't specify {{hdfs://hadoopserver:8020}} prefix, because the program 
read it from  {{HADOOP_CONF}}({{YARN_CONF}}) environmen.


> HDFSFileSource should be able to read the HADOOP_CONF_DIR(YARN_CONF_DIR) 
> environmen variable
> 
>
> Key: BEAM-1491
> URL: https://issues.apache.org/jira/browse/BEAM-1491
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 0.5.0
>Reporter: yangping wu
>Assignee: Jean-Baptiste Onofré
>
> Currently, if we want to read file store on HDFS, we will do it as follow:
> {code} PCollection> resultCollection = 
> p.apply(HDFSFileSource.readFrom(
> "hdfs://hadoopserver:8020/tmp/data.txt",
> TextInputFormat.class, LongWritable.class, Text.class));
> {code}
> As we have seen above, we must be set {{hdfs://hadoopserver:8020}} in the 
> file path, and we cann't set any variables when read file, because in 
> [HDFSFileSource.java|https://github.com/apache/beam/blob/master/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java#L310]
>  we initialize {{job}} instance as follow:
> {code}
> this.job = Job.getInstance();
> {code}
> we should initialize {{job}} instance by configure:
> {code}
> this.job = Job.getInstance(conf);
> {code}
> where {{conf}} is instance of {{Configuration}}, and we initialize {{conf}} 
> by reading {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen variable,then we 
> can read HDFS file like this:
> {code} PCollection> resultCollection = 
> p.apply(HDFSFileSource.readFrom(
> "/tmp/data.txt",
> TextInputFormat.class, LongWritable.class, Text.class));
> {code}
> note we don't specify {{hdfs://hadoopserver:8020}} prefix, because the 
> program read it from  {{HADOOP_CONF_DIR}}({{YARN_CONF_DIR}}) environmen.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1491) HDFSFileSource should be able to read the HADOOP_CONF_DIR(YARN_CONF_DIR) environmen variable

[yangping wu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yangping wu updated BEAM-1491:
--
Summary: HDFSFileSource should be able to read the 
HADOOP_CONF_DIR(YARN_CONF_DIR) environmen variable  (was: HDFSFileSource should 
be able to read the HADOOP_CONF(YARN_CONF) environmen variable)

> HDFSFileSource should be able to read the HADOOP_CONF_DIR(YARN_CONF_DIR) 
> environmen variable
> 
>
> Key: BEAM-1491
> URL: https://issues.apache.org/jira/browse/BEAM-1491
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 0.5.0
>Reporter: yangping wu
>Assignee: Jean-Baptiste Onofré
>
> Currently, if we want to read file store on HDFS, we will do it as follow:
> {code} PCollection> resultCollection = 
> p.apply(HDFSFileSource.readFrom(
> "hdfs://hadoopserver:8020/tmp/data.txt",
> TextInputFormat.class, LongWritable.class, Text.class));
> {code}
> As we have seen above, we must be set {{hdfs://hadoopserver:8020}} in the 
> file path, and we cann't set any variables when read file, because in 
> [HDFSFileSource.java|https://github.com/apache/beam/blob/master/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs/HDFSFileSource.java#L310]
>  we initialize {{job}} instance as follow:
> {code}
> this.job = Job.getInstance();
> {code}
> we should initialize {{job}} instance by configure:
> {code}
> this.job = Job.getInstance(conf);
> {code}
> where {{conf}} is instance of {{Configuration}}, and we initialize {{conf}} 
> by reading {{HADOOP_CONF}}({{YARN_CONF}}) environmen variable,then we can 
> read HDFS file like this:
> {code} PCollection> resultCollection = 
> p.apply(HDFSFileSource.readFrom(
> "/tmp/data.txt",
> TextInputFormat.class, LongWritable.class, Text.class));
> {code}
> note we don't specify {{hdfs://hadoopserver:8020}} prefix, because the 
> program read it from  {{HADOOP_CONF}}({{YARN_CONF}}) environmen.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PerformanceTests_Dataflow #255

[Apache Jenkins Server]

See 


Changes:

[kirpichov] KafkaIO : Add withTopic() api that takes single topic.

[tgroh] Separate View Evaluation and Overrides

[robertwb] Only encode PCollection outputs in Runner API protos.

[robertwb] Ensure transforms are picklable before materializing to protos.

[robertwb] Fix side inputs on dataflow runner.

--
[...truncated 271.57 KB...]
 * [new ref] refs/pull/928/head -> origin/pr/928/head
 * [new ref] refs/pull/929/head -> origin/pr/929/head
 * [new ref] refs/pull/93/head -> origin/pr/93/head
 * [new ref] refs/pull/930/head -> origin/pr/930/head
 * [new ref] refs/pull/930/merge -> origin/pr/930/merge
 * [new ref] refs/pull/931/head -> origin/pr/931/head
 * [new ref] refs/pull/931/merge -> origin/pr/931/merge
 * [new ref] refs/pull/932/head -> origin/pr/932/head
 * [new ref] refs/pull/932/merge -> origin/pr/932/merge
 * [new ref] refs/pull/933/head -> origin/pr/933/head
 * [new ref] refs/pull/933/merge -> origin/pr/933/merge
 * [new ref] refs/pull/934/head -> origin/pr/934/head
 * [new ref] refs/pull/934/merge -> origin/pr/934/merge
 * [new ref] refs/pull/935/head -> origin/pr/935/head
 * [new ref] refs/pull/936/head -> origin/pr/936/head
 * [new ref] refs/pull/936/merge -> origin/pr/936/merge
 * [new ref] refs/pull/937/head -> origin/pr/937/head
 * [new ref] refs/pull/937/merge -> origin/pr/937/merge
 * [new ref] refs/pull/938/head -> origin/pr/938/head
 * [new ref] refs/pull/939/head -> origin/pr/939/head
 * [new ref] refs/pull/94/head -> origin/pr/94/head
 * [new ref] refs/pull/940/head -> origin/pr/940/head
 * [new ref] refs/pull/940/merge -> origin/pr/940/merge
 * [new ref] refs/pull/941/head -> origin/pr/941/head
 * [new ref] refs/pull/941/merge -> origin/pr/941/merge
 * [new ref] refs/pull/942/head -> origin/pr/942/head
 * [new ref] refs/pull/942/merge -> origin/pr/942/merge
 * [new ref] refs/pull/943/head -> origin/pr/943/head
 * [new ref] refs/pull/943/merge -> origin/pr/943/merge
 * [new ref] refs/pull/944/head -> origin/pr/944/head
 * [new ref] refs/pull/945/head -> origin/pr/945/head
 * [new ref] refs/pull/945/merge -> origin/pr/945/merge
 * [new ref] refs/pull/946/head -> origin/pr/946/head
 * [new ref] refs/pull/946/merge -> origin/pr/946/merge
 * [new ref] refs/pull/947/head -> origin/pr/947/head
 * [new ref] refs/pull/947/merge -> origin/pr/947/merge
 * [new ref] refs/pull/948/head -> origin/pr/948/head
 * [new ref] refs/pull/948/merge -> origin/pr/948/merge
 * [new ref] refs/pull/949/head -> origin/pr/949/head
 * [new ref] refs/pull/949/merge -> origin/pr/949/merge
 * [new ref] refs/pull/95/head -> origin/pr/95/head
 * [new ref] refs/pull/95/merge -> origin/pr/95/merge
 * [new ref] refs/pull/950/head -> origin/pr/950/head
 * [new ref] refs/pull/951/head -> origin/pr/951/head
 * [new ref] refs/pull/951/merge -> origin/pr/951/merge
 * [new ref] refs/pull/952/head -> origin/pr/952/head
 * [new ref] refs/pull/952/merge -> origin/pr/952/merge
 * [new ref] refs/pull/953/head -> origin/pr/953/head
 * [new ref] refs/pull/954/head -> origin/pr/954/head
 * [new ref] refs/pull/954/merge -> origin/pr/954/merge
 * [new ref] refs/pull/955/head -> origin/pr/955/head
 * [new ref] refs/pull/955/merge -> origin/pr/955/merge
 * [new ref] refs/pull/956/head -> origin/pr/956/head
 * [new ref] refs/pull/957/head -> origin/pr/957/head
 * [new ref] refs/pull/958/head -> origin/pr/958/head
 * [new ref] refs/pull/959/head -> origin/pr/959/head
 * [new ref] refs/pull/959/merge -> origin/pr/959/merge
 * [new ref] refs/pull/96/head -> origin/pr/96/head
 * [new ref] refs/pull/96/merge -> origin/pr/96/merge
 * [new ref] refs/pull/960/head -> origin/pr/960/head
 * [new ref] refs/pull/960/merge -> origin/pr/960/merge
 * [new ref] refs/pull/961/head -> origin/pr/961/head
 * [new ref] refs/pull/962/head -> origin/pr/962/head
 * [new ref] refs/pull/962/merge -> origin/pr/962/merge
 * [new ref] refs/pull/963/head -> origin/pr/963/head
 * [new ref] refs/pull/963/merge -> origin/pr/963/merge
 * [new ref] refs/pull/964/head -> origin/pr/964/head
 * [new ref] refs/pull/965/head -> origin/pr/965/head
 * [new ref] refs/pull/965/merge -> origin/pr/965/merge
 * [new ref] refs/pull/966/head -> origin/pr/966/head
 * [new ref] refs/pull/967/head -> origin/pr/967/head
 * [new ref] refs/pull/967/merge -> origin/pr/967/merge
 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #1702

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3124

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #3123

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #1701

[Apache Jenkins Server]

See 


Changes:

[robertwb] Only encode PCollection outputs in Runner API protos.

[robertwb] Ensure transforms are picklable before materializing to protos.

[robertwb] Fix side inputs on dataflow runner.

--
[...truncated 876.35 KB...]
}, 
"output_name": "out", 
"user_name": "assert_that/Group/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s12"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Group/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert_that/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": ""
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s15", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Unkey.out"
  }
], 
"parallel

[jira] [Commented] (BEAM-1853) AttributeError: 'AsIter' object has no attribute 'producer' (post commit failure)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952018#comment-15952018
 ] 

ASF GitHub Bot commented on BEAM-1853:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2395


> AttributeError: 'AsIter' object has no attribute 'producer' (post commit 
> failure)
> -
>
> Key: BEAM-1853
> URL: https://issues.apache.org/jira/browse/BEAM-1853
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Robert Bradshaw
>
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/1694/console
> One of the failures (there are other related errors):
> ==
> ERROR: test_iterable_side_input 
> (apache_beam.transforms.sideinputs_test.SideInputsTest)
> --
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/transforms/sideinputs_test.py",
>  line 179, in test_iterable_side_input
> pipeline.run()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/test_pipeline.py",
>  line 91, in run
> result = super(TestPipeline, self).run()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 169, in run
> return self.runner.run(self)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
>  line 32, in run
> self.result = super(TestDataflowRunner, self).run(pipeline)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
>  line 161, in run
> super(DataflowRunner, self).run(pipeline)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
>  line 122, in run
> pipeline.visit(RunVisitor(self))
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 192, in visit
> self._root_transform().visit(visitor, self, visited)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 471, in visit
> part.visit(visitor, pipeline, visited)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 474, in visit
> visitor.visit_transform(self)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
>  line 117, in visit_transform
> self.runner.run_transform(transform_node)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
>  line 192, in run_transform
> return m(transform_node)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
>  line 378, in run_ParDo
> si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
>  line 266, in get_pvalue
> self._ensure_pvalue_has_real_producer(pvalue)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
>  line 239, in _ensure_pvalue_has_real_producer
> real_producer = pvalue.producer
> AttributeError: 'AsIter' object has no attribute 'producer'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2395: [BEAM-1853] Fix side inputs on dataflow runner.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2395


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Fix side inputs on dataflow runner.

[robertwb]

Fix side inputs on dataflow runner.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/07daf3a5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/07daf3a5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/07daf3a5

Branch: refs/heads/master
Commit: 07daf3a54544ce842165ffe15264e43ebced28ba
Parents: 60901f8
Author: Robert Bradshaw 
Authored: Fri Mar 31 14:57:44 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Mar 31 21:45:16 2017 -0700

--
 .../apache_beam/runners/dataflow/dataflow_runner.py| 13 +++--
 1 file changed, 7 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/07daf3a5/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index db433df..fe9f8c0 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -287,7 +287,7 @@ class DataflowRunner(PipelineRunner):
   def _add_singleton_step(self, label, full_label, tag, input_step):
 """Creates a CollectionToSingleton step used to handle ParDo side 
inputs."""
 # Import here to avoid adding the dependency for local running scenarios.
-from google.cloud.dataflow.internal import apiclient
+from apache_beam.runners.dataflow.internal import apiclient
 step = apiclient.Step(TransformNames.COLLECTION_TO_SINGLETON, label)
 self.job.proto.steps.append(step.proto)
 step.add_property(PropertyNames.USER_NAME, full_label)
@@ -302,7 +302,7 @@ class DataflowRunner(PipelineRunner):
 [{PropertyNames.USER_NAME: (
 '%s.%s' % (full_label, PropertyNames.OUTPUT)),
   PropertyNames.ENCODING: step.encoding,
-  PropertyNames.OUTPUT_NAME: PropertyNames.OUTPUT}])
+  PropertyNames.OUTPUT_NAME: PropertyNames.OUT}])
 return step
 
   def run_Flatten(self, transform_node):
@@ -374,12 +374,10 @@ class DataflowRunner(PipelineRunner):
 si_dict = {}
 # We must call self._cache.get_pvalue exactly once due to refcounting.
 si_labels = {}
-for side_pval in transform_node.side_inputs:
-  si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
 lookup_label = lambda side_pval: si_labels[side_pval]
 for side_pval in transform_node.side_inputs:
   assert isinstance(side_pval, AsSideInput)
-  si_label = self._get_unique_step_name()
+  si_label = 'SideInput-' + self._get_unique_step_name()
   si_full_label = '%s/%s' % (transform_node.full_label, si_label)
   self._add_singleton_step(
   si_label, si_full_label, side_pval.pvalue.tag,
@@ -388,10 +386,13 @@ class DataflowRunner(PipelineRunner):
   '@type': 'OutputReference',
   PropertyNames.STEP_NAME: si_label,
   PropertyNames.OUTPUT_NAME: PropertyNames.OUT}
+  si_labels[side_pval] = si_label
 
 # Now create the step for the ParDo transform being handled.
 step = self._add_step(
-TransformNames.DO, transform_node.full_label, transform_node,
+TransformNames.DO,
+transform_node.full_label + '/Do' if transform_node.side_inputs else 
'',
+transform_node,
 transform_node.transform.side_output_tags)
 fn_data = self._pardo_fn_data(transform_node, lookup_label)
 step.add_property(PropertyNames.SERIALIZED_FN, pickler.dumps(fn_data))

[jira] [Commented] (BEAM-1843) 'PDone' object has no attribute 'to_runner_api'

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952017#comment-15952017
 ] 

ASF GitHub Bot commented on BEAM-1843:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2396


> 'PDone' object has no attribute 'to_runner_api'
> ---
>
> Key: BEAM-1843
> URL: https://issues.apache.org/jira/browse/BEAM-1843
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Robert Bradshaw
>
> Post commit failure with 
> (https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1676/consoleFull):
> ...
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 512, in 
> for tag, out in self.outputs.items()},
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/pipeline_context.py",
>  line 52, in get_id
> self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
> AttributeError: 'PDone' object has no attribute 'to_runner_api'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Closes #2395

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 60901f876 -> 03dce6dcc


Closes #2395


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/03dce6dc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/03dce6dc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/03dce6dc

Branch: refs/heads/master
Commit: 03dce6dccef50210dd420cb9f79baf063bc0b0e8
Parents: 60901f8 07daf3a
Author: Robert Bradshaw 
Authored: Fri Mar 31 21:45:16 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Mar 31 21:45:16 2017 -0700

--
 .../apache_beam/runners/dataflow/dataflow_runner.py| 13 +++--
 1 file changed, 7 insertions(+), 6 deletions(-)
--

[GitHub] beam pull request #2396: [BEAM-1843] Ensure transforms are picklable before ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2396


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam git commit: Closes #2396

[robertwb]

Closes #2396


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/60901f87
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/60901f87
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/60901f87

Branch: refs/heads/master
Commit: 60901f876aaf9e85c11501557516f9d47b87d994
Parents: 0749982 bd40c5a
Author: Robert Bradshaw 
Authored: Fri Mar 31 21:44:22 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Mar 31 21:44:22 2017 -0700

--
 sdks/python/apache_beam/pipeline.py | 12 +++-
 1 file changed, 11 insertions(+), 1 deletion(-)
--

[1/3] beam git commit: Ensure transforms are picklable before materializing to protos.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 07499824b -> 60901f876


Ensure transforms are picklable before materializing to protos.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bd40c5a2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bd40c5a2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bd40c5a2

Branch: refs/heads/master
Commit: bd40c5a27e379914914a30f79854fd7a38621c66
Parents: c9ff44a
Author: Robert Bradshaw 
Authored: Fri Mar 31 16:57:22 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Mar 31 21:44:21 2017 -0700

--
 sdks/python/apache_beam/pipeline.py | 6 ++
 1 file changed, 6 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/bd40c5a2/sdks/python/apache_beam/pipeline.py
--
diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index 0841e5f..3c416eb 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -311,6 +311,12 @@ class Pipeline(object):
 
   def visit_transform(self, transform_node):
 if transform_node.side_inputs:
+  # No side inputs (yet).
+  Visitor.ok = False
+try:
+  # Transforms must be picklable.
+  pickler.loads(pickler.dumps(transform_node.transform))
+except Exception:
   Visitor.ok = False
 self.visit(Visitor())
 return Visitor.ok

[2/3] beam git commit: Only encode PCollection outputs in Runner API protos.

[robertwb]

Only encode PCollection outputs in Runner API protos.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c9ff44af
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c9ff44af
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c9ff44af

Branch: refs/heads/master
Commit: c9ff44afa3fcb47b7f0c4288f4f7d520f063d442
Parents: 0749982
Author: Robert Bradshaw 
Authored: Fri Mar 31 16:57:01 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Mar 31 21:44:21 2017 -0700

--
 sdks/python/apache_beam/pipeline.py | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c9ff44af/sdks/python/apache_beam/pipeline.py
--
diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index ee5904b..0841e5f 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -494,6 +494,10 @@ class AppliedPTransform(object):
 return {str(ix): input for ix, input in enumerate(self.inputs)
 if isinstance(input, pvalue.PCollection)}
 
+  def named_outputs(self):
+return {str(tag): output for tag, output in self.outputs.items()
+if isinstance(output, pvalue.PCollection)}
+
   def to_runner_api(self, context):
 from apache_beam.runners.api import beam_runner_api_pb2
 return beam_runner_api_pb2.PTransform(
@@ -507,7 +511,7 @@ class AppliedPTransform(object):
 inputs={tag: context.pcollections.get_id(pc)
 for tag, pc in self.named_inputs().items()},
 outputs={str(tag): context.pcollections.get_id(out)
- for tag, out in self.outputs.items()},
+ for tag, out in self.named_outputs().items()},
 # TODO(BEAM-115): display_data
 display_data=None)
 

[jira] [Commented] (BEAM-1684) Add unit tests for iobase.py

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951980#comment-15951980
 ] 

ASF GitHub Bot commented on BEAM-1684:
--

GitHub user duraza opened a pull request:

https://github.com/apache/beam/pull/2397

[BEAM-1684] Add unit tests for iobase.py.

**Summary**: This pull request adds unit tests for Dataflow's `iobase.py`
**Comments**: As this is my first PR to Beam, I erred on the side of 
caution, adding a large number of tests. The file I am testing is largely a 
base class with limited functionality, so these tests are very likely to be 
redundant, and debatably unnecessary. Please let me know, and I'll be sure to 
correct accordingly.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/duraza/beam add-dataflow-iobase-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2397.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2397


commit 96f77cd61cb6f9b001e5f53d15c9ef2418111b48
Author: Rahul Sabbineni 
Date:   2017-04-01T03:35:21Z

[BEAM-1684] Add unit tests for iobase.py.




> Add unit tests for iobase.py
> 
>
> Key: BEAM-1684
> URL: https://issues.apache.org/jira/browse/BEAM-1684
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py
>Reporter: Tibor Kiss
>Assignee: Rahul Sabbineni
>Priority: Minor
>
> Python-SDK's {{apache_beam/runners/dataflow/native_io/iobase.py}} does not 
> have associated unit tests and has low (indirect) test coverage.
>  
> Create the respective tests to ensure code quality and increase test coverage.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2397: [BEAM-1684] Add unit tests for iobase.py.

[duraza]

GitHub user duraza opened a pull request:

https://github.com/apache/beam/pull/2397

[BEAM-1684] Add unit tests for iobase.py.

**Summary**: This pull request adds unit tests for Dataflow's `iobase.py`
**Comments**: As this is my first PR to Beam, I erred on the side of 
caution, adding a large number of tests. The file I am testing is largely a 
base class with limited functionality, so these tests are very likely to be 
redundant, and debatably unnecessary. Please let me know, and I'll be sure to 
correct accordingly.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/duraza/beam add-dataflow-iobase-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2397.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2397


commit 96f77cd61cb6f9b001e5f53d15c9ef2418111b48
Author: Rahul Sabbineni 
Date:   2017-04-01T03:35:21Z

[BEAM-1684] Add unit tests for iobase.py.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Python_Verify #1700

[Apache Jenkins Server]

See 


--
[...truncated 665.52 KB...]
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsList' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_with_different_defaults 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 248, in test_as_singleton_with_different_defaults
pipeline.run()
  File 
"
 line 91, in run
result = super(TestPipeline, self).run()
  File 
"
 line 169, in run
return self.runner.run(self)
  File 
"
 line 32, in run
self.result = super(TestDataflowRunner, self).run(pipeline)
  File 
"
 line 161, in run
super(DataflowRunner, self).run(pipeline)
  File 
"
 line 122, in run
pipeline.visit(RunVisitor(self))
  File 
"
 line 192, in visit
self._root_transform().visit(visitor, self, visited)
  File 
"
 line 471, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 474, in visit
visitor.visit_transform(self)
  File 
"
 line 117, in visit_transform
self.runner.run_transform(transform_node)
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsSingleton' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_without_unique_labels 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 226, in test_as_si

[jira] [Commented] (BEAM-1854) Replace the magic number 42 in PAssert.

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951951#comment-15951951
 ] 

Pei He commented on BEAM-1854:
--

For example, a string like "PAssert-global-key" would be better.

> Replace the magic number 42 in PAssert.
> ---
>
> Key: BEAM-1854
> URL: https://issues.apache.org/jira/browse/BEAM-1854
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Pei He
>Assignee: Davor Bonaci
>Priority: Minor
>
> PAssert used integer 42 as the global key. It causes some confusion when 
> debugging ValidatesRunner tests.
> final int combinedKey = 42;
> https://github.com/apache/beam/find/master#L935



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1854) Replace the magic number 42 in PAssert.

[Pei He (JIRA)]

Pei He created BEAM-1854:


 Summary: Replace the magic number 42 in PAssert.
 Key: BEAM-1854
 URL: https://issues.apache.org/jira/browse/BEAM-1854
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Pei He
Assignee: Davor Bonaci
Priority: Minor


PAssert used integer 42 as the global key. It causes some confusion when 
debugging ValidatesRunner tests.

final int combinedKey = 42;
https://github.com/apache/beam/find/master#L935



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3122

[Apache Jenkins Server]

See 


Changes:

[tgroh] Separate View Evaluation and Overrides

--
[...truncated 2.26 MB...]
2017-04-01T02:01:14.698 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-framework/2.6.0/curator-framework-2.6.0.jar
 (181 KB at 166.4 KB/sec)
2017-04-01T02:01:14.698 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-recipes/2.6.0/curator-recipes-2.6.0.jar
2017-04-01T02:01:14.718 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.6.0/hadoop-yarn-api-2.6.0.jar
 (1827 KB at 1649.8 KB/sec)
2017-04-01T02:01:14.719 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/htrace/htrace-core/3.0.4/htrace-core-3.0.4.jar
2017-04-01T02:01:14.743 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-client/2.6.0/curator-client-2.6.0.jar
 (68 KB at 59.4 KB/sec)
2017-04-01T02:01:14.743 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/lingala/zip4j/zip4j/1.3.2/zip4j-1.3.2.jar
2017-04-01T02:01:14.772 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-recipes/2.6.0/curator-recipes-2.6.0.jar
 (243 KB at 208.7 KB/sec)
2017-04-01T02:01:14.774 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/xbean/xbean-asm5-shaded/4.3/xbean-asm5-shaded-4.3.jar
2017-04-01T02:01:14.775 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/htrace/htrace-core/3.0.4/htrace-core-3.0.4.jar
 (31 KB at 26.2 KB/sec)
2017-04-01T02:01:14.776 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jctools/jctools-core/1.1/jctools-core-1.1.jar
2017-04-01T02:01:14.800 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/lingala/zip4j/zip4j/1.3.2/zip4j-1.3.2.jar
 (129 KB at 107.9 KB/sec)
2017-04-01T02:01:14.841 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0-tests.jar
 (1717 KB at 1395.2 KB/sec)
2017-04-01T02:01:14.842 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/xbean/xbean-asm5-shaded/4.3/xbean-asm5-shaded-4.3.jar
 (142 KB at 114.8 KB/sec)
2017-04-01T02:01:14.842 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jctools/jctools-core/1.1/jctools-core-1.1.jar
 (98 KB at 79.2 KB/sec)
2017-04-01T02:01:15.691 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/it/unimi/dsi/fastutil/7.0.6/fastutil-7.0.6.jar
 (16937 KB at 8142.6 KB/sec)
2017-04-01T02:01:15.705 [INFO] 
2017-04-01T02:01:15.705 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-apex ---
2017-04-01T02:01:15.706 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/], excludes = [])
2017-04-01T02:01:15.757 [INFO] 
2017-04-01T02:01:15.757 [INFO] --- maven-enforcer-plugin:1.4.1:enforce 
(enforce) @ beam-runners-apex ---
2017-04-01T02:01:18.462 [INFO] 
2017-04-01T02:01:18.463 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-apex ---
2017-04-01T02:01:19.746 [INFO] 
2017-04-01T02:01:19.746 [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ beam-runners-apex ---
2017-04-01T02:01:19.748 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-04-01T02:01:19.748 [INFO] skip non existing resourceDirectory 

2017-04-01T02:01:19.748 [INFO] Copying 3 resources
2017-04-01T02:01:19.836 [INFO] 
2017-04-01T02:01:19.836 [INFO] --- maven-compiler-plugin:3.6.1:compile 
(default-compile) @ beam-runners-apex ---
2017-04-01T02:01:19.843 [INFO] Changes detected - recompiling the module!
2017-04-01T02:01:19.843 [INFO] Compiling 30 source files to 

2017-04-01T02:01:20.377 [WARNING] bootstrap class path not set in conjunction 
with -source 1.7
2017-04-01T02:01:20.378 [INFO] 
:
 Some input files use or override a deprecated API.
2017-04-01T02:01:20.378 [INFO] 
:
 Recompile with -Xlint:deprecation for details.
2017-04-01T02:01:20.378 [INFO] 
:
 Some input files use unchecked or unsafe operations.
2017-04-01T02:01:20.378 [INFO] 

Build failed in Jenkins: beam_PostCommit_Python_Verify #1699

[Apache Jenkins Server]

See 


Changes:

[tgroh] Separate View Evaluation and Overrides

--
[...truncated 651.07 KB...]
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsList' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_with_different_defaults 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 248, in test_as_singleton_with_different_defaults
pipeline.run()
  File 
"
 line 91, in run
result = super(TestPipeline, self).run()
  File 
"
 line 169, in run
return self.runner.run(self)
  File 
"
 line 32, in run
self.result = super(TestDataflowRunner, self).run(pipeline)
  File 
"
 line 161, in run
super(DataflowRunner, self).run(pipeline)
  File 
"
 line 122, in run
pipeline.visit(RunVisitor(self))
  File 
"
 line 192, in visit
self._root_transform().visit(visitor, self, visited)
  File 
"
 line 471, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 474, in visit
visitor.visit_transform(self)
  File 
"
 line 117, in visit_transform
self.runner.run_transform(transform_node)
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsSingleton' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_without_unique_labels 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"

[2/2] beam git commit: Separate View Evaluation and Overrides

[tgroh]

Separate View Evaluation and Overrides

This simplifies the View Evaluation by separating it from the logic of
overriding CreatePCollectionView.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/69d0b307
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/69d0b307
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/69d0b307

Branch: refs/heads/master
Commit: 69d0b3070eaee55f650b15ceda3608cc27807caf
Parents: 5f72b83
Author: Thomas Groh 
Authored: Mon Mar 27 18:03:30 2017 -0700
Committer: Thomas Groh 
Committed: Fri Mar 31 18:08:46 2017 -0700

--
 .../beam/runners/direct/DirectRunner.java   |   1 -
 .../direct/TransformEvaluatorRegistry.java  |   3 +-
 .../runners/direct/ViewEvaluatorFactory.java|  79 ++--
 .../runners/direct/ViewOverrideFactory.java | 115 +
 .../direct/ViewEvaluatorFactoryTest.java|  11 +-
 .../runners/direct/ViewOverrideFactoryTest.java | 124 +++
 6 files changed, 249 insertions(+), 84 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/69d0b307/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index 11fe3f5..bd210c3 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -35,7 +35,6 @@ import org.apache.beam.runners.core.TimerInternals.TimerData;
 import org.apache.beam.runners.core.construction.PTransformMatchers;
 import org.apache.beam.runners.direct.DirectRunner.DirectPipelineResult;
 import 
org.apache.beam.runners.direct.TestStreamEvaluatorFactory.DirectTestStreamFactory;
-import org.apache.beam.runners.direct.ViewEvaluatorFactory.ViewOverrideFactory;
 import org.apache.beam.sdk.AggregatorRetrievalException;
 import org.apache.beam.sdk.AggregatorValues;
 import org.apache.beam.sdk.Pipeline;

http://git-wip-us.apache.org/repos/asf/beam/blob/69d0b307/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java
index 5ad8709..ae7ad93 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java
@@ -30,6 +30,7 @@ import 
org.apache.beam.runners.direct.DirectGroupByKey.DirectGroupAlsoByWindow;
 import org.apache.beam.runners.direct.DirectGroupByKey.DirectGroupByKeyOnly;
 import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
 import org.apache.beam.runners.direct.ParDoMultiOverrideFactory.StatefulParDo;
+import org.apache.beam.runners.direct.ViewOverrideFactory.WriteView;
 import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.Flatten.PCollections;
@@ -54,7 +55,7 @@ class TransformEvaluatorRegistry implements 
TransformEvaluatorFactory {
 .put(ParDo.MultiOutput.class, new ParDoEvaluatorFactory<>(ctxt))
 .put(StatefulParDo.class, new 
StatefulParDoEvaluatorFactory<>(ctxt))
 .put(PCollections.class, new FlattenEvaluatorFactory(ctxt))
-.put(ViewEvaluatorFactory.WriteView.class, new 
ViewEvaluatorFactory(ctxt))
+.put(WriteView.class, new ViewEvaluatorFactory(ctxt))
 .put(Window.Assign.class, new WindowEvaluatorFactory(ctxt))
 // Runner-specific primitives used in expansion of GroupByKey
 .put(DirectGroupByKeyOnly.class, new 
GroupByKeyOnlyEvaluatorFactory(ctxt))

http://git-wip-us.apache.org/repos/asf/beam/blob/69d0b307/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java
index 9dcbf9e..dc74d3e 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ViewEvaluatorFactory.java
@@ -20,32 +20,25 @@ package org.apache.beam.runners

[GitHub] beam pull request #2347: Separate View Evaluation and Overrides

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2347


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: This closes #2347

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 5f72b83c0 -> 07499824b


This closes #2347


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/07499824
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/07499824
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/07499824

Branch: refs/heads/master
Commit: 07499824b3a97dd120372c1474b9bbe5e4817e18
Parents: 5f72b83 69d0b30
Author: Thomas Groh 
Authored: Fri Mar 31 18:08:46 2017 -0700
Committer: Thomas Groh 
Committed: Fri Mar 31 18:08:46 2017 -0700

--
 .../beam/runners/direct/DirectRunner.java   |   1 -
 .../direct/TransformEvaluatorRegistry.java  |   3 +-
 .../runners/direct/ViewEvaluatorFactory.java|  79 ++--
 .../runners/direct/ViewOverrideFactory.java | 115 +
 .../direct/ViewEvaluatorFactoryTest.java|  11 +-
 .../runners/direct/ViewOverrideFactoryTest.java | 124 +++
 6 files changed, 249 insertions(+), 84 deletions(-)
--

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #1463

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3120

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3119

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #1698

[Apache Jenkins Server]

See 


Changes:

[kirpichov] KafkaIO : Add withTopic() api that takes single topic.

--
[...truncated 661.97 KB...]
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsList' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_with_different_defaults 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 248, in test_as_singleton_with_different_defaults
pipeline.run()
  File 
"
 line 91, in run
result = super(TestPipeline, self).run()
  File 
"
 line 169, in run
return self.runner.run(self)
  File 
"
 line 32, in run
self.result = super(TestDataflowRunner, self).run(pipeline)
  File 
"
 line 161, in run
super(DataflowRunner, self).run(pipeline)
  File 
"
 line 122, in run
pipeline.visit(RunVisitor(self))
  File 
"
 line 192, in visit
self._root_transform().visit(visitor, self, visited)
  File 
"
 line 471, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 474, in visit
visitor.visit_transform(self)
  File 
"
 line 117, in visit_transform
self.runner.run_transform(transform_node)
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsSingleton' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_without_unique_labels 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #1462

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1830) add 'withTopic()' api to KafkaIO Reader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951838#comment-15951838
 ] 

ASF GitHub Bot commented on BEAM-1830:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2364


> add 'withTopic()' api to KafkaIO Reader
> ---
>
> Key: BEAM-1830
> URL: https://issues.apache.org/jira/browse/BEAM-1830
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: 0.6.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Minor
> Fix For: First stable release
>
>
> Most of instances of KafkaIO readers consume just one topic. The existing 
> method {{withTopics(List topics)}} forces the users to make a list 
> containing single topic.
> It would be simpler to add {{withTopic(String topic)}} method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2364: [BEAM-1830] KafkaIO : Add withTopic() api that take...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2364


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: KafkaIO : Add withTopic() api that takes single topic.

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master 0b8932fd3 -> 5f72b83c0


KafkaIO : Add withTopic() api that takes single topic.

Remove need for setting key coder for Writer while writing
values only. If we didn't specifiy the key coder, validation
succeeded but it failed a check while instantiating Kafka producer.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/37b0d45c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/37b0d45c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/37b0d45c

Branch: refs/heads/master
Commit: 37b0d45c76b5fb03cdf5749dee52483fa3811d5b
Parents: 0b8932f
Author: Raghu Angadi 
Authored: Wed Mar 29 08:17:25 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Mar 31 16:51:20 2017 -0700

--
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   | 11 -
 .../apache/beam/sdk/io/kafka/KafkaIOTest.java   | 24 
 2 files changed, 34 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/37b0d45c/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
--
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
index bb7d971..80b40be 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
@@ -124,7 +124,7 @@ import org.slf4j.LoggerFactory;
  *  pipeline
  *.apply(KafkaIO.read()
  *   .withBootstrapServers("broker_1:9092,broker_2:9092")
- *   .withTopics(ImmutableList.of("topic_a", "topic_b"))
+ *   .withTopic("my_topic")  // use withTopics(List) to read from 
multiple topics.
  *   // set a Coder for Key and Value
  *   .withKeyCoder(BigEndianLongCoder.of())
  *   .withValueCoder(StringUtf8Coder.of())
@@ -308,6 +308,15 @@ public class KafkaIO {
 }
 
 /**
+ * Returns a new {@link Read} that reads from the topic.
+ * See {@link UnboundedKafkaSource#generateInitialSplits(int, 
PipelineOptions)} for description
+ * of how the partitions are distributed among the splits.
+ */
+public Read withTopic(String topic) {
+  return withTopics(ImmutableList.of(topic));
+}
+
+/**
  * Returns a new {@link Read} that reads from the topics. All the 
partitions from each
  * of the topics are read.
  * See {@link UnboundedKafkaSource#generateInitialSplits(int, 
PipelineOptions)} for description

http://git-wip-us.apache.org/repos/asf/beam/blob/37b0d45c/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
--
diff --git 
a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
 
b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
index d1696d0..7e77512 100644
--- 
a/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
+++ 
b/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java
@@ -295,6 +295,30 @@ public class KafkaIOTest {
   }
 
   @Test
+  public void testUnboundedSourceWithSingleTopic() {
+// same as testUnboundedSource, but with single topic
+
+int numElements = 1000;
+String topic = "my_topic";
+
+KafkaIO.Read reader = KafkaIO.read()
+.withBootstrapServers("none")
+.withTopic("my_topic")
+.withConsumerFactoryFn(new ConsumerFactoryFn(
+ImmutableList.of(topic), 10, numElements, 
OffsetResetStrategy.EARLIEST))
+.withKeyCoder(BigEndianIntegerCoder.of())
+.withValueCoder(BigEndianLongCoder.of())
+.withMaxNumRecords(numElements);
+
+PCollection input = p
+.apply(reader.withoutMetadata())
+.apply(Values.create());
+
+addCountingAsserts(input, numElements);
+p.run();
+  }
+
+  @Test
   @Category(NeedsRunner.class)
   public void testUnboundedSourceWithExplicitPartitions() {
 int numElements = 1000;

[2/2] beam git commit: This closes #2364

[jkff]

This closes #2364


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5f72b83c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5f72b83c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5f72b83c

Branch: refs/heads/master
Commit: 5f72b83c054845ecfc0be03e27acfc24a676a936
Parents: 0b8932f 37b0d45
Author: Eugene Kirpichov 
Authored: Fri Mar 31 16:51:38 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Mar 31 16:51:38 2017 -0700

--
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   | 11 -
 .../apache/beam/sdk/io/kafka/KafkaIOTest.java   | 24 
 2 files changed, 34 insertions(+), 1 deletion(-)
--

Build failed in Jenkins: beam_PerformanceTests_Dataflow #254

[Apache Jenkins Server]

See 


Changes:

[kirpichov] Removes unused name parameter

[kirpichov] Fixes a few warnings in Window

[kirpichov] Uses AutoValue in Window

[kirpichov] Replaced static Window.blah() methods with Window.configure().blah()

[kirpichov] Replaces Window.Bound with simply Window

[robertwb] Change side inputs to be references rather than full PValues.

[altay] use custom hostname for datastore

[tgroh] Use Batch Replacement in the Apex Runner

[kirpichov] Adds requested number of splits to display data in DatastoreIO

--
[...truncated 273.48 KB...]
 * [new ref] refs/pull/928/head -> origin/pr/928/head
 * [new ref] refs/pull/929/head -> origin/pr/929/head
 * [new ref] refs/pull/93/head -> origin/pr/93/head
 * [new ref] refs/pull/930/head -> origin/pr/930/head
 * [new ref] refs/pull/930/merge -> origin/pr/930/merge
 * [new ref] refs/pull/931/head -> origin/pr/931/head
 * [new ref] refs/pull/931/merge -> origin/pr/931/merge
 * [new ref] refs/pull/932/head -> origin/pr/932/head
 * [new ref] refs/pull/932/merge -> origin/pr/932/merge
 * [new ref] refs/pull/933/head -> origin/pr/933/head
 * [new ref] refs/pull/933/merge -> origin/pr/933/merge
 * [new ref] refs/pull/934/head -> origin/pr/934/head
 * [new ref] refs/pull/934/merge -> origin/pr/934/merge
 * [new ref] refs/pull/935/head -> origin/pr/935/head
 * [new ref] refs/pull/936/head -> origin/pr/936/head
 * [new ref] refs/pull/936/merge -> origin/pr/936/merge
 * [new ref] refs/pull/937/head -> origin/pr/937/head
 * [new ref] refs/pull/937/merge -> origin/pr/937/merge
 * [new ref] refs/pull/938/head -> origin/pr/938/head
 * [new ref] refs/pull/939/head -> origin/pr/939/head
 * [new ref] refs/pull/94/head -> origin/pr/94/head
 * [new ref] refs/pull/940/head -> origin/pr/940/head
 * [new ref] refs/pull/940/merge -> origin/pr/940/merge
 * [new ref] refs/pull/941/head -> origin/pr/941/head
 * [new ref] refs/pull/941/merge -> origin/pr/941/merge
 * [new ref] refs/pull/942/head -> origin/pr/942/head
 * [new ref] refs/pull/942/merge -> origin/pr/942/merge
 * [new ref] refs/pull/943/head -> origin/pr/943/head
 * [new ref] refs/pull/943/merge -> origin/pr/943/merge
 * [new ref] refs/pull/944/head -> origin/pr/944/head
 * [new ref] refs/pull/945/head -> origin/pr/945/head
 * [new ref] refs/pull/945/merge -> origin/pr/945/merge
 * [new ref] refs/pull/946/head -> origin/pr/946/head
 * [new ref] refs/pull/946/merge -> origin/pr/946/merge
 * [new ref] refs/pull/947/head -> origin/pr/947/head
 * [new ref] refs/pull/947/merge -> origin/pr/947/merge
 * [new ref] refs/pull/948/head -> origin/pr/948/head
 * [new ref] refs/pull/948/merge -> origin/pr/948/merge
 * [new ref] refs/pull/949/head -> origin/pr/949/head
 * [new ref] refs/pull/949/merge -> origin/pr/949/merge
 * [new ref] refs/pull/95/head -> origin/pr/95/head
 * [new ref] refs/pull/95/merge -> origin/pr/95/merge
 * [new ref] refs/pull/950/head -> origin/pr/950/head
 * [new ref] refs/pull/951/head -> origin/pr/951/head
 * [new ref] refs/pull/951/merge -> origin/pr/951/merge
 * [new ref] refs/pull/952/head -> origin/pr/952/head
 * [new ref] refs/pull/952/merge -> origin/pr/952/merge
 * [new ref] refs/pull/953/head -> origin/pr/953/head
 * [new ref] refs/pull/954/head -> origin/pr/954/head
 * [new ref] refs/pull/954/merge -> origin/pr/954/merge
 * [new ref] refs/pull/955/head -> origin/pr/955/head
 * [new ref] refs/pull/955/merge -> origin/pr/955/merge
 * [new ref] refs/pull/956/head -> origin/pr/956/head
 * [new ref] refs/pull/957/head -> origin/pr/957/head
 * [new ref] refs/pull/958/head -> origin/pr/958/head
 * [new ref] refs/pull/959/head -> origin/pr/959/head
 * [new ref] refs/pull/959/merge -> origin/pr/959/merge
 * [new ref] refs/pull/96/head -> origin/pr/96/head
 * [new ref] refs/pull/96/merge -> origin/pr/96/merge
 * [new ref] refs/pull/960/head -> origin/pr/960/head
 * [new ref] refs/pull/960/merge -> origin/pr/960/merge
 * [new ref] refs/pull/961/head -> origin/pr/961/head
 * [new ref] refs/pull/962/head -> origin/pr/962/head
 * [new ref] refs/pull/962/merge -> origin/pr/962/merge
 * [new ref] refs/pull/963/head -> origin/pr/963/head
 * [new ref] refs/pull/963/merge -> origin/pr/963/merge
 * [new ref] refs/pull/964/head -> origin/pr/964/head
 * [new ref] refs/pull/965/head -> origin/pr/965/head
 * [new ref] refs/pull/965/merge -> origin/pr/9

Build failed in Jenkins: beam_PostCommit_Python_Verify #1697

[Apache Jenkins Server]

See 


Changes:

[kirpichov] Adds requested number of splits to display data in DatastoreIO

--
[...truncated 659.25 KB...]
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsList' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_with_different_defaults 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 248, in test_as_singleton_with_different_defaults
pipeline.run()
  File 
"
 line 91, in run
result = super(TestPipeline, self).run()
  File 
"
 line 169, in run
return self.runner.run(self)
  File 
"
 line 32, in run
self.result = super(TestDataflowRunner, self).run(pipeline)
  File 
"
 line 161, in run
super(DataflowRunner, self).run(pipeline)
  File 
"
 line 122, in run
pipeline.visit(RunVisitor(self))
  File 
"
 line 192, in visit
self._root_transform().visit(visitor, self, visited)
  File 
"
 line 471, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 474, in visit
visitor.visit_transform(self)
  File 
"
 line 117, in visit_transform
self.runner.run_transform(transform_node)
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsSingleton' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_without_unique_labels 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"

[GitHub] beam pull request #2385: Add request number of splits to display data in Dat...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2385


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2385

[jkff]

This closes #2385


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0b8932fd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0b8932fd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0b8932fd

Branch: refs/heads/master
Commit: 0b8932fd37c51144cc25313643a0ab098c0d4b21
Parents: 16486e2 3df8dcc
Author: Eugene Kirpichov 
Authored: Fri Mar 31 16:41:32 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Mar 31 16:41:32 2017 -0700

--
 .../beam/sdk/io/gcp/datastore/DatastoreV1.java  |  4 +++
 .../sdk/io/gcp/datastore/DatastoreV1Test.java   | 28 +---
 2 files changed, 16 insertions(+), 16 deletions(-)
--

[1/2] beam git commit: Adds requested number of splits to display data in DatastoreIO

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master 16486e2ad -> 0b8932fd3


Adds requested number of splits to display data in DatastoreIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3df8dccd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3df8dccd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3df8dccd

Branch: refs/heads/master
Commit: 3df8dccdbe8bd65c45dc0f73f8987946e03c5b93
Parents: 16486e2
Author: Vikas Kedigehalli 
Authored: Thu Mar 30 17:31:18 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Mar 31 16:41:22 2017 -0700

--
 .../beam/sdk/io/gcp/datastore/DatastoreV1.java  |  4 +++
 .../sdk/io/gcp/datastore/DatastoreV1Test.java   | 28 +---
 2 files changed, 16 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3df8dccd/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
index 73ac8df..98787fa 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
@@ -793,6 +793,10 @@ public class DatastoreV1 {
   public void populateDisplayData(DisplayData.Builder builder) {
 super.populateDisplayData(builder);
 builder.include("options", options);
+if (numSplits > 0) {
+  builder.add(DisplayData.item("numQuerySplits", numSplits)
+  .withLabel("Requested number of Query splits"));
+}
   }
 }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/3df8dccd/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
index af27926..5d3b278 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1Test.java
@@ -272,23 +272,19 @@ public class DatastoreV1Test {
   @Test
   public void testSourcePrimitiveDisplayData() {
 DisplayDataEvaluator evaluator = DisplayDataEvaluator.create();
-PTransform> read = 
DatastoreIO.v1().read().withProjectId(
-"myProject").withQuery(Query.newBuilder().build());
-
+int numSplits = 98;
+PTransform> read =
+DatastoreIO.v1().read()
+.withProjectId(PROJECT_ID)
+.withQuery(Query.newBuilder().build())
+.withNumQuerySplits(numSplits);
+
+String assertMessage = "DatastoreIO read should include the '%s' in its 
primitive display data";
 Set displayData = 
evaluator.displayDataForPrimitiveSourceTransforms(read);
-assertThat("DatastoreIO read should include the project in its primitive 
display data",
-displayData, hasItem(hasDisplayItem("projectId")));
-  }
-
-  @Test
-  public void testSourcePrimitiveDisplayDataWithGqlQuery() {
-DisplayDataEvaluator evaluator = DisplayDataEvaluator.create();
-PTransform> read = 
DatastoreIO.v1().read().withProjectId(
-"myProject").withLiteralGqlQuery(GQL_QUERY);
-
-Set displayData = 
evaluator.displayDataForPrimitiveSourceTransforms(read);
-assertThat("DatastoreIO read should include the project in its primitive 
display data",
-displayData, hasItem(hasDisplayItem("projectId")));
+assertThat(String.format(assertMessage, "project id"),
+displayData, hasItem(hasDisplayItem("projectId", PROJECT_ID)));
+assertThat(String.format(assertMessage, "number of query splits"),
+displayData, hasItem(hasDisplayItem("numQuerySplits", numSplits)));
   }
 
   @Test

[jira] [Commented] (BEAM-1843) 'PDone' object has no attribute 'to_runner_api'

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951804#comment-15951804
 ] 

ASF GitHub Bot commented on BEAM-1843:
--

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/2396

[BEAM-1843] Ensure transforms are picklable before materializing to protos.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam native-write

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2396.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2396


commit c6abd8ea8260ff51614d9b309975975adc4cf32e
Author: Robert Bradshaw 
Date:   2017-03-31T23:30:56Z

Ensure transforms are picklable before materializing to protos.

See BEAM-1843.




> 'PDone' object has no attribute 'to_runner_api'
> ---
>
> Key: BEAM-1843
> URL: https://issues.apache.org/jira/browse/BEAM-1843
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Robert Bradshaw
>
> Post commit failure with 
> (https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1676/consoleFull):
> ...
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 512, in 
> for tag, out in self.outputs.items()},
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/pipeline_context.py",
>  line 52, in get_id
> self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
> AttributeError: 'PDone' object has no attribute 'to_runner_api'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2396: [BEAM-1843] Ensure transforms are picklable before ...

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/2396

[BEAM-1843] Ensure transforms are picklable before materializing to protos.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam native-write

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2396.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2396


commit c6abd8ea8260ff51614d9b309975975adc4cf32e
Author: Robert Bradshaw 
Date:   2017-03-31T23:30:56Z

Ensure transforms are picklable before materializing to protos.

See BEAM-1843.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1853) AttributeError: 'AsIter' object has no attribute 'producer' (post commit failure)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951746#comment-15951746
 ] 

ASF GitHub Bot commented on BEAM-1853:
--

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/2395

[BEAM-1853] Fix side inputs on dataflow runner.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam side-inputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2395.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2395


commit 3aa157a9be4f5bf8e390fcb3acf37d8bdb250954
Author: Robert Bradshaw 
Date:   2017-03-31T21:57:44Z

Fix side inputs on dataflow runner.




> AttributeError: 'AsIter' object has no attribute 'producer' (post commit 
> failure)
> -
>
> Key: BEAM-1853
> URL: https://issues.apache.org/jira/browse/BEAM-1853
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Robert Bradshaw
>
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/1694/console
> One of the failures (there are other related errors):
> ==
> ERROR: test_iterable_side_input 
> (apache_beam.transforms.sideinputs_test.SideInputsTest)
> --
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/transforms/sideinputs_test.py",
>  line 179, in test_iterable_side_input
> pipeline.run()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/test_pipeline.py",
>  line 91, in run
> result = super(TestPipeline, self).run()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 169, in run
> return self.runner.run(self)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
>  line 32, in run
> self.result = super(TestDataflowRunner, self).run(pipeline)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
>  line 161, in run
> super(DataflowRunner, self).run(pipeline)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
>  line 122, in run
> pipeline.visit(RunVisitor(self))
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 192, in visit
> self._root_transform().visit(visitor, self, visited)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 471, in visit
> part.visit(visitor, pipeline, visited)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 474, in visit
> visitor.visit_transform(self)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
>  line 117, in visit_transform
> self.runner.run_transform(transform_node)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
>  line 192, in run_transform
> return m(transform_node)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
>  line 378, in run_ParDo
> si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
>  line 266, in get_pvalue
> self._ens

[GitHub] beam pull request #2395: [BEAM-1853] Fix side inputs on dataflow runner.

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/2395

[BEAM-1853] Fix side inputs on dataflow runner.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam side-inputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2395.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2395


commit 3aa157a9be4f5bf8e390fcb3acf37d8bdb250954
Author: Robert Bradshaw 
Date:   2017-03-31T21:57:44Z

Fix side inputs on dataflow runner.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2394: [Proof-Of-Concept] Remove PDone

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/2394

[Proof-Of-Concept] Remove PDone

@chamikaramj @robertwb What do you think about this?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam post

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2394.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2394


commit c225716f7f1618b905396115f2e691aaa697ea04
Author: Ahmet Altay 
Date:   2017-03-31T22:33:12Z

Remove PDone




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2393: Update integration tests in post commits

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/2393

Update integration tests in post commits

R: @markflyhigh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam it

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2393.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2393


commit ff3330fb69588e95612696eedeff117a4f604cd2
Author: Ahmet Altay 
Date:   2017-03-31T22:11:43Z

Update integration tests in post commits




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3118

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #3117

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #1461

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #1696

[Apache Jenkins Server]

See 


Changes:

[tgroh] Use Batch Replacement in the Apex Runner

--
[...truncated 660.10 KB...]
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsList' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_with_different_defaults 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 248, in test_as_singleton_with_different_defaults
pipeline.run()
  File 
"
 line 91, in run
result = super(TestPipeline, self).run()
  File 
"
 line 169, in run
return self.runner.run(self)
  File 
"
 line 32, in run
self.result = super(TestDataflowRunner, self).run(pipeline)
  File 
"
 line 161, in run
super(DataflowRunner, self).run(pipeline)
  File 
"
 line 122, in run
pipeline.visit(RunVisitor(self))
  File 
"
 line 192, in visit
self._root_transform().visit(visitor, self, visited)
  File 
"
 line 471, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 474, in visit
visitor.visit_transform(self)
  File 
"
 line 117, in visit_transform
self.runner.run_transform(transform_node)
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsSingleton' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_without_unique_labels 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #1460

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #1695

[Apache Jenkins Server]

See 


--
[...truncated 659.63 KB...]
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsList' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_with_different_defaults 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 248, in test_as_singleton_with_different_defaults
pipeline.run()
  File 
"
 line 91, in run
result = super(TestPipeline, self).run()
  File 
"
 line 169, in run
return self.runner.run(self)
  File 
"
 line 32, in run
self.result = super(TestDataflowRunner, self).run(pipeline)
  File 
"
 line 161, in run
super(DataflowRunner, self).run(pipeline)
  File 
"
 line 122, in run
pipeline.visit(RunVisitor(self))
  File 
"
 line 192, in visit
self._root_transform().visit(visitor, self, visited)
  File 
"
 line 471, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 474, in visit
visitor.visit_transform(self)
  File 
"
 line 117, in visit_transform
self.runner.run_transform(transform_node)
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsSingleton' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_without_unique_labels 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 226, in test_as_si

[jira] [Created] (BEAM-1853) AttributeError: 'AsIter' object has no attribute 'producer' (post commit failure)

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-1853:
-

 Summary: AttributeError: 'AsIter' object has no attribute 
'producer' (post commit failure)
 Key: BEAM-1853
 URL: https://issues.apache.org/jira/browse/BEAM-1853
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Robert Bradshaw


https://builds.apache.org/job/beam_PostCommit_Python_Verify/1694/console

One of the failures (there are other related errors):
==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/transforms/sideinputs_test.py",
 line 179, in test_iterable_side_input
pipeline.run()
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/test_pipeline.py",
 line 91, in run
result = super(TestPipeline, self).run()
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
 line 169, in run
return self.runner.run(self)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
 line 32, in run
self.result = super(TestDataflowRunner, self).run(pipeline)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 161, in run
super(DataflowRunner, self).run(pipeline)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
 line 122, in run
pipeline.visit(RunVisitor(self))
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
 line 192, in visit
self._root_transform().visit(visitor, self, visited)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
 line 471, in visit
part.visit(visitor, pipeline, visited)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
 line 474, in visit
visitor.visit_transform(self)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
 line 117, in visit_transform
self.runner.run_transform(transform_node)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
 line 192, in run_transform
return m(transform_node)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/runner.py",
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsIter' object has no attribute 'producer'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1852) dead links on beam website

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-1852:
--

 Summary: dead links on beam website
 Key: BEAM-1852
 URL: https://issues.apache.org/jira/browse/BEAM-1852
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Stephen Sisk
Assignee: Davor Bonaci


Output from rake: 

Checking 270 external links...
Ran on 98 files!


- ./content/subdir/blog/2017/02/01/graduation-media-recap.html
  *  External link 
https://globenewswire.com/news-release/2017/01/10/904692/0/en/The-Apache-Software-Foundation-Announces-Apache-Beam-as-a-Top-Level-Project.html
 failed: response code 0 means something's wrong.
 It's possible libcurl couldn't connect to the server or perhaps 
the request timed out.
 Sometimes, making too many requests at once also breaks things.
 Either way, the return message (if any) from the server is: Peer 
certificate cannot be authenticated with given CA certificates
[stephen note: I could not confirm this manually, but that error seems.. 
interesting]


- ./content/subdir/contribute/contribution-guide/index.html
  *  External link https://www.apache.org/licenses/icla.txt failed: 404 No error
[stephen note: manually confirmed this]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1851) Sample.fixedSizedGlobally documentation should include single worker memory constraint

[Mark Shalda (JIRA)]

Mark Shalda created BEAM-1851:
-

 Summary: Sample.fixedSizedGlobally documentation should include 
single worker memory constraint
 Key: BEAM-1851
 URL: https://issues.apache.org/jira/browse/BEAM-1851
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Affects Versions: 0.6.0
Reporter: Mark Shalda
Assignee: Davor Bonaci
Priority: Trivial


Sample.any mentions in the documentation:

" All of the elements of the output {@code PCollection} should fit into main 
memory of a single worker machine.  This operation does not run in parallel."

This should also be mentioned for fixedSizedGlobally.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1579) Runners should verify that PT overrides converged

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951650#comment-15951650
 ] 

ASF GitHub Bot commented on BEAM-1579:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2381


> Runners should verify that PT overrides converged
> -
>
> Key: BEAM-1579
> URL: https://issues.apache.org/jira/browse/BEAM-1579
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-dataflow, runner-direct
>Reporter: Eugene Kirpichov
>Assignee: Thomas Groh
>
> PT overrides are applied in order, see 
> https://issues.apache.org/jira/browse/BEAM-1578 .
> To make sure that the order is correct and avoid confusing errors, after 
> applying the overrides in order, we should verify that the pipeline has 
> converged, i.e. the overrides no longer match.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2381: [BEAM-1579] Use Batch Replacement in the Apex Runne...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2381


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: This closes #2381

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master c81f83bc7 -> 16486e2ad


This closes #2381


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/16486e2a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/16486e2a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/16486e2a

Branch: refs/heads/master
Commit: 16486e2ad45a1e511d7533a359d176ab3617199a
Parents: c81f83b d89e9d7
Author: Thomas Groh 
Authored: Fri Mar 31 14:08:55 2017 -0700
Committer: Thomas Groh 
Committed: Fri Mar 31 14:08:55 2017 -0700

--
 .../apache/beam/runners/apex/ApexRunner.java| 40 ++--
 1 file changed, 20 insertions(+), 20 deletions(-)
--

[2/2] beam git commit: Use Batch Replacement in the Apex Runner

[tgroh]

Use Batch Replacement in the Apex Runner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d89e9d7d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d89e9d7d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d89e9d7d

Branch: refs/heads/master
Commit: d89e9d7d3a3ea952e6eb0784f717203460afe90f
Parents: c81f83b
Author: Thomas Groh 
Authored: Thu Mar 30 15:55:28 2017 -0700
Committer: Thomas Groh 
Committed: Fri Mar 31 14:08:55 2017 -0700

--
 .../apache/beam/runners/apex/ApexRunner.java| 40 ++--
 1 file changed, 20 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d89e9d7d/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
index dfc8f63..d23fc14 100644
--- a/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
+++ b/runners/apex/src/main/java/org/apache/beam/runners/apex/ApexRunner.java
@@ -22,7 +22,7 @@ import com.datatorrent.api.Context.DAGContext;
 import com.datatorrent.api.DAG;
 import com.datatorrent.api.StreamingApplication;
 import com.google.common.base.Throwables;
-import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableList;
 import java.io.File;
 import java.io.IOException;
 import java.io.InputStream;
@@ -31,7 +31,6 @@ import java.net.URISyntaxException;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
-import java.util.Map;
 import java.util.Properties;
 import java.util.concurrent.atomic.AtomicReference;
 import org.apache.apex.api.EmbeddedAppLauncher;
@@ -48,9 +47,7 @@ import org.apache.beam.sdk.coders.CoderRegistry;
 import org.apache.beam.sdk.coders.ListCoder;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsValidator;
-import org.apache.beam.sdk.runners.PTransformMatcher;
 import org.apache.beam.sdk.runners.PTransformOverride;
-import org.apache.beam.sdk.runners.PTransformOverrideFactory;
 import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.Combine;
 import org.apache.beam.sdk.transforms.Combine.GloballyAsSingletonView;
@@ -96,27 +93,30 @@ public class ApexRunner extends 
PipelineRunner {
 return new ApexRunner(apexPipelineOptions);
   }
 
-  private Map getOverrides() {
-return ImmutableMap.builder()
-.put(PTransformMatchers.classEqualTo(Create.Values.class), new 
PrimitiveCreate.Factory())
-.put(
-PTransformMatchers.classEqualTo(View.AsSingleton.class),
-new StreamingViewAsSingleton.Factory())
-.put(
-PTransformMatchers.classEqualTo(View.AsIterable.class),
-new StreamingViewAsIterable.Factory())
-.put(
-
PTransformMatchers.classEqualTo(Combine.GloballyAsSingletonView.class),
-new StreamingCombineGloballyAsSingletonView.Factory())
+  private List getOverrides() {
+return ImmutableList.builder()
+.add(
+PTransformOverride.of(
+PTransformMatchers.classEqualTo(Create.Values.class),
+new PrimitiveCreate.Factory()))
+.add(
+PTransformOverride.of(
+PTransformMatchers.classEqualTo(View.AsSingleton.class),
+new StreamingViewAsSingleton.Factory()))
+.add(
+PTransformOverride.of(
+PTransformMatchers.classEqualTo(View.AsIterable.class),
+new StreamingViewAsIterable.Factory()))
+.add(
+PTransformOverride.of(
+
PTransformMatchers.classEqualTo(Combine.GloballyAsSingletonView.class),
+new StreamingCombineGloballyAsSingletonView.Factory()))
 .build();
   }
 
   @Override
   public ApexRunnerResult run(final Pipeline pipeline) {
-for (Map.Entry override :
-getOverrides().entrySet()) {
-  pipeline.replace(PTransformOverride.of(override.getKey(), 
override.getValue()));
-}
+pipeline.replaceAll(getOverrides());
 
 final ApexPipelineTranslator translator = new 
ApexPipelineTranslator(options);
 final AtomicReference apexDAG = new AtomicReference<>();

Build failed in Jenkins: beam_PostCommit_Python_Verify #1694

[Apache Jenkins Server]

See 


Changes:

[altay] use custom hostname for datastore

--
[...truncated 658.43 KB...]
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsList' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_with_different_defaults 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 248, in test_as_singleton_with_different_defaults
pipeline.run()
  File 
"
 line 91, in run
result = super(TestPipeline, self).run()
  File 
"
 line 169, in run
return self.runner.run(self)
  File 
"
 line 32, in run
self.result = super(TestDataflowRunner, self).run(pipeline)
  File 
"
 line 161, in run
super(DataflowRunner, self).run(pipeline)
  File 
"
 line 122, in run
pipeline.visit(RunVisitor(self))
  File 
"
 line 192, in visit
self._root_transform().visit(visitor, self, visited)
  File 
"
 line 471, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 474, in visit
visitor.visit_transform(self)
  File 
"
 line 117, in visit_transform
self.runner.run_transform(transform_node)
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsSingleton' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_without_unique_labels 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"

[GitHub] beam pull request #2367: use custom hostname for datastore

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2367


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: use custom hostname for datastore

[altay]

Repository: beam
Updated Branches:
  refs/heads/master affb926cc -> c81f83bc7


use custom hostname for datastore


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4a8547cd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4a8547cd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4a8547cd

Branch: refs/heads/master
Commit: 4a8547cd0b72d7ce92328002dc814034ce1502fa
Parents: affb926
Author: Ahmet Altay 
Authored: Wed Mar 29 18:00:16 2017 -0700
Committer: Ahmet Altay 
Committed: Fri Mar 31 13:40:51 2017 -0700

--
 sdks/python/apache_beam/io/gcp/datastore/v1/helper.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4a8547cd/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
--
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py 
b/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
index 45c794f..b1ef9af 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
@@ -106,7 +106,7 @@ def str_compare(s1, s2):
 def get_datastore(project):
   """Returns a Cloud Datastore client."""
   credentials = auth.get_service_credentials()
-  return Datastore(project, credentials)
+  return Datastore(project, credentials, host='batch-datastore.googleapis.com')
 
 
 def make_request(project, namespace, query):

[2/2] beam git commit: This closes #2367

[altay]

This closes #2367


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c81f83bc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c81f83bc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c81f83bc

Branch: refs/heads/master
Commit: c81f83bc73388b432c39e5ba7c546eb7d2de3030
Parents: affb926 4a8547c
Author: Ahmet Altay 
Authored: Fri Mar 31 13:40:58 2017 -0700
Committer: Ahmet Altay 
Committed: Fri Mar 31 13:40:58 2017 -0700

--
 sdks/python/apache_beam/io/gcp/datastore/v1/helper.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3116

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-1850) Improve interplay between PusbackSideInputRunner and GroupAlsoByWindowViaWindowSetDoFn

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned BEAM-1850:
--

Assignee: (was: Kenneth Knowles)

> Improve interplay between PusbackSideInputRunner and 
> GroupAlsoByWindowViaWindowSetDoFn
> --
>
> Key: BEAM-1850
> URL: https://issues.apache.org/jira/browse/BEAM-1850
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core, runner-flink
>Reporter: Aljoscha Krettek
>
> This originated from a discussion on a PR: 
> https://github.com/apache/beam/pull/2235
> {{GroupAlsoByWindowViaWindowSetDoFn}}/{{GroupAlsoByWindowViaWindowSetNewDoFn}}
>  and {{PushbackSideInputDoFnRunner}} don't work well together and we manually 
> need to explode windows in 
> {{FlinkStreamingTransformTranslators.ToKeyedWorkItem}} because of this:
>  - {{GroupAlsoByWindowViaWindowSetDoFn}} is a {{DoFn InputT>, KV>}} so you have to push in {{KeyedWorkItem}}. These 
> themselves contain {{WindowedValue}} (or timers).
>  - For executing a {{DoFn}} we use a {{DoFnRunner}}. For our problem the 
> interesting case is using a {{PushbackSideInputDoFnRunner}}. The interesting 
> method is {{processElementInReadyWindows(WindowedValue elem)}} where 
> {{InputT}} is the input type of the {{DoFn}} which, for the windowing case, 
> is {{KeyedWorkItem}} (from above). The actual expanded type 
> signature is thus 
> {{processElementInReadyWindows(WindowedValue> 
> elem)}} where the keyed work items again contain {{WindowedValues}} (again, 
> from above).
> I think the {{PushbackSideInputDoFnRunner}} was not initially meant for 
> executing {{GroupAlsoByWindowViaWindowSetDoFns}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1850) Improve interplay between PusbackSideInputRunner and GroupAlsoByWindowViaWindowSetDoFn

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created BEAM-1850:
--

 Summary: Improve interplay between PusbackSideInputRunner and 
GroupAlsoByWindowViaWindowSetDoFn
 Key: BEAM-1850
 URL: https://issues.apache.org/jira/browse/BEAM-1850
 Project: Beam
  Issue Type: Improvement
  Components: runner-core, runner-flink
Reporter: Aljoscha Krettek
Assignee: Kenneth Knowles


This originated from a discussion on a PR: 
https://github.com/apache/beam/pull/2235

{{GroupAlsoByWindowViaWindowSetDoFn}}/{{GroupAlsoByWindowViaWindowSetNewDoFn}} 
and {{PushbackSideInputDoFnRunner}} don't work well together and we manually 
need to explode windows in 
{{FlinkStreamingTransformTranslators.ToKeyedWorkItem}} because of this:

 - {{GroupAlsoByWindowViaWindowSetDoFn}} is a {{DoFn, 
KV>}} so you have to push in {{KeyedWorkItem}}. These themselves 
contain {{WindowedValue}} (or timers).
 - For executing a {{DoFn}} we use a {{DoFnRunner}}. For our problem the 
interesting case is using a {{PushbackSideInputDoFnRunner}}. The interesting 
method is {{processElementInReadyWindows(WindowedValue elem)}} where 
{{InputT}} is the input type of the {{DoFn}} which, for the windowing case, is 
{{KeyedWorkItem}} (from above). The actual expanded type signature 
is thus {{processElementInReadyWindows(WindowedValue> 
elem)}} where the keyed work items again contain {{WindowedValues}} (again, 
from above).
I think the {{PushbackSideInputDoFnRunner}} was not initially meant for 
executing {{GroupAlsoByWindowViaWindowSetDoFns}}.







--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #1459

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #1693

[Apache Jenkins Server]

See 


Changes:

[robertwb] Change side inputs to be references rather than full PValues.

--
[...truncated 652.91 KB...]
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsList' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_with_different_defaults 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 248, in test_as_singleton_with_different_defaults
pipeline.run()
  File 
"
 line 91, in run
result = super(TestPipeline, self).run()
  File 
"
 line 169, in run
return self.runner.run(self)
  File 
"
 line 32, in run
self.result = super(TestDataflowRunner, self).run(pipeline)
  File 
"
 line 161, in run
super(DataflowRunner, self).run(pipeline)
  File 
"
 line 122, in run
pipeline.visit(RunVisitor(self))
  File 
"
 line 192, in visit
self._root_transform().visit(visitor, self, visited)
  File 
"
 line 471, in visit
part.visit(visitor, pipeline, visited)
  File 
"
 line 474, in visit
visitor.visit_transform(self)
  File 
"
 line 117, in visit_transform
self.runner.run_transform(transform_node)
  File 
"
 line 192, in run_transform
return m(transform_node)
  File 
"
 line 378, in run_ParDo
si_labels[side_pval] = self._cache.get_pvalue(side_pval).step_name
  File 
"
 line 266, in get_pvalue
self._ensure_pvalue_has_real_producer(pvalue)
  File 
"
 line 239, in _ensure_pvalue_has_real_producer
real_producer = pvalue.producer
AttributeError: 'AsSingleton' object has no attribute 'producer'
 >> begin captured logging << 
root: DEBUG: PValue computed by main input (tag None): refcount: 1 => 0
root: ERROR: Error while visiting Map()
- >> end captured logging << -

==
ERROR: test_as_singleton_without_unique_labels 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"

[GitHub] beam pull request #2388: Change side inputs to be references rather than ful...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2388


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Closes #2388

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 132d3c5f6 -> affb926cc


Closes #2388


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/affb926c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/affb926c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/affb926c

Branch: refs/heads/master
Commit: affb926ccd9302734805d8c0db418d006ee37672
Parents: 132d3c5 207de81
Author: Robert Bradshaw 
Authored: Fri Mar 31 12:11:00 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Mar 31 12:11:00 2017 -0700

--
 sdks/python/apache_beam/pipeline.py |  12 +-
 sdks/python/apache_beam/pvalue.py   | 258 +++
 sdks/python/apache_beam/pvalue_test.py  |  33 ---
 .../runners/dataflow/dataflow_runner.py |  29 ++-
 .../runners/direct/bundle_factory.py|   3 +-
 .../consumer_tracking_pipeline_visitor.py   |  11 +-
 .../consumer_tracking_pipeline_visitor_test.py  |   4 +-
 .../runners/direct/evaluation_context.py|  60 +++--
 .../apache_beam/runners/direct/executor.py  |   7 +-
 .../runners/direct/transform_evaluator.py   |  51 +---
 sdks/python/apache_beam/transforms/core.py  |   2 +-
 .../python/apache_beam/transforms/ptransform.py |   4 +-
 .../python/apache_beam/transforms/sideinputs.py | 132 --
 .../apache_beam/transforms/sideinputs_test.py   |  91 +++
 14 files changed, 198 insertions(+), 499 deletions(-)
--

[2/2] beam git commit: Change side inputs to be references rather than full PValues.

[robertwb]

Change side inputs to be references rather than full PValues.

This is more consistent with the Runner API's structure.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/207de81b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/207de81b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/207de81b

Branch: refs/heads/master
Commit: 207de81bca4c3761cf663d32f9b95a022ef97165
Parents: 132d3c5
Author: Robert Bradshaw 
Authored: Thu Mar 30 08:20:21 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Mar 31 12:11:00 2017 -0700

--
 sdks/python/apache_beam/pipeline.py |  12 +-
 sdks/python/apache_beam/pvalue.py   | 258 +++
 sdks/python/apache_beam/pvalue_test.py  |  33 ---
 .../runners/dataflow/dataflow_runner.py |  29 ++-
 .../runners/direct/bundle_factory.py|   3 +-
 .../consumer_tracking_pipeline_visitor.py   |  11 +-
 .../consumer_tracking_pipeline_visitor_test.py  |   4 +-
 .../runners/direct/evaluation_context.py|  60 +++--
 .../apache_beam/runners/direct/executor.py  |   7 +-
 .../runners/direct/transform_evaluator.py   |  51 +---
 sdks/python/apache_beam/transforms/core.py  |   2 +-
 .../python/apache_beam/transforms/ptransform.py |   4 +-
 .../python/apache_beam/transforms/sideinputs.py | 132 --
 .../apache_beam/transforms/sideinputs_test.py   |  91 +++
 14 files changed, 198 insertions(+), 499 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/207de81b/sdks/python/apache_beam/pipeline.py
--
diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index be2a79d..ee5904b 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -142,9 +142,6 @@ class Pipeline(object):
 # If a transform is applied and the full label is already in the set
 # then the transform will have to be cloned with a new label.
 self.applied_labels = set()
-# Store cache of views created from PCollections.  For reference, see
-# pvalue._cache_view().
-self._view_cache = {}
 
   def _current_transform(self):
 """Returns the transform currently on the top of the stack."""
@@ -271,8 +268,8 @@ class Pipeline(object):
 result.producer = current
   # TODO(robertwb): Multi-input, multi-output inference.
   # TODO(robertwb): Ideally we'd do intersection here.
-  if (type_options is not None and type_options.pipeline_type_check and
-  isinstance(result, (pvalue.PCollection, pvalue.PCollectionView))
+  if (type_options is not None and type_options.pipeline_type_check
+  and isinstance(result, pvalue.PCollection)
   and not result.element_type):
 input_element_type = (
 inputs[0].element_type
@@ -416,7 +413,7 @@ class AppliedPTransform(object):
 if not isinstance(main_input, pvalue.PBegin):
   real_producer(main_input).refcounts[main_input.tag] += 1
   for side_input in self.side_inputs:
-real_producer(side_input).refcounts[side_input.tag] += 1
+real_producer(side_input.pvalue).refcounts[side_input.pvalue.tag] += 1
 
   def add_output(self, output, tag=None):
 if isinstance(output, pvalue.DoOutputsTuple):
@@ -456,7 +453,8 @@ class AppliedPTransform(object):
 
 # Visit side inputs.
 for pval in self.side_inputs:
-  if isinstance(pval, pvalue.PCollectionView) and pval not in visited:
+  if isinstance(pval, pvalue.AsSideInput) and pval.pvalue not in visited:
+pval = pval.pvalue  # Unpack marker-object-wrapped pvalue.
 assert pval.producer is not None
 pval.producer.visit(visitor, pipeline, visited)
 # The value should be visited now since we visit outputs too.

http://git-wip-us.apache.org/repos/asf/beam/blob/207de81b/sdks/python/apache_beam/pvalue.py
--
diff --git a/sdks/python/apache_beam/pvalue.py 
b/sdks/python/apache_beam/pvalue.py
index 4114b3f..bfe1745 100644
--- a/sdks/python/apache_beam/pvalue.py
+++ b/sdks/python/apache_beam/pvalue.py
@@ -26,9 +26,10 @@ produced when the pipeline gets executed.
 
 from __future__ import absolute_import
 
-import collections
 import itertools
 
+from apache_beam import typehints
+
 
 class PValue(object):
   """Base class for PCollection.
@@ -250,20 +251,22 @@ class SideOutputValue(object):
 self.value = value
 
 
-class PCollectionView(PValue):
-  """An immutable view of a PCollection that can be used as a side input."""
+class AsSideInput(object):
+  """Marker specifying that a PCollection will be used as a side input.
 
-  def __init__(self, pipeline, window_mapping_fn):
-"""Initializes a

Build failed in Jenkins: beam_PostCommit_Python_Verify #1692

[Apache Jenkins Server]

See 


Changes:

[kirpichov] Removes unused name parameter

[kirpichov] Fixes a few warnings in Window

[kirpichov] Uses AutoValue in Window

[kirpichov] Replaced static Window.blah() methods with Window.configure().blah()

[kirpichov] Replaces Window.Bound with simply Window

--
[...truncated 665.74 KB...]
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:132:
 UserWarning: Using fallback coder for typehint: List[int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:132:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-34.3.3.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-2.0.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/appdirs-1.4.3.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/pyparsing-2.2.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging 
appdirs pyparsing
test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:132:
 UserWarning: Using fallback coder for typehint: List[int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:132:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-34.3.3.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-2.0.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/appdirs-1.4.3.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #1458

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1848) GroupByKey stuck with more than one worker on Dataflow

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951462#comment-15951462
 ] 

Kenneth Knowles commented on BEAM-1848:
---

This is also now fixed in the Dataflow runner for the 0.6.0 release. Please try 
it out and see if your problem recurs.

> GroupByKey stuck with more than one worker on Dataflow
> --
>
> Key: BEAM-1848
> URL: https://issues.apache.org/jira/browse/BEAM-1848
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 0.6.0
>Reporter: peay
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> I have a simple pipeline which has a sliding window, a {{GroupByKey}} and 
> then a simple stateful {{DoFn}}. I run in batch mode ({{--streaming=false}}) 
> on Dataflow. 
> On a very small dataset  of a couple KBs, I can run the pipeline to 
> completion. Dataflow does show "successful".  On a larger dataset (but still 
> very small, 100s MB read by source), the pipeline stays stuck, no matter how 
> long I wait. In addition, it never really gets stuck at the same point. I 
> expect about 340k output records, and never get more than 70k before it gets 
> stuck.
> Dataflow always autoscales from 1 to 8 workers, which is my limit.
> Run A: after 30mins+: no elements added out of {{GroupByKey}}, but logs have 
> repeating occurrences of
> {code}
> Proposing dynamic split of work unit xx;aa;bb at 
> {"position":{"shufflePosition":"AAD_AP8A_wEAAQ"}}
> Refusing to split  [ShufflePosition(base64:AAD_AP8A_wD_AAE), 
> ShufflePosition(base64:AQD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AAD_AP8A_wEAAQ): unstarted   
> Refused to split GroupingShuffleReader  [ShufflePosition(base64:AAD_AP8A_wD_AAE), 
> ShufflePosition(base64:AQD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AAD_AP8A_wEAAQ)
> {code}
> Run B: after a couple minutes, elements get added to output of 
> {{GroupByKey}}, up to 56,128 and then stays stuck doing nothing, but logs 
> have repeating occurrences of
> {code}
> Proposing dynamic split of work unit xx;ccc;dd at 
> {"position":{"shufflePosition":"AQD_AP8A_wEAAQ"}}
> Refusing to split  [ShufflePosition(base64:AQD_AP8A_wD_AAE), 
> ShufflePosition(base64:AgD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AQD_AP8A_wEAAQ): unstarted
> Refused to split GroupingShuffleReader  [ShufflePosition(base64:AQD_AP8A_wD_AAE), 
> ShufflePosition(base64:AgD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AQD_AP8A_wEAAQ)
> {code}
> Run C: after 10mins: elements get added to output of {{GroupByKey}}, up to 
> 70,262 and then stays stuck doing nothing. No logs as above as far as I can 
> find.
> I've run this about a dozen times and it always gets stuck. I am trying out 
> right now to run the pipeline with the worker limit set to one, and 
> {{GroupByKey}} has output 150k so far, still increasing. This seems like a 
> workaround, but using one worker only is not ideal.
> cc [~dhalp...@google.com]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #1457

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #1691

[Apache Jenkins Server]

See 


Changes:

[altay] Add TestStream to Python SDK

--
[...truncated 650.01 KB...]
test_empty_side_outputs (apache_beam.transforms.ptransform_test.PTransformTest) 
... ok
:132:
 UserWarning: Using fallback coder for typehint: List[int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:132:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-34.3.3.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-2.0.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/appdirs-1.4.3.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/pyparsing-2.2.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging 
appdirs pyparsing
test_par_do_with_multiple_outputs_and_using_yield 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:132:
 UserWarning: Using fallback coder for typehint: List[int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:132:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-34.3.3.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-2.0.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/appdirs-1.4.3.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/pyparsing-2.2.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging 
appdirs pyparsing
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ...

[jira] [Commented] (BEAM-1848) GroupByKey stuck with more than one worker on Dataflow

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951454#comment-15951454
 ] 

Kenneth Knowles commented on BEAM-1848:
---

This is a known bug, fixed at HEAD, unfortunately discovered after the release 
of 0.6.0. What is happening is that the output is being re-assigned to windows 
- since the timer fires for each window, the sliding windows will be multiplied.

Filed BEAM-1849 for reference, since this was noticed and fixed as part of 
BEAM-1776. Root cause is roughly BEAM-1283, tracking deletion of the entire 
problematic code path.

> GroupByKey stuck with more than one worker on Dataflow
> --
>
> Key: BEAM-1848
> URL: https://issues.apache.org/jira/browse/BEAM-1848
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 0.6.0
>Reporter: peay
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> I have a simple pipeline which has a sliding window, a {{GroupByKey}} and 
> then a simple stateful {{DoFn}}. I run in batch mode ({{--streaming=false}}) 
> on Dataflow. 
> On a very small dataset  of a couple KBs, I can run the pipeline to 
> completion. Dataflow does show "successful".  On a larger dataset (but still 
> very small, 100s MB read by source), the pipeline stays stuck, no matter how 
> long I wait. In addition, it never really gets stuck at the same point. I 
> expect about 340k output records, and never get more than 70k before it gets 
> stuck.
> Dataflow always autoscales from 1 to 8 workers, which is my limit.
> Run A: after 30mins+: no elements added out of {{GroupByKey}}, but logs have 
> repeating occurrences of
> {code}
> Proposing dynamic split of work unit xx;aa;bb at 
> {"position":{"shufflePosition":"AAD_AP8A_wEAAQ"}}
> Refusing to split  [ShufflePosition(base64:AAD_AP8A_wD_AAE), 
> ShufflePosition(base64:AQD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AAD_AP8A_wEAAQ): unstarted   
> Refused to split GroupingShuffleReader  [ShufflePosition(base64:AAD_AP8A_wD_AAE), 
> ShufflePosition(base64:AQD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AAD_AP8A_wEAAQ)
> {code}
> Run B: after a couple minutes, elements get added to output of 
> {{GroupByKey}}, up to 56,128 and then stays stuck doing nothing, but logs 
> have repeating occurrences of
> {code}
> Proposing dynamic split of work unit xx;ccc;dd at 
> {"position":{"shufflePosition":"AQD_AP8A_wEAAQ"}}
> Refusing to split  [ShufflePosition(base64:AQD_AP8A_wD_AAE), 
> ShufflePosition(base64:AgD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AQD_AP8A_wEAAQ): unstarted
> Refused to split GroupingShuffleReader  [ShufflePosition(base64:AQD_AP8A_wD_AAE), 
> ShufflePosition(base64:AgD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AQD_AP8A_wEAAQ)
> {code}
> Run C: after 10mins: elements get added to output of {{GroupByKey}}, up to 
> 70,262 and then stays stuck doing nothing. No logs as above as far as I can 
> find.
> I've run this about a dozen times and it always gets stuck. I am trying out 
> right now to run the pipeline with the worker limit set to one, and 
> {{GroupByKey}} has output 150k so far, still increasing. This seems like a 
> workaround, but using one worker only is not ideal.
> cc [~dhalp...@google.com]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-1425) Window should comply with PTransform style guide

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-1425.
--
Resolution: Fixed
  Assignee: Eugene Kirpichov

> Window should comply with PTransform style guide
> 
>
> Key: BEAM-1425
> URL: https://issues.apache.org/jira/browse/BEAM-1425
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>  Labels: backward-incompatible, starter
> Fix For: First stable release
>
>
> Suggested changes:
> - Remove static builder-like methods such as triggering(), 
> discardingFiredPanes() - the only static entry point should be .into().
> - (optional) use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1849) Output from OnTimer method has windows re-assigned

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-1849.
---
Resolution: Fixed

This was already fixed, but filing for reference.

> Output from OnTimer method has windows re-assigned
> --
>
> Key: BEAM-1849
> URL: https://issues.apache.org/jira/browse/BEAM-1849
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 0.6.0
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1849) Output from OnTimer method has windows re-assigned

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1849:
-

 Summary: Output from OnTimer method has windows re-assigned
 Key: BEAM-1849
 URL: https://issues.apache.org/jira/browse/BEAM-1849
 Project: Beam
  Issue Type: Bug
  Components: runner-core
Affects Versions: 0.6.0
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
 Fix For: First stable release






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1425) Window should comply with PTransform style guide

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951432#comment-15951432
 ] 

ASF GitHub Bot commented on BEAM-1425:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2362


> Window should comply with PTransform style guide
> 
>
> Key: BEAM-1425
> URL: https://issues.apache.org/jira/browse/BEAM-1425
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>  Labels: backward-incompatible, starter
> Fix For: First stable release
>
>
> Suggested changes:
> - Remove static builder-like methods such as triggering(), 
> discardingFiredPanes() - the only static entry point should be .into().
> - (optional) use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2362: [BEAM-1425] Window should comply with PTransform st...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2362


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/6] beam git commit: Fixes a few warnings in Window

[jkff]

Fixes a few warnings in Window


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3563c4b9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3563c4b9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3563c4b9

Branch: refs/heads/master
Commit: 3563c4b9db8411b52b95c3997bc510ecc4cc8983
Parents: a092f6a
Author: Eugene Kirpichov 
Authored: Tue Mar 28 18:04:37 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Mar 31 10:59:37 2017 -0700

--
 .../beam/sdk/transforms/windowing/Window.java   | 27 ++--
 1 file changed, 14 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3563c4b9/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
index a10f112..5f5295d 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
@@ -169,7 +169,7 @@ public class Window {
* properties can be set on it first.
*/
   public static  Bound into(WindowFn fn) {
-return new Bound().into(fn);
+return new Bound().into(fn);
   }
 
   /**
@@ -182,7 +182,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound triggering(Trigger trigger) {
-return new Bound().triggering(trigger);
+return new Bound().triggering(trigger);
   }
 
   /**
@@ -194,7 +194,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound discardingFiredPanes() {
-return new Bound().discardingFiredPanes();
+return new Bound().discardingFiredPanes();
   }
 
   /**
@@ -206,7 +206,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound accumulatingFiredPanes() {
-return new Bound().accumulatingFiredPanes();
+return new Bound().accumulatingFiredPanes();
   }
 
   /**
@@ -222,7 +222,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound withAllowedLateness(Duration allowedLateness) {
-return new Bound().withAllowedLateness(allowedLateness);
+return new Bound().withAllowedLateness(allowedLateness);
   }
 
   /**
@@ -231,7 +231,7 @@ public class Window {
*/
   @Experimental(Kind.OUTPUT_TIME)
   public static  Bound withOutputTimeFn(OutputTimeFn outputTimeFn) {
-return new Bound().withOutputTimeFn(outputTimeFn);
+return new Bound().withOutputTimeFn(outputTimeFn);
   }
 
   /**
@@ -300,7 +300,7 @@ public class Window {
  */
 @Experimental(Kind.TRIGGER)
 public Bound triggering(Trigger trigger) {
-  return new Bound(
+  return new Bound<>(
   windowFn,
   trigger,
   mode,
@@ -318,7 +318,7 @@ public class Window {
 */
 @Experimental(Kind.TRIGGER)
public Bound discardingFiredPanes() {
- return new Bound(
+ return new Bound<>(
  windowFn,
  trigger,
  AccumulationMode.DISCARDING_FIRED_PANES,
@@ -336,7 +336,7 @@ public class Window {
 */
@Experimental(Kind.TRIGGER)
public Bound accumulatingFiredPanes() {
- return new Bound(
+ return new Bound<>(
  windowFn,
  trigger,
  AccumulationMode.ACCUMULATING_FIRED_PANES,
@@ -360,7 +360,7 @@ public class Window {
  */
 @Experimental(Kind.TRIGGER)
 public Bound withAllowedLateness(Duration allowedLateness) {
-  return new Bound(
+  return new Bound<>(
   windowFn, trigger, mode, allowedLateness, closingBehavior, 
outputTimeFn);
 }
 
@@ -370,7 +370,7 @@ public class Window {
  */
 @Experimental(Kind.OUTPUT_TIME)
 public Bound withOutputTimeFn(OutputTimeFn outputTimeFn) {
-  return new Bound(
+  return new Bound<>(
   windowFn, trigger, mode, allowedLateness, closingBehavior, 
outputTimeFn);
 }
 
@@ -386,7 +386,7 @@ public class Window {
  */
 @Experimental(Kind.TRIGGER)
 public Bound withAllowedLateness(Duration allowedLateness, 
ClosingBehavior behavior) {
-  return new Bound(windowFn, trigger, mode, allowedLateness, behavior, 
outputTimeFn);
+  return new Bound<>(windowFn, trigger, mode, allowedLateness, behavior, 
outputTimeFn);
 }
 
 /**
@@ -422,6 +422,7 @@ public class Window {
 /**
  * Get the {@link WindowFn} of this {@link Window.Bound Window PTransform}.
  */
+@Nullable
 public WindowFn getWindowFn() {
   return windowFn;
 }
@@ -568,7 +569,7 @@ public class Window {
* {@link org.apache.beam.sdk.transforms.GroupByKey}.
*/
   public static  R

[1/6] beam git commit: Uses AutoValue in Window

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master 023e6ab94 -> 132d3c5f6


Uses AutoValue in Window


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/876d13dd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/876d13dd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/876d13dd

Branch: refs/heads/master
Commit: 876d13dd367909490bdd052d5c140a784dacff14
Parents: 3563c4b
Author: Eugene Kirpichov 
Authored: Tue Mar 28 18:14:30 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Mar 31 10:59:37 2017 -0700

--
 .../beam/sdk/transforms/windowing/Window.java   | 165 +++
 1 file changed, 64 insertions(+), 101 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/876d13dd/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
index 5f5295d..a6c7adf 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.sdk.transforms.windowing;
 
+import com.google.auto.value.AutoValue;
 import com.google.common.annotations.VisibleForTesting;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.annotations.Experimental;
@@ -169,7 +170,7 @@ public class Window {
* properties can be set on it first.
*/
   public static  Bound into(WindowFn fn) {
-return new Bound().into(fn);
+return Window.configure().into(fn);
   }
 
   /**
@@ -182,7 +183,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound triggering(Trigger trigger) {
-return new Bound().triggering(trigger);
+return Window.configure().triggering(trigger);
   }
 
   /**
@@ -194,7 +195,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound discardingFiredPanes() {
-return new Bound().discardingFiredPanes();
+return Window.configure().discardingFiredPanes();
   }
 
   /**
@@ -206,7 +207,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound accumulatingFiredPanes() {
-return new Bound().accumulatingFiredPanes();
+return Window.configure().accumulatingFiredPanes();
   }
 
   /**
@@ -222,16 +223,11 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound withAllowedLateness(Duration allowedLateness) {
-return new Bound().withAllowedLateness(allowedLateness);
+return Window.configure().withAllowedLateness(allowedLateness);
   }
 
-  /**
-   * (Experimental) Override the default {@link OutputTimeFn}, 
to control
-   * the output timestamp of values output from a {@link GroupByKey} operation.
-   */
-  @Experimental(Kind.OUTPUT_TIME)
-  public static  Bound withOutputTimeFn(OutputTimeFn outputTimeFn) {
-return new Bound().withOutputTimeFn(outputTimeFn);
+  public static  Bound configure() {
+return new AutoValue_Window_Bound.Builder().build();
   }
 
   /**
@@ -240,33 +236,29 @@ public class Window {
*
* @param  The type of elements this {@code Window} is applied to
*/
-  public static class Bound extends PTransform, 
PCollection> {
-
-
-@Nullable private final WindowFn windowFn;
-@Nullable private final Trigger trigger;
-@Nullable private final AccumulationMode mode;
-@Nullable private final Duration allowedLateness;
-@Nullable private final ClosingBehavior closingBehavior;
-@Nullable private final OutputTimeFn outputTimeFn;
-
-private Bound(
-@Nullable WindowFn windowFn,
-@Nullable Trigger trigger,
-@Nullable AccumulationMode mode,
-@Nullable Duration allowedLateness,
-ClosingBehavior behavior,
-@Nullable OutputTimeFn outputTimeFn) {
-  this.windowFn = windowFn;
-  this.trigger = trigger;
-  this.mode = mode;
-  this.allowedLateness = allowedLateness;
-  this.closingBehavior = behavior;
-  this.outputTimeFn = outputTimeFn;
-}
-
-private Bound() {
-  this(null, null, null, null, null, null);
+  @AutoValue
+  public abstract static class Bound extends PTransform, 
PCollection> {
+@Nullable
+public abstract WindowFn getWindowFn();
+
+@Nullable abstract Trigger getTrigger();
+@Nullable abstract AccumulationMode getAccumulationMode();
+@Nullable abstract Duration getAllowedLateness();
+@Nullable abstract ClosingBehavior getClosingBehavior();
+@Nullable abstract OutputTimeFn getOutputTimeFn();
+
+abstract Builder toBuilder();
+
+@AutoValue.Build

[6/6] beam git commit: This closes #2362

[jkff]

This closes #2362


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/132d3c5f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/132d3c5f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/132d3c5f

Branch: refs/heads/master
Commit: 132d3c5f685279ed3e4c0623f6d956650152e86e
Parents: 023e6ab 6848950
Author: Eugene Kirpichov 
Authored: Fri Mar 31 10:59:53 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Mar 31 10:59:53 2017 -0700

--
 .../translation/WindowAssignTranslator.java |   2 +-
 .../direct/ParDoMultiOverrideFactory.java   |   4 +-
 .../direct/WindowEvaluatorFactoryTest.java  |   7 +-
 .../beam/runners/dataflow/AssignWindows.java|   4 +-
 .../dataflow/ReshuffleOverrideFactory.java  |   2 +-
 .../translation/streaming/CreateStreamTest.java |   4 +-
 .../org/apache/beam/sdk/testing/PAssert.java|  10 +-
 .../org/apache/beam/sdk/transforms/DoFn.java|   6 +-
 .../apache/beam/sdk/transforms/GroupByKey.java  |   2 +-
 .../beam/sdk/transforms/WithTimestamps.java |   6 +-
 .../beam/sdk/transforms/windowing/PaneInfo.java |   2 +-
 .../beam/sdk/transforms/windowing/Window.java   | 508 ---
 .../org/apache/beam/sdk/util/Reshuffle.java |   2 +-
 .../java/org/apache/beam/sdk/io/WriteTest.java  |   4 +-
 .../apache/beam/sdk/testing/TestStreamTest.java |   6 +-
 .../org/apache/beam/sdk/transforms/TopTest.java |   3 +-
 .../sdk/transforms/windowing/WindowTest.java|  39 +-
 17 files changed, 252 insertions(+), 359 deletions(-)
--

[5/6] beam git commit: Replaces Window.Bound with simply Window

[jkff]

Replaces Window.Bound with simply Window


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6848950c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6848950c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6848950c

Branch: refs/heads/master
Commit: 6848950cca5bee2dddc18ddca229a5deb9e34754
Parents: 6b67e54
Author: Eugene Kirpichov 
Authored: Wed Mar 29 13:09:49 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Mar 31 10:59:38 2017 -0700

--
 .../translation/WindowAssignTranslator.java |   2 +-
 .../direct/WindowEvaluatorFactoryTest.java  |   7 +-
 .../beam/runners/dataflow/AssignWindows.java|   4 +-
 .../dataflow/ReshuffleOverrideFactory.java  |   2 +-
 .../org/apache/beam/sdk/testing/PAssert.java|   4 +-
 .../org/apache/beam/sdk/transforms/DoFn.java|   2 +-
 .../apache/beam/sdk/transforms/GroupByKey.java  |   6 +-
 .../beam/sdk/transforms/WithTimestamps.java |   6 +-
 .../beam/sdk/transforms/windowing/Never.java|   2 +-
 .../beam/sdk/transforms/windowing/PaneInfo.java |   2 +-
 .../beam/sdk/transforms/windowing/Window.java   | 447 +--
 .../org/apache/beam/sdk/util/Reshuffle.java |   2 +-
 .../java/org/apache/beam/sdk/io/WriteTest.java  |   4 +-
 .../org/apache/beam/sdk/transforms/TopTest.java |   3 +-
 .../sdk/transforms/windowing/WindowTest.java|  17 +-
 15 files changed, 249 insertions(+), 261 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6848950c/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowAssignTranslator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowAssignTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowAssignTranslator.java
index b3aef8d..6106f75 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowAssignTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowAssignTranslator.java
@@ -35,7 +35,7 @@ import org.apache.beam.sdk.values.TupleTag;
 import org.apache.beam.sdk.values.TupleTagList;
 
 /**
- * {@link Window.Bound} is translated to {link ApexParDoOperator} that wraps 
an {@link
+ * {@link Window} is translated to {link ApexParDoOperator} that wraps an 
{@link
  * AssignWindowsDoFn}.
  */
 class WindowAssignTranslator implements 
TransformTranslator> {

http://git-wip-us.apache.org/repos/asf/beam/blob/6848950c/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java
index a71a75b..eb58629 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WindowEvaluatorFactoryTest.java
@@ -43,7 +43,6 @@ import org.apache.beam.sdk.transforms.windowing.PaneInfo;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo.Timing;
 import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
 import org.apache.beam.sdk.transforms.windowing.Window;
-import org.apache.beam.sdk.transforms.windowing.Window.Bound;
 import org.apache.beam.sdk.transforms.windowing.WindowFn;
 import org.apache.beam.sdk.transforms.windowing.WindowMappingFn;
 import org.apache.beam.sdk.util.WindowedValue;
@@ -113,7 +112,7 @@ public class WindowEvaluatorFactoryTest {
   @Test
   public void singleWindowFnSucceeds() throws Exception {
 Duration windowDuration = Duration.standardDays(7);
-Bound transform = Window.into(FixedWindows.of(windowDuration));
+Window transform = 
Window.into(FixedWindows.of(windowDuration));
 PCollection windowed = input.apply(transform);
 
 CommittedBundle inputBundle = createInputBundle();
@@ -152,7 +151,7 @@ public class WindowEvaluatorFactoryTest {
   public void multipleWindowsWindowFnSucceeds() throws Exception {
 Duration windowDuration = Duration.standardDays(6);
 Duration slidingBy = Duration.standardDays(3);
-Bound transform = 
Window.into(SlidingWindows.of(windowDuration).every(slidingBy));
+Window transform = 
Window.into(SlidingWindows.of(windowDuration).every(slidingBy));
 PCollection windowed = input.apply(transform);
 
 CommittedBundle inputBundle = createInputBundle();
@@ -209,7 +208,7 @@ public class WindowEvaluatorFactoryTest {
 
   @Test
   public void referencesEarlierWindowsSucceeds() throws Exception {
-Bound transform = Window.into(new Evalua

[2/6] beam git commit: Removes unused name parameter

[jkff]

Removes unused name parameter


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a092f6a4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a092f6a4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a092f6a4

Branch: refs/heads/master
Commit: a092f6a4ed3b7d3cfe7a0e157e241e8f410948b1
Parents: 023e6ab
Author: Eugene Kirpichov 
Authored: Tue Mar 28 17:59:11 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Mar 31 10:59:37 2017 -0700

--
 .../beam/sdk/transforms/windowing/Window.java   | 29 
 1 file changed, 12 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a092f6a4/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
index 373bbbd..a10f112 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
@@ -169,7 +169,7 @@ public class Window {
* properties can be set on it first.
*/
   public static  Bound into(WindowFn fn) {
-return new Bound(null).into(fn);
+return new Bound().into(fn);
   }
 
   /**
@@ -182,7 +182,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound triggering(Trigger trigger) {
-return new Bound(null).triggering(trigger);
+return new Bound().triggering(trigger);
   }
 
   /**
@@ -194,7 +194,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound discardingFiredPanes() {
-return new Bound(null).discardingFiredPanes();
+return new Bound().discardingFiredPanes();
   }
 
   /**
@@ -206,7 +206,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound accumulatingFiredPanes() {
-return new Bound(null).accumulatingFiredPanes();
+return new Bound().accumulatingFiredPanes();
   }
 
   /**
@@ -222,7 +222,7 @@ public class Window {
*/
   @Experimental(Kind.TRIGGER)
   public static  Bound withAllowedLateness(Duration allowedLateness) {
-return new Bound(null).withAllowedLateness(allowedLateness);
+return new Bound().withAllowedLateness(allowedLateness);
   }
 
   /**
@@ -231,7 +231,7 @@ public class Window {
*/
   @Experimental(Kind.OUTPUT_TIME)
   public static  Bound withOutputTimeFn(OutputTimeFn outputTimeFn) {
-return new Bound(null).withOutputTimeFn(outputTimeFn);
+return new Bound().withOutputTimeFn(outputTimeFn);
   }
 
   /**
@@ -251,14 +251,12 @@ public class Window {
 @Nullable private final OutputTimeFn outputTimeFn;
 
 private Bound(
-String name,
 @Nullable WindowFn windowFn,
 @Nullable Trigger trigger,
 @Nullable AccumulationMode mode,
 @Nullable Duration allowedLateness,
 ClosingBehavior behavior,
 @Nullable OutputTimeFn outputTimeFn) {
-  super(name);
   this.windowFn = windowFn;
   this.trigger = trigger;
   this.mode = mode;
@@ -267,8 +265,8 @@ public class Window {
   this.outputTimeFn = outputTimeFn;
 }
 
-private Bound(String name) {
-  this(name, null, null, null, null, null, null);
+private Bound() {
+  this(null, null, null, null, null, null);
 }
 
 /**
@@ -286,7 +284,7 @@ public class Window {
   }
 
   return new Bound<>(
-  name, windowFn, trigger, mode, allowedLateness, closingBehavior, 
outputTimeFn);
+  windowFn, trigger, mode, allowedLateness, closingBehavior, 
outputTimeFn);
 }
 
 /**
@@ -303,7 +301,6 @@ public class Window {
 @Experimental(Kind.TRIGGER)
 public Bound triggering(Trigger trigger) {
   return new Bound(
-  name,
   windowFn,
   trigger,
   mode,
@@ -322,7 +319,6 @@ public class Window {
 @Experimental(Kind.TRIGGER)
public Bound discardingFiredPanes() {
  return new Bound(
- name,
  windowFn,
  trigger,
  AccumulationMode.DISCARDING_FIRED_PANES,
@@ -341,7 +337,6 @@ public class Window {
@Experimental(Kind.TRIGGER)
public Bound accumulatingFiredPanes() {
  return new Bound(
- name,
  windowFn,
  trigger,
  AccumulationMode.ACCUMULATING_FIRED_PANES,
@@ -366,7 +361,7 @@ public class Window {
 @Experimental(Kind.TRIGGER)
 public Bound withAllowedLateness(Duration allowedLateness) {
   return new Bound(
-  name, windowFn, trigger, mode, allowedLateness, closingBehavior, 
outputTimeFn);
+  windowFn, trigger, mode, allowedLateness, clo

[4/6] beam git commit: Replaced static Window.blah() methods with Window.configure().blah() except Window.into()

[jkff]

Replaced static Window.blah() methods with Window.configure().blah() except 
Window.into()


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6b67e547
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6b67e547
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6b67e547

Branch: refs/heads/master
Commit: 6b67e547aab7658bcb6dfdf6eb5bf7e220ef7558
Parents: 876d13d
Author: Eugene Kirpichov 
Authored: Wed Mar 29 12:58:20 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Mar 31 10:59:38 2017 -0700

--
 .../direct/ParDoMultiOverrideFactory.java   |  4 +-
 .../translation/streaming/CreateStreamTest.java |  4 +-
 .../org/apache/beam/sdk/testing/PAssert.java|  6 +-
 .../org/apache/beam/sdk/transforms/DoFn.java|  8 +-
 .../apache/beam/sdk/transforms/GroupByKey.java  |  4 +-
 .../beam/sdk/transforms/WithTimestamps.java | 10 +--
 .../beam/sdk/transforms/windowing/Never.java|  2 +-
 .../beam/sdk/transforms/windowing/Window.java   | 80 +++-
 .../apache/beam/sdk/testing/TestStreamTest.java |  6 +-
 .../sdk/transforms/windowing/WindowTest.java| 24 +++---
 10 files changed, 47 insertions(+), 101 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6b67e547/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java
index 4604fcc..056a0c3 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoMultiOverrideFactory.java
@@ -135,8 +135,8 @@ class ParDoMultiOverrideFactory
   //  - ensure this GBK holds to the minimum of those timestamps 
(via OutputTimeFn)
   //  - discard past panes as it is "just a stream" of elements
   .apply(
-  Window.>>>triggering(
-  Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
+  Window.>>>configure()
+  
.triggering(Repeatedly.forever(AfterPane.elementCountAtLeast(1)))
   .discardingFiredPanes()
   
.withAllowedLateness(inputWindowingStrategy.getAllowedLateness())
   
.withOutputTimeFn(OutputTimeFns.outputAtEarliestInputTimestamp()))

http://git-wip-us.apache.org/repos/asf/beam/blob/6b67e547/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/CreateStreamTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/CreateStreamTest.java
 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/CreateStreamTest.java
index 75abc8b..78b8039 100644
--- 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/CreateStreamTest.java
+++ 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/CreateStreamTest.java
@@ -275,7 +275,7 @@ public class CreateStreamTest implements Serializable {
 PCollection createStrings =
 p.apply("CreateStrings", source)
 .apply("WindowStrings",
-Window.triggering(AfterPane.elementCountAtLeast(2))
+
Window.configure().triggering(AfterPane.elementCountAtLeast(2))
 .withAllowedLateness(Duration.ZERO)
 .accumulatingFiredPanes());
 PAssert.that(createStrings).containsInAnyOrder("foo", "bar");
@@ -283,7 +283,7 @@ public class CreateStreamTest implements Serializable {
 PCollection createInts =
 p.apply("CreateInts", other)
 .apply("WindowInts",
-Window.triggering(AfterPane.elementCountAtLeast(4))
+
Window.configure().triggering(AfterPane.elementCountAtLeast(4))
 .withAllowedLateness(Duration.ZERO)
 .accumulatingFiredPanes());
 PAssert.that(createInts).containsInAnyOrder(1, 2, 3, 4);

http://git-wip-us.apache.org/repos/asf/beam/blob/6b67e547/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
index 56df449..ab412c4 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sd

[jira] [Updated] (BEAM-1776) Timers should be delivered in the window they were set in

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1776:
--
Affects Version/s: 0.6.0

> Timers should be delivered in the window they were set in
> -
>
> Key: BEAM-1776
> URL: https://issues.apache.org/jira/browse/BEAM-1776
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 0.6.0
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> Currently timers are delivered by the runner to StatefulParDoEvaluator, and 
> the window contained in the StateNamespace is not used when delivering the 
> timer to the user.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1848) GroupByKey stuck with more than one worker on Dataflow

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951419#comment-15951419
 ] 

Daniel Halperin commented on BEAM-1848:
---

Suspect Kenn should take a look at this one :)

> GroupByKey stuck with more than one worker on Dataflow
> --
>
> Key: BEAM-1848
> URL: https://issues.apache.org/jira/browse/BEAM-1848
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 0.6.0
>Reporter: peay
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> I have a simple pipeline which has a sliding window, a {{GroupByKey}} and 
> then a simple stateful {{DoFn}}. I run in batch mode ({{--streaming=false}}) 
> on Dataflow. 
> On a very small dataset  of a couple KBs, I can run the pipeline to 
> completion. Dataflow does show "successful".  On a larger dataset (but still 
> very small, 100s MB read by source), the pipeline stays stuck, no matter how 
> long I wait. In addition, it never really gets stuck at the same point. I 
> expect about 340k output records, and never get more than 70k before it gets 
> stuck.
> Dataflow always autoscales from 1 to 8 workers, which is my limit.
> Run A: after 30mins+: no elements added out of {{GroupByKey}}, but logs have 
> repeating occurrences of
> {code}
> Proposing dynamic split of work unit xx;aa;bb at 
> {"position":{"shufflePosition":"AAD_AP8A_wEAAQ"}}
> Refusing to split  [ShufflePosition(base64:AAD_AP8A_wD_AAE), 
> ShufflePosition(base64:AQD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AAD_AP8A_wEAAQ): unstarted   
> Refused to split GroupingShuffleReader  [ShufflePosition(base64:AAD_AP8A_wD_AAE), 
> ShufflePosition(base64:AQD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AAD_AP8A_wEAAQ)
> {code}
> Run B: after a couple minutes, elements get added to output of 
> {{GroupByKey}}, up to 56,128 and then stays stuck doing nothing, but logs 
> have repeating occurrences of
> {code}
> Proposing dynamic split of work unit xx;ccc;dd at 
> {"position":{"shufflePosition":"AQD_AP8A_wEAAQ"}}
> Refusing to split  [ShufflePosition(base64:AQD_AP8A_wD_AAE), 
> ShufflePosition(base64:AgD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AQD_AP8A_wEAAQ): unstarted
> Refused to split GroupingShuffleReader  [ShufflePosition(base64:AQD_AP8A_wD_AAE), 
> ShufflePosition(base64:AgD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AQD_AP8A_wEAAQ)
> {code}
> Run C: after 10mins: elements get added to output of {{GroupByKey}}, up to 
> 70,262 and then stays stuck doing nothing. No logs as above as far as I can 
> find.
> I've run this about a dozen times and it always gets stuck. I am trying out 
> right now to run the pipeline with the worker limit set to one, and 
> {{GroupByKey}} has output 150k so far, still increasing. This seems like a 
> workaround, but using one worker only is not ideal.
> cc [~dhalp...@google.com]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1848) GroupByKey stuck with more than one worker on Dataflow

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin reassigned BEAM-1848:
-

Assignee: Kenneth Knowles  (was: Davor Bonaci)

> GroupByKey stuck with more than one worker on Dataflow
> --
>
> Key: BEAM-1848
> URL: https://issues.apache.org/jira/browse/BEAM-1848
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 0.6.0
>Reporter: peay
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> I have a simple pipeline which has a sliding window, a {{GroupByKey}} and 
> then a simple stateful {{DoFn}}. I run in batch mode ({{--streaming=false}}) 
> on Dataflow. 
> On a very small dataset  of a couple KBs, I can run the pipeline to 
> completion. Dataflow does show "successful".  On a larger dataset (but still 
> very small, 100s MB read by source), the pipeline stays stuck, no matter how 
> long I wait. In addition, it never really gets stuck at the same point. I 
> expect about 340k output records, and never get more than 70k before it gets 
> stuck.
> Dataflow always autoscales from 1 to 8 workers, which is my limit.
> Run A: after 30mins+: no elements added out of {{GroupByKey}}, but logs have 
> repeating occurrences of
> {code}
> Proposing dynamic split of work unit xx;aa;bb at 
> {"position":{"shufflePosition":"AAD_AP8A_wEAAQ"}}
> Refusing to split  [ShufflePosition(base64:AAD_AP8A_wD_AAE), 
> ShufflePosition(base64:AQD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AAD_AP8A_wEAAQ): unstarted   
> Refused to split GroupingShuffleReader  [ShufflePosition(base64:AAD_AP8A_wD_AAE), 
> ShufflePosition(base64:AQD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AAD_AP8A_wEAAQ)
> {code}
> Run B: after a couple minutes, elements get added to output of 
> {{GroupByKey}}, up to 56,128 and then stays stuck doing nothing, but logs 
> have repeating occurrences of
> {code}
> Proposing dynamic split of work unit xx;ccc;dd at 
> {"position":{"shufflePosition":"AQD_AP8A_wEAAQ"}}
> Refusing to split  [ShufflePosition(base64:AQD_AP8A_wD_AAE), 
> ShufflePosition(base64:AgD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AQD_AP8A_wEAAQ): unstarted
> Refused to split GroupingShuffleReader  [ShufflePosition(base64:AQD_AP8A_wD_AAE), 
> ShufflePosition(base64:AgD_AP8A_wD_AAE))> at 
> ShufflePosition(base64:AQD_AP8A_wEAAQ)
> {code}
> Run C: after 10mins: elements get added to output of {{GroupByKey}}, up to 
> 70,262 and then stays stuck doing nothing. No logs as above as far as I can 
> find.
> I've run this about a dozen times and it always gets stuck. I am trying out 
> right now to run the pipeline with the worker limit set to one, and 
> {{GroupByKey}} has output 150k so far, still increasing. This seems like a 
> workaround, but using one worker only is not ideal.
> cc [~dhalp...@google.com]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PerformanceTests_Dataflow #253

[Apache Jenkins Server]

See 


Changes:

[altay] Fixed typos in the Python SDK examples. ( tornatoes -> tornadoes )

[tgroh] Improve PTransformMatcher ToStrings

[tgroh] Include the creating PCollection in PCollectionView

[altay] Add TestStream to Python SDK

--
[...truncated 255.11 KB...]
error: unable to resolve reference refs/remotes/origin/pr/2322/merge: No such 
file or directory
 ! 3dc8bbd...c5feb35 refs/pull/2322/merge -> origin/pr/2322/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2323/merge: No such 
file or directory
 ! a3e177c...bc8d0de refs/pull/2323/merge -> origin/pr/2323/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2325/merge: No such 
file or directory
 ! 2ea8d57...b32e6c1 refs/pull/2325/merge -> origin/pr/2325/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2326/merge: No such 
file or directory
 ! b9919b1...a4c411d refs/pull/2326/merge -> origin/pr/2326/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2328/merge: No such 
file or directory
 ! 11f18cd...b7ef20d refs/pull/2328/merge -> origin/pr/2328/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2329/merge: No such 
file or directory
 ! 4053dbe...b1b38f9 refs/pull/2329/merge -> origin/pr/2329/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2330/merge: No such 
file or directory
 ! 320cef4...5d4c028 refs/pull/2330/merge -> origin/pr/2330/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2331/merge: No such 
file or directory
 ! 1fe14f4...7a396e9 refs/pull/2331/merge -> origin/pr/2331/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2332/merge: No such 
file or directory
 ! 4e92315...0177d82 refs/pull/2332/merge -> origin/pr/2332/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2333/merge: No such 
file or directory
 ! be865ca...55589d8 refs/pull/2333/merge -> origin/pr/2333/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2334/merge: No such 
file or directory
 ! 8e941b4...eaa378c refs/pull/2334/merge -> origin/pr/2334/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2335/merge: No such 
file or directory
 ! ba1e7ef...5db235e refs/pull/2335/merge -> origin/pr/2335/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2336/merge: No such 
file or directory
 ! a5c62da...42a0800 refs/pull/2336/merge -> origin/pr/2336/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2339/merge: No such 
file or directory
 ! 5f16b77...abfa9d3 refs/pull/2339/merge -> origin/pr/2339/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2340/merge: No such 
file or directory
 ! 36acc4d...375e9d0 refs/pull/2340/merge -> origin/pr/2340/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2341/merge: No such 
file or directory
 ! f10d47e...1d6959e refs/pull/2341/merge -> origin/pr/2341/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2342/merge: No such 
file or directory
 ! 13c1f5a...fcde5dd refs/pull/2342/merge -> origin/pr/2342/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2343/merge: No such 
file or directory
 ! 45eef58...8787f6f refs/pull/2343/merge -> origin/pr/2343/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2344/merge: No such 
file or directory
 ! bddb67a...7b5f7eb refs/pull/2344/merge -> origin/pr/2344/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2346/merge: No such 
file or directory
 ! 1a1c8a8...d41320d refs/pull/2346/merge -> origin/pr/2346/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2347/head: No such 
file or directory
 ! 1b97b35...953d6cf refs/pull/2347/head -> origin/pr/2347/head  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2347/merge: No such 
file or directory
 ! f8f773f...7200b15 refs/pull/2347/merge -> origin/pr/2347/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2348/merge: No such 
file or directory
 ! 21dca0a...7f5ca62 refs/pull/2348/merge -> origin/pr/2348/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2350/merge: No such 
file or directory
 ! 5d61cb3...9cee5f1 refs/pull/2350/mer

[jira] [Updated] (BEAM-1840) shaded classes are not getting into the proper package

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1840:

Affects Version/s: (was: First stable release)

> shaded classes are not getting into the proper package
> --
>
> Key: BEAM-1840
> URL: https://issues.apache.org/jira/browse/BEAM-1840
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Ismaël Mejía
>Priority: Trivial
>  Labels: newbie, starter
>
> The current shade configuration relocates classes into packages based in the 
> artifact name, however this is inconsistent with the package nams because the 
> beam artifact ids follow the directory structure beam-sdks-java-io-* but the 
> package structure is beam/sdk/java/io so there is an extra 's' that creates a 
> different package.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3110

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2389: Add TestStream to Python SDK

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2389


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2389

[altay]

This closes #2389


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/023e6ab9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/023e6ab9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/023e6ab9

Branch: refs/heads/master
Commit: 023e6ab94aad0990681eeaa7125d45260cbb1a4c
Parents: 62473ae 55db47d
Author: Ahmet Altay 
Authored: Fri Mar 31 10:53:20 2017 -0700
Committer: Ahmet Altay 
Committed: Fri Mar 31 10:53:20 2017 -0700

--
 sdks/python/apache_beam/transforms/window.py|   5 +
 .../apache_beam/transforms/window_test.py   |   6 +
 sdks/python/apache_beam/utils/test_stream.py| 163 +++
 .../apache_beam/utils/test_stream_test.py   |  82 ++
 4 files changed, 256 insertions(+)
--

[1/2] beam git commit: Add TestStream to Python SDK

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 62473ae4b -> 023e6ab94


Add TestStream to Python SDK

The TestStream will be used for verifying streaming runner semantics.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/55db47d5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/55db47d5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/55db47d5

Branch: refs/heads/master
Commit: 55db47d50bb97b238926c2a1b0b80c36b5345d44
Parents: 62473ae
Author: Charles Chen 
Authored: Thu Mar 30 18:20:04 2017 -0700
Committer: Ahmet Altay 
Committed: Fri Mar 31 10:53:16 2017 -0700

--
 sdks/python/apache_beam/transforms/window.py|   5 +
 .../apache_beam/transforms/window_test.py   |   6 +
 sdks/python/apache_beam/utils/test_stream.py| 163 +++
 .../apache_beam/utils/test_stream_test.py   |  82 ++
 4 files changed, 256 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/55db47d5/sdks/python/apache_beam/transforms/window.py
--
diff --git a/sdks/python/apache_beam/transforms/window.py 
b/sdks/python/apache_beam/transforms/window.py
index 3878dff..dcc58b7 100644
--- a/sdks/python/apache_beam/transforms/window.py
+++ b/sdks/python/apache_beam/transforms/window.py
@@ -246,6 +246,11 @@ class TimestampedValue(object):
 self.value = value
 self.timestamp = Timestamp.of(timestamp)
 
+  def __cmp__(self, other):
+if type(self) is not type(other):
+  return cmp(type(self), type(other))
+return cmp((self.value, self.timestamp), (other.value, other.timestamp))
+
 
 class GlobalWindow(BoundedWindow):
   """The default window into which all data is placed (via GlobalWindows)."""

http://git-wip-us.apache.org/repos/asf/beam/blob/55db47d5/sdks/python/apache_beam/transforms/window_test.py
--
diff --git a/sdks/python/apache_beam/transforms/window_test.py 
b/sdks/python/apache_beam/transforms/window_test.py
index 11c8a68..1ac95e4 100644
--- a/sdks/python/apache_beam/transforms/window_test.py
+++ b/sdks/python/apache_beam/transforms/window_test.py
@@ -62,6 +62,12 @@ reify_windows = core.ParDo(ReifyWindowsFn())
 
 class WindowTest(unittest.TestCase):
 
+  def test_timestamped_value_cmp(self):
+self.assertEqual(TimestampedValue('a', 2), TimestampedValue('a', 2))
+self.assertEqual(TimestampedValue('a', 2), TimestampedValue('a', 2.0))
+self.assertNotEqual(TimestampedValue('a', 2), TimestampedValue('a', 2.1))
+self.assertNotEqual(TimestampedValue('a', 2), TimestampedValue('b', 2))
+
   def test_global_window(self):
 self.assertEqual(GlobalWindow(), GlobalWindow())
 self.assertNotEqual(GlobalWindow(),

http://git-wip-us.apache.org/repos/asf/beam/blob/55db47d5/sdks/python/apache_beam/utils/test_stream.py
--
diff --git a/sdks/python/apache_beam/utils/test_stream.py 
b/sdks/python/apache_beam/utils/test_stream.py
new file mode 100644
index 000..7ae27b7
--- /dev/null
+++ b/sdks/python/apache_beam/utils/test_stream.py
@@ -0,0 +1,163 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Provides TestStream for verifying streaming runner semantics."""
+
+from abc import ABCMeta
+from abc import abstractmethod
+
+from apache_beam import coders
+from apache_beam import pvalue
+from apache_beam.transforms import PTransform
+from apache_beam.transforms.window import TimestampedValue
+from apache_beam.utils import timestamp
+from apache_beam.utils.windowed_value import WindowedValue
+
+
+class Event(object):
+  """Test stream event to be emitted during execution of a TestStream."""
+
+  __metaclass__ = ABCMeta
+
+  def __cmp__(self, other):
+if type(self) is not type(other):
+  return cmp(type(self), type(other))
+return self._typed_cmp(other)
+
+  @abstractmethod
+  def _typed_cmp(self, other):
+raise NotImplementedError
+
+
+class ElementEvent(Event):
+  """Element-producing test stre

Build failed in Jenkins: beam_PostCommit_Python_Verify #1690

[Apache Jenkins Server]

See 


Changes:

[tgroh] Improve PTransformMatcher ToStrings

[tgroh] Include the creating PCollection in PCollectionView

--
[...truncated 654.97 KB...]
test_par_do_with_multiple_outputs_and_using_return 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:132:
 UserWarning: Using fallback coder for typehint: List[int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:132:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-34.3.3.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-2.0.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/appdirs-1.4.3.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/pyparsing-2.2.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging 
appdirs pyparsing
test_undeclared_side_outputs 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ok
:132:
 UserWarning: Using fallback coder for typehint: List[int].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
:132:
 UserWarning: Using fallback coder for typehint: Union[].
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in 
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-34.3.3.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-2.0.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/appdirs-1.4.3.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r 
postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/pyparsing-2.2.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging 
appdirs pyparsing
test_as_dict_with_uniqu

[jira] [Resolved] (BEAM-1844) test_memory_usage fails in post commit

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-1844.
---
   Resolution: Fixed
Fix Version/s: First stable release

> test_memory_usage fails in post commit
> --
>
> Key: BEAM-1844
> URL: https://issues.apache.org/jira/browse/BEAM-1844
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
> Fix For: First stable release
>
>
> ...
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/typehints/typecheck.py",
>  line 78, in wrapper
> result = method(*args, **kwargs)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/transforms/core.py",
>  line 719, in 
> wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)]
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline_test.py",
>  line 245, in check_memory
> 'High memory usage: %d > %d' % (memory_usage, memory_threshold))
> RuntimeError: High memory usage: 125566976 > 123487104 [while running 
> 'oom:check']
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1677/consoleFull
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1675/consoleFull



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)