[jira] [Commented] (BEAM-1293) Python post commits failing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832870#comment-15832870
 ] 

ASF GitHub Bot commented on BEAM-1293:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/1809

[BEAM-1293] Update DataflowPipelineResult.state at the end of 
poll_for_job_completion

https://github.com/apache/beam/pull/1802 exposed a bug in 
`poll_for_job_completion`. At the end of the wait the new `state` was not 
propagated to the result object, causing it to raise an exception.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam drfail

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1809.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1809


commit 56512ab442c599c64bfdb9fc6cabce95d76ee4dc
Author: Ahmet Altay 
Date:   2017-01-21T07:43:42Z

Update DataflowPipelineResult.state at the end of poll_for_job_completion.




> Python post commits failing
> ---
>
> Key: BEAM-1293
> URL: https://issues.apache.org/jira/browse/BEAM-1293
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1058/consoleFull
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/dataflow_test.py",
>  line 323, in test_as_singleton_with_different_defaults_with_unique_labels
> pipeline.run()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/test_pipeline.py",
>  line 91, in run
> result = super(TestPipeline, self).run()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
>  line 163, in run
> return self.runner.run(self)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/test/test_dataflow_runner.py",
>  line 33, in run
> self.result.wait_until_finish()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow_runner.py",
>  line 711, in wait_until_finish
> (self.state, getattr(self._runner, 'last_error_msg', None)), self)
> DataflowRuntimeException: Dataflow pipeline failed. State: UNKNOWN, Error:
> None
> Caused by: https://github.com/apache/beam/pull/1802



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1809: [BEAM-1293] Update DataflowPipelineResult.state at ...

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/1809

[BEAM-1293] Update DataflowPipelineResult.state at the end of 
poll_for_job_completion

https://github.com/apache/beam/pull/1802 exposed a bug in 
`poll_for_job_completion`. At the end of the wait the new `state` was not 
propagated to the result object, causing it to raise an exception.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam drfail

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1809.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1809


commit 56512ab442c599c64bfdb9fc6cabce95d76ee4dc
Author: Ahmet Altay 
Date:   2017-01-21T07:43:42Z

Update DataflowPipelineResult.state at the end of poll_for_job_completion.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Python_Verify #1060

[Apache Jenkins Server]

See 

--
[...truncated 1 lines...]
  }
}, 
{
  "kind": "GroupByKey", 
  "name": "s8", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Group.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s7"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Group"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s9", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/UnKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputRefer

Build failed in Jenkins: beam_PostCommit_Python_Verify #1059

[Apache Jenkins Server]

See 

Changes:

[dhalperi] Clean *.pyc files with mvn clean.

--
[...truncated 18927 lines...]
  }
}, 
{
  "kind": "GroupByKey", 
  "name": "s8", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Group.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s7"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Group"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s9", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/UnKey.out"
  }

[jira] [Created] (BEAM-1293) Python post commits failing

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-1293:
-

 Summary: Python post commits failing
 Key: BEAM-1293
 URL: https://issues.apache.org/jira/browse/BEAM-1293
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Ahmet Altay


https://builds.apache.org/view/Beam/job/beam_PostCommit_Python_Verify/1058/consoleFull

Traceback (most recent call last):
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/dataflow_test.py",
 line 323, in test_as_singleton_with_different_defaults_with_unique_labels
pipeline.run()
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/test_pipeline.py",
 line 91, in run
result = super(TestPipeline, self).run()
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/pipeline.py",
 line 163, in run
return self.runner.run(self)
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/test/test_dataflow_runner.py",
 line 33, in run
self.result.wait_until_finish()
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/sdks/python/apache_beam/runners/dataflow_runner.py",
 line 711, in wait_until_finish
(self.state, getattr(self._runner, 'last_error_msg', None)), self)
DataflowRuntimeException: Dataflow pipeline failed. State: UNKNOWN, Error:
None

Caused by: https://github.com/apache/beam/pull/1802



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Python_Verify #1058

[Apache Jenkins Server]

See 

Changes:

[robertwb] Make TestPipeline.run fail when the underlying execution fails.

[robertwb] DataflowRunner will raise an exception on failures.

--
[...truncated 18920 lines...]
  }
}, 
{
  "kind": "GroupByKey", 
  "name": "s8", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Group.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s7"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Group"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s9", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
  

[jira] [Commented] (BEAM-1288) `mvn verify` leaves .pyc files throughout source tree

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832758#comment-15832758
 ] 

ASF GitHub Bot commented on BEAM-1288:
--

Github user aaltay closed the pull request at:

https://github.com/apache/beam/pull/1806


> `mvn verify` leaves .pyc files throughout source tree
> -
>
> Key: BEAM-1288
> URL: https://issues.apache.org/jira/browse/BEAM-1288
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Daniel Halperin
>Assignee: Ahmet Altay
>
> even after running `mvn clean`



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1806: [BEAM-1288] Clean *.pyc files with mvn clean.

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/beam/pull/1806


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1185) Remove the word Pipeline from the name of all PipelineRunner implementations

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-1185.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Remove the word Pipeline from the name of all PipelineRunner implementations
> 
>
> Key: BEAM-1185
> URL: https://issues.apache.org/jira/browse/BEAM-1185
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>  Labels: backward-incompatible, sdk-consistency
> Fix For: Not applicable
>
>
> See: https://issues.apache.org/jira/browse/BEAM-234
> Rename all runners to remove the Pipeline word from their name in the Python 
> SDK (e.g. DirectPipelineRunner -> DirectRunner).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] beam git commit: This closes #1806

[dhalperi]

This closes #1806


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c03e6f37
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c03e6f37
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c03e6f37

Branch: refs/heads/python-sdk
Commit: c03e6f37595b676cfc2821f2bfa33b0a81aad092
Parents: 82599a2 086ea86
Author: Dan Halperin 
Authored: Fri Jan 20 18:15:37 2017 -0800
Committer: Dan Halperin 
Committed: Fri Jan 20 18:15:37 2017 -0800

--
 pom.xml | 18 ++
 1 file changed, 18 insertions(+)
--

[1/2] beam git commit: Clean *.pyc files with mvn clean.

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/python-sdk 82599a241 -> c03e6f375


Clean *.pyc files with mvn clean.

Added maven-clean-plugin configuration to clean artifacts generated by
other phases of mvn.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/086ea862
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/086ea862
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/086ea862

Branch: refs/heads/python-sdk
Commit: 086ea8621706a0170059fec4ac8b5a8046d5098d
Parents: 82599a2
Author: Ahmet Altay 
Authored: Fri Jan 20 16:29:35 2017 -0800
Committer: Dan Halperin 
Committed: Fri Jan 20 18:15:32 2017 -0800

--
 pom.xml | 18 ++
 1 file changed, 18 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/086ea862/pom.xml
--
diff --git a/pom.xml b/pom.xml
index b83c4cf..563bca8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1208,6 +1208,24 @@
 
   
 
+
+
+  org.apache.maven.plugins
+  maven-clean-plugin
+  3.0.0
+  
+
+  
+.
+
+  **/*.pyc
+  **/*.egg-info/
+
+false
+  
+
+  
+
   
 
 

[GitHub] beam-site pull request #129: [BEAM-751] Extract Python code snippets from te...

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam-site/pull/129

[BEAM-751] Extract Python code snippets from tested examples.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam-site snippets

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/129.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #129


commit 0b44ef7770f121558d4008290c49dcd3dd4f8bae
Author: Robert Bradshaw 
Date:   2017-01-21T01:26:41Z

Add plugin for snippet extraction.

commit b8673ddee75cbdf92067d5787426c0a68a681d02
Author: Robert Bradshaw 
Date:   2017-01-21T01:55:13Z

Replace literal code samples with extracted, tested snippets.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-751) infrastructure for extracting code snippets into documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832734#comment-15832734
 ] 

ASF GitHub Bot commented on BEAM-751:
-

GitHub user robertwb opened a pull request:

https://github.com/apache/beam-site/pull/129

[BEAM-751] Extract Python code snippets from tested examples.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam-site snippets

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/129.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #129


commit 0b44ef7770f121558d4008290c49dcd3dd4f8bae
Author: Robert Bradshaw 
Date:   2017-01-21T01:26:41Z

Add plugin for snippet extraction.

commit b8673ddee75cbdf92067d5787426c0a68a681d02
Author: Robert Bradshaw 
Date:   2017-01-21T01:55:13Z

Replace literal code samples with extracted, tested snippets.




> infrastructure for extracting code snippets into documentation
> --
>
> Key: BEAM-751
> URL: https://issues.apache.org/jira/browse/BEAM-751
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Frances Perry
>Assignee: James Malone
>  Labels: starter
>
> As we fill in more and more documentation, the number of code snippets is 
> going to drastically increase, and we should ensure the quality of those 
> snippets by automatically extracting them from code that is regularly 
> compiled and tested.  
> Goals:
> * automatically extract code snippets from incubator-beam for use in the beam 
> website documentation
> * use stable references so folks editing the code can clearly tell what 
> documentation changes this will result in (good: specially formatted comment, 
> bad: line number)
> * freshness (is live possible? or at least during the general 'jekyll build' 
> phase?)
> The best we've found so far is using jekyll-gist with gist-it, but that would 
> rely on fragile line numbers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-59) IOChannelFactory rethinking/redesign

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832733#comment-15832733
 ] 

ASF GitHub Bot commented on BEAM-59:


GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/1808

[BEAM-59] Beam FileSystem: initial implementation.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam 
file-system-local-gs-other-methods

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1808.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1808


commit 5be0ac12051522c754ea0dbc07ae42e632ca09f1
Author: Pei He 
Date:   2017-01-21T00:57:48Z

[BEAM-59] Beam FileSystem: initial implementation.




> IOChannelFactory rethinking/redesign
> 
>
> Key: BEAM-59
> URL: https://issues.apache.org/jira/browse/BEAM-59
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Daniel Halperin
>Assignee: Pei He
>
> Right now, FileBasedSource and FileBasedSink communication is mediated by 
> IOChannelFactory. There are a number of issues:
> * Global configuration -- e.g., all 'gs://' URIs use the same credentials. 
> This should be per-source/per-sink/etc.
> * Supported APIs -- currently IOChannelFactory is in the "non-public API" 
> util package and subject to change. We need users to be able to add new 
> backends ('s3://', 'hdfs://', etc.) directly, without fear that they will be 
> broken.
> * Per-backend features: e.g., creating buckets in GCS/s3, setting expiration 
> time, etc.
> Updates:
> Design docs posted on dev@ list:
> Part 1: IOChannelFactory Redesign: 
> https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M/edit#
> Part 2: Configurable BeamFileSystem:
> https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit#heading=h.p3gc3colc2cs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1808: [BEAM-59] Beam FileSystem: initial implementation.

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/1808

[BEAM-59] Beam FileSystem: initial implementation.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam 
file-system-local-gs-other-methods

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1808.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1808


commit 5be0ac12051522c754ea0dbc07ae42e632ca09f1
Author: Pei He 
Date:   2017-01-21T00:57:48Z

[BEAM-59] Beam FileSystem: initial implementation.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1100) Consider removing pylint from mvn verify

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832726#comment-15832726
 ] 

ASF GitHub Bot commented on BEAM-1100:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/1807

[BEAM-1100] Skip pylint when running outside of a git repository.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam pyl

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1807.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1807


commit f2d0b2bb0c3f79f9a24d2c22a41976cd99b85e8f
Author: Ahmet Altay 
Date:   2017-01-21T01:32:29Z

Skip pylint when running outside of a git repository.




> Consider removing pylint from mvn verify
> 
>
> Key: BEAM-1100
> URL: https://issues.apache.org/jira/browse/BEAM-1100
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Critical
>
> pylint depends on git commands for getting a list of changed files to run the 
> linter on. This does not make sense in some cases (for example when in the 
> case code was copied/packaged out of git structure). A few possible remedies 
> could be considered:
> - Remove pylint step from mvn verify
> - Add an option to pylint run on whole code base and use this in mvn verify
> - Change pylint to run on the whole code base (instead of just changed files)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1807: [BEAM-1100] Skip pylint when running outside of a g...

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/1807

[BEAM-1100] Skip pylint when running outside of a git repository.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam pyl

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1807.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1807


commit f2d0b2bb0c3f79f9a24d2c22a41976cd99b85e8f
Author: Ahmet Altay 
Date:   2017-01-21T01:32:29Z

Skip pylint when running outside of a git repository.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Reopened] (BEAM-759) PipelineResult needs waitToFinish() and cancel()

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reopened BEAM-759:
--

(Want to keep it open for the removal of BlockingDataflowRunner)

> PipelineResult needs waitToFinish() and cancel()
> 
>
> Key: BEAM-759
> URL: https://issues.apache.org/jira/browse/BEAM-759
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>  Labels: sdk-consistency
> Fix For: Not applicable
>
>
> Java SDK, added waitToFinish() and cancel() to the PipelineResults, and as a 
> result were able to remove BlockingDataflowRunner.
> (See: https://issues.apache.org/jira/browse/BEAM-443)
> The same changes needs to happen in python sdk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-759) PipelineResult needs waitToFinish() and cancel()

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-759.
--
   Resolution: Fixed
 Assignee: Ahmet Altay
Fix Version/s: Not applicable

> PipelineResult needs waitToFinish() and cancel()
> 
>
> Key: BEAM-759
> URL: https://issues.apache.org/jira/browse/BEAM-759
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>  Labels: sdk-consistency
> Fix For: Not applicable
>
>
> Java SDK, added waitToFinish() and cancel() to the PipelineResults, and as a 
> result were able to remove BlockingDataflowRunner.
> (See: https://issues.apache.org/jira/browse/BEAM-443)
> The same changes needs to happen in python sdk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-759) PipelineResult needs waitToFinish() and cancel()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832693#comment-15832693
 ] 

ASF GitHub Bot commented on BEAM-759:
-

Github user aaltay closed the pull request at:

https://github.com/apache/beam/pull/1802


> PipelineResult needs waitToFinish() and cancel()
> 
>
> Key: BEAM-759
> URL: https://issues.apache.org/jira/browse/BEAM-759
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: sdk-consistency
>
> Java SDK, added waitToFinish() and cancel() to the PipelineResults, and as a 
> result were able to remove BlockingDataflowRunner.
> (See: https://issues.apache.org/jira/browse/BEAM-443)
> The same changes needs to happen in python sdk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1802: [BEAM-759] Make TestPipeline.run fail when the unde...

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/beam/pull/1802


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam git commit: Closes #1802

[robertwb]

Closes #1802


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/82599a24
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/82599a24
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/82599a24

Branch: refs/heads/python-sdk
Commit: 82599a241454ba53640c0da4cae3f60ea7668e3e
Parents: c57c66e 1ed81a2
Author: Robert Bradshaw 
Authored: Fri Jan 20 16:46:22 2017 -0800
Committer: Robert Bradshaw 
Committed: Fri Jan 20 16:46:22 2017 -0800

--
 .../apache_beam/runners/dataflow_runner.py  |  7 ++-
 .../apache_beam/runners/dataflow_runner_test.py | 64 
 sdks/python/apache_beam/test_pipeline.py|  5 +-
 .../apache_beam/transforms/aggregator_test.py   |  2 +-
 4 files changed, 75 insertions(+), 3 deletions(-)
--

[2/3] beam git commit: DataflowRunner will raise an exception on failures.

[robertwb]

DataflowRunner will raise an exception on failures.

This is the same behavior as before.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1ed81a26
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1ed81a26
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1ed81a26

Branch: refs/heads/python-sdk
Commit: 1ed81a2655a2c98655d8e5ce965eb72681388926
Parents: aa3a2cb
Author: Ahmet Altay 
Authored: Fri Jan 20 11:06:38 2017 -0800
Committer: Robert Bradshaw 
Committed: Fri Jan 20 16:46:21 2017 -0800

--
 .../apache_beam/runners/dataflow_runner.py  | 17 --
 .../apache_beam/runners/dataflow_runner_test.py | 64 
 2 files changed, 77 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1ed81a26/sdks/python/apache_beam/runners/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow_runner.py
index fd22753..bd25dbf 100644
--- a/sdks/python/apache_beam/runners/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow_runner.py
@@ -151,7 +151,6 @@ class DataflowRunner(PipelineRunner):
 if not page_token:
   break
 
-runner.result = DataflowPipelineResult(response, runner)
 runner.last_error_msg = last_error_msg
 
   def run(self, pipeline):
@@ -705,9 +704,11 @@ class DataflowPipelineResult(PipelineResult):
   while thread.isAlive():
 time.sleep(5.0)
   if self.state != PipelineState.DONE:
-logging.error(
-'Dataflow pipeline failed. State: %s, Error:\n%s',
-self.state, getattr(self._runner, 'last_error_msg', None))
+# TODO(BEAM-1290): Consider converting this to an error log based on 
the
+# resolution of the issue.
+raise DataflowRuntimeException(
+'Dataflow pipeline failed. State: %s, Error:\n%s' %
+(self.state, getattr(self._runner, 'last_error_msg', None)), self)
 return self.state
 
   def __str__(self):
@@ -718,3 +719,11 @@ class DataflowPipelineResult(PipelineResult):
 
   def __repr__(self):
 return '<%s %s at %s>' % (self.__class__.__name__, self._job, 
hex(id(self)))
+
+
+class DataflowRuntimeException(Exception):
+  """Indicates an error has occurred in running this pipeline."""
+
+  def __init__(self, msg, result):
+super(DataflowRuntimeException, self).__init__(msg)
+self.result = result

http://git-wip-us.apache.org/repos/asf/beam/blob/1ed81a26/sdks/python/apache_beam/runners/dataflow_runner_test.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow_runner_test.py 
b/sdks/python/apache_beam/runners/dataflow_runner_test.py
new file mode 100644
index 000..a935c98
--- /dev/null
+++ b/sdks/python/apache_beam/runners/dataflow_runner_test.py
@@ -0,0 +1,64 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for the DataflowRunner class."""
+
+import unittest
+import mock
+
+from apache_beam.internal.clients import dataflow as dataflow_api
+from apache_beam.runners.dataflow_runner import DataflowRuntimeException
+from apache_beam.runners.dataflow_runner import DataflowPipelineResult
+
+
+class DataflowRunnerTest(unittest.TestCase):
+
+  @mock.patch('time.sleep', return_value=None)
+  def test_wait_until_finish(self, patched_time_sleep):
+values_enum = dataflow_api.Job.CurrentStateValueValuesEnum
+
+class MockDataflowRunner(object):
+
+  def __init__(self, final_state):
+self.dataflow_client = mock.MagicMock()
+self.job = mock.MagicMock()
+self.job.currentState = values_enum.JOB_STATE_UNKNOWN
+
+def get_job_side_effect(*args, **kwargs):
+  self.job.currentState = final_state
+  return mock.DEFAULT
+
+self.dataflow_client.get_job = mock.MagicMock(
+return_value=self.job, side_effect=get_job_side_effect)
+self.dataflow_client.list_messages = mock.MagicMock

[1/3] beam git commit: Make TestPipeline.run fail when the underlying execution fails.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/python-sdk c57c66ed4 -> 82599a241


Make TestPipeline.run fail when the underlying execution fails.

Also, DataflowRunner will log the last error from its wait_until_finish
method.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/aa3a2cb3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/aa3a2cb3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/aa3a2cb3

Branch: refs/heads/python-sdk
Commit: aa3a2cb326a5f761eba9fe87fe7d57da9ce78555
Parents: c57c66e
Author: Ahmet Altay 
Authored: Thu Jan 19 16:13:07 2017 -0800
Committer: Robert Bradshaw 
Committed: Fri Jan 20 16:46:20 2017 -0800

--
 sdks/python/apache_beam/runners/dataflow_runner.py| 12 
 sdks/python/apache_beam/test_pipeline.py  |  5 -
 sdks/python/apache_beam/transforms/aggregator_test.py |  2 +-
 3 files changed, 9 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/aa3a2cb3/sdks/python/apache_beam/runners/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow_runner.py
index 330472b..fd22753 100644
--- a/sdks/python/apache_beam/runners/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow_runner.py
@@ -704,6 +704,10 @@ class DataflowPipelineResult(PipelineResult):
   thread.start()
   while thread.isAlive():
 time.sleep(5.0)
+  if self.state != PipelineState.DONE:
+logging.error(
+'Dataflow pipeline failed. State: %s, Error:\n%s',
+self.state, getattr(self._runner, 'last_error_msg', None))
 return self.state
 
   def __str__(self):
@@ -714,11 +718,3 @@ class DataflowPipelineResult(PipelineResult):
 
   def __repr__(self):
 return '<%s %s at %s>' % (self.__class__.__name__, self._job, 
hex(id(self)))
-
-
-class DataflowRuntimeException(Exception):
-  """Indicates an error has occurred in running this pipeline."""
-
-  def __init__(self, msg, result):
-super(DataflowRuntimeException, self).__init__(msg)
-self.result = result

http://git-wip-us.apache.org/repos/asf/beam/blob/aa3a2cb3/sdks/python/apache_beam/test_pipeline.py
--
diff --git a/sdks/python/apache_beam/test_pipeline.py 
b/sdks/python/apache_beam/test_pipeline.py
index c29a879..7d85af9 100644
--- a/sdks/python/apache_beam/test_pipeline.py
+++ b/sdks/python/apache_beam/test_pipeline.py
@@ -22,6 +22,7 @@ import shlex
 
 from apache_beam.internal import pickler
 from apache_beam.pipeline import Pipeline
+from apache_beam.runners.runner import PipelineState
 from apache_beam.utils.pipeline_options import PipelineOptions
 from nose.plugins.skip import SkipTest
 
@@ -89,7 +90,9 @@ class TestPipeline(Pipeline):
   def run(self):
 result = super(TestPipeline, self).run()
 if self.blocking:
-  result.wait_until_finish()
+  state = result.wait_until_finish()
+  assert state == PipelineState.DONE, "Pipeline execution failed."
+
 return result
 
   def _parse_test_option_args(self, argv):

http://git-wip-us.apache.org/repos/asf/beam/blob/aa3a2cb3/sdks/python/apache_beam/transforms/aggregator_test.py
--
diff --git a/sdks/python/apache_beam/transforms/aggregator_test.py 
b/sdks/python/apache_beam/transforms/aggregator_test.py
index d493c46..a2a4144 100644
--- a/sdks/python/apache_beam/transforms/aggregator_test.py
+++ b/sdks/python/apache_beam/transforms/aggregator_test.py
@@ -20,9 +20,9 @@
 import unittest
 
 import apache_beam as beam
+from apache_beam.test_pipeline import TestPipeline
 from apache_beam.transforms import combiners
 from apache_beam.transforms.aggregator import Aggregator
-from apache_beam.test_pipeline import TestPipeline
 
 
 class AggregatorTest(unittest.TestCase):

[jira] [Commented] (BEAM-1288) `mvn verify` leaves .pyc files throughout source tree

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832669#comment-15832669
 ] 

ASF GitHub Bot commented on BEAM-1288:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/1806

[BEAM-1288] Clean *.pyc files with mvn clean.

Added maven-clean-plugin configuration to clean artifacts generated by
other phases of mvn.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam mvn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1806.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1806


commit 7ee9d743e26a5151a15174dc70b3861f76d5c5db
Author: Ahmet Altay 
Date:   2017-01-21T00:29:35Z

Clean *.pyc files with mvn clean.

Added maven-clean-plugin configuration to clean artifacts generated by
other phases of mvn.




> `mvn verify` leaves .pyc files throughout source tree
> -
>
> Key: BEAM-1288
> URL: https://issues.apache.org/jira/browse/BEAM-1288
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Daniel Halperin
>Assignee: Ahmet Altay
>
> even after running `mvn clean`



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1806: [BEAM-1288] Clean *.pyc files with mvn clean.

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/1806

[BEAM-1288] Clean *.pyc files with mvn clean.

Added maven-clean-plugin configuration to clean artifacts generated by
other phases of mvn.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam mvn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1806.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1806


commit 7ee9d743e26a5151a15174dc70b3861f76d5c5db
Author: Ahmet Altay 
Date:   2017-01-21T00:29:35Z

Clean *.pyc files with mvn clean.

Added maven-clean-plugin configuration to clean artifacts generated by
other phases of mvn.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Spark #716

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #2067

[Apache Jenkins Server]

See 

[GitHub] beam pull request #1764: Standard coder tests

[robertwb]

Github user robertwb closed the pull request at:

https://github.com/apache/beam/pull/1764


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1292) PubSubIO: throws error when configured with topic

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1292:
-

 Summary: PubSubIO: throws error when configured with topic
 Key: BEAM-1292
 URL: https://issues.apache.org/jira/browse/BEAM-1292
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Daniel Halperin
Assignee: Daniel Halperin
 Fix For: 0.5.0


Seems related to value provider refactor:



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Spark #715

[Apache Jenkins Server]

See 

beam git commit: Automated refactoring of PubsubIO to fix indentation

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master c6e46b655 -> f799a57af


Automated refactoring of PubsubIO to fix indentation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f799a57a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f799a57a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f799a57a

Branch: refs/heads/master
Commit: f799a57af14722e1da26baee25bda03bf6a52b6e
Parents: c6e46b6
Author: Dan Halperin 
Authored: Fri Jan 20 15:00:05 2017 -0800
Committer: Dan Halperin 
Committed: Fri Jan 20 15:00:05 2017 -0800

--
 .../java/org/apache/beam/sdk/io/PubsubIO.java   | 803 ++-
 1 file changed, 412 insertions(+), 391 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f799a57a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java
index 2802871..1471953 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/PubsubIO.java
@@ -71,6 +71,7 @@ import org.slf4j.LoggerFactory;
  * {@link PipelineRunner PipelineRunners} for more details.
  */
 public class PubsubIO {
+
   private static final Logger LOG = LoggerFactory.getLogger(PubsubIO.class);
 
   /** Factory for creating pubsub client to manage transport. */
@@ -157,6 +158,7 @@ public class PubsubIO {
* a map of attached attributes.
*/
   public static class PubsubMessage {
+
 private byte[] message;
 private Map attributes;
 
@@ -193,7 +195,8 @@ public class PubsubIO {
* Class representing a Cloud Pub/Sub Subscription.
*/
   public static class PubsubSubscription implements Serializable {
-private enum Type { NORMAL, FAKE }
+
+private enum Type {NORMAL, FAKE}
 
 private final Type type;
 private final String project;
@@ -299,6 +302,7 @@ public class PubsubIO {
*/
   private static class SubscriptionTranslator
   implements SerializableFunction {
+
 @Override
 public PubsubSubscription apply(String from) {
   return PubsubSubscription.fromPath(from);
@@ -310,6 +314,7 @@ public class PubsubIO {
*/
   private static class SubscriptionPathTranslator
   implements SerializableFunction {
+
 @Override
 public SubscriptionPath apply(PubsubSubscription from) {
   return PubsubClient.subscriptionPathFromName(from.project, 
from.subscription);
@@ -321,6 +326,7 @@ public class PubsubIO {
*/
   private static class TopicTranslator
   implements SerializableFunction {
+
 @Override
 public PubsubTopic apply(String from) {
   return PubsubTopic.fromPath(from);
@@ -332,6 +338,7 @@ public class PubsubIO {
*/
   private static class TopicPathTranslator
   implements SerializableFunction {
+
 @Override
 public TopicPath apply(PubsubTopic from) {
   return PubsubClient.topicPathFromName(from.project, from.topic);
@@ -343,6 +350,7 @@ public class PubsubIO {
*/
   private static class ProjectPathTranslator
   implements SerializableFunction {
+
 @Override
 public ProjectPath apply(PubsubTopic from) {
   return PubsubClient.projectPathFromId(from.project);
@@ -353,7 +361,8 @@ public class PubsubIO {
* Class representing a Cloud Pub/Sub Topic.
*/
   public static class PubsubTopic implements Serializable {
-private enum Type { NORMAL, FAKE }
+
+private enum Type {NORMAL, FAKE}
 
 private final Type type;
 private final String project;
@@ -471,451 +480,458 @@ public class PubsubIO {
* can be processed. As such, either {@link 
PubsubIO.Read#maxNumRecords(int)} or
* {@link PubsubIO.Read#maxReadTime(Duration)} must be set.
*/
-   public static class Read extends PTransform> {
-  /** The Cloud Pub/Sub topic to read from. */
-  @Nullable private final ValueProvider topic;
+  public static class Read extends PTransform> {
 
-  /** The Cloud Pub/Sub subscription to read from. */
-  @Nullable private final ValueProvider subscription;
+/** The Cloud Pub/Sub topic to read from. */
+@Nullable
+private final ValueProvider topic;
 
-  /** The name of the message attribute to read timestamps from. */
-  @Nullable private final String timestampLabel;
+/** The Cloud Pub/Sub subscription to read from. */
+@Nullable
+private final ValueProvider subscription;
 
-  /** The name of the message attribute to read unique message IDs from. */
-  @Nullable private final String idLabel;
+/** The name of the message attribute to read timestamps from. */
+@Nullable
+private final String timestampLabel;
 
-  /** 

[jira] [Commented] (BEAM-974) Adds attributes support to PubsubIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832565#comment-15832565
 ] 

ASF GitHub Bot commented on BEAM-974:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1358


> Adds attributes support to PubsubIO
> ---
>
> Key: BEAM-974
> URL: https://issues.apache.org/jira/browse/BEAM-974
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 0.3.0-incubating
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>
> Adds support for Pubsub attributes by introducing a new PubsubMessage class 
> that allows manipulation of payload and attributes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] beam git commit: Add PubSub attributes support to PubsubIO.

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master a6810372b -> c6e46b655


http://git-wip-us.apache.org/repos/asf/beam/blob/f032facb/sdks/java/core/src/test/java/org/apache/beam/sdk/io/PubsubUnboundedSinkTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/PubsubUnboundedSinkTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/PubsubUnboundedSinkTest.java
index 5bc1664..7a4be62 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/PubsubUnboundedSinkTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/PubsubUnboundedSinkTest.java
@@ -19,10 +19,14 @@
 package org.apache.beam.sdk.io;
 
 import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
 import com.google.common.hash.Hashing;
 import java.io.IOException;
+import java.io.Serializable;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Map;
+
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.io.PubsubUnboundedSink.RecordIdMethod;
 import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
@@ -32,6 +36,7 @@ import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SimpleFunction;
 import org.apache.beam.sdk.util.PubsubClient;
 import org.apache.beam.sdk.util.PubsubClient.OutgoingMessage;
 import org.apache.beam.sdk.util.PubsubClient.TopicPath;
@@ -49,9 +54,11 @@ import org.junit.runners.JUnit4;
  * Test PubsubUnboundedSink.
  */
 @RunWith(JUnit4.class)
-public class PubsubUnboundedSinkTest {
+public class PubsubUnboundedSinkTest implements Serializable {
   private static final TopicPath TOPIC = 
PubsubClient.topicPathFromName("testProject", "testTopic");
   private static final String DATA = "testData";
+  private static final Map ATTRIBUTES =
+  ImmutableMap.builder().put("a", "b").put("c", 
"d").build();
   private static final long TIMESTAMP = 1234L;
   private static final String TIMESTAMP_LABEL = "timestamp";
   private static final String ID_LABEL = "id";
@@ -69,11 +76,12 @@ public class PubsubUnboundedSinkTest {
   }
 
   @Rule
-  public TestPipeline p = TestPipeline.create();
+  public transient TestPipeline p = TestPipeline.create();
 
   @Test
   public void saneCoder() throws Exception {
-OutgoingMessage message = new OutgoingMessage(DATA.getBytes(), TIMESTAMP, 
getRecordId(DATA));
+OutgoingMessage message = new OutgoingMessage(
+DATA.getBytes(), ImmutableMap.of(), TIMESTAMP, 
getRecordId(DATA));
 CoderProperties.coderDecodeEncodeEqual(PubsubUnboundedSink.CODER, message);
 CoderProperties.coderSerializable(PubsubUnboundedSink.CODER);
   }
@@ -82,7 +90,10 @@ public class PubsubUnboundedSinkTest {
   @Category(NeedsRunner.class)
   public void sendOneMessage() throws IOException {
 List outgoing =
-ImmutableList.of(new OutgoingMessage(DATA.getBytes(), TIMESTAMP, 
getRecordId(DATA)));
+ImmutableList.of(new OutgoingMessage(
+DATA.getBytes(),
+ATTRIBUTES,
+TIMESTAMP, getRecordId(DATA)));
 int batchSize = 1;
 int batchBytes = 1;
 try (PubsubTestClientFactory factory =
@@ -91,8 +102,14 @@ public class PubsubUnboundedSinkTest {
   PubsubUnboundedSink sink =
   new PubsubUnboundedSink<>(factory, StaticValueProvider.of(TOPIC), 
StringUtf8Coder.of(),
   TIMESTAMP_LABEL, ID_LABEL, NUM_SHARDS, batchSize, batchBytes,
-  Duration.standardSeconds(2), RecordIdMethod.DETERMINISTIC);
-
+  Duration.standardSeconds(2),
+  new SimpleFunction() {
+@Override
+public PubsubIO.PubsubMessage apply(String input) {
+  return new PubsubIO.PubsubMessage(input.getBytes(), 
ATTRIBUTES);
+}
+  },
+  RecordIdMethod.DETERMINISTIC);
   p.apply(Create.of(ImmutableList.of(DATA)))
.apply(ParDo.of(new Stamp()))
.apply(sink);
@@ -111,7 +128,8 @@ public class PubsubUnboundedSinkTest {
 int batchBytes = 1000;
 for (int i = 0; i < batchSize * 10; i++) {
   String str = String.valueOf(i);
-  outgoing.add(new OutgoingMessage(str.getBytes(), TIMESTAMP, 
getRecordId(str)));
+  outgoing.add(new OutgoingMessage(
+  str.getBytes(), ImmutableMap.of(), TIMESTAMP, 
getRecordId(str)));
   data.add(str);
 }
 try (PubsubTestClientFactory factory =
@@ -120,8 +138,7 @@ public class PubsubUnboundedSinkTest {
   PubsubUnboundedSink sink =
   new PubsubUnboundedSink<>(factory, StaticValueProvider.of(TOPIC), 
StringUtf8Coder.of(),
   TIMESTAMP_LABEL, ID_LABEL, NUM_SHARDS, batchSize, batchBytes,
-  Duration.standardSeconds(2), RecordIdMethod.DETERMIN

[3/3] beam git commit: Closes #1358

[dhalperi]

Closes #1358


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c6e46b65
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c6e46b65
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c6e46b65

Branch: refs/heads/master
Commit: c6e46b655f3a3a1c6003e1bccdb52ae50efa2882
Parents: a681037 f032fac
Author: Dan Halperin 
Authored: Fri Jan 20 14:57:57 2017 -0800
Committer: Dan Halperin 
Committed: Fri Jan 20 14:57:57 2017 -0800

--
 .../beam/examples/complete/game/GameStats.java  |   5 +-
 .../examples/complete/game/LeaderBoard.java |   5 +-
 .../beam/runners/dataflow/DataflowRunner.java   |  30 +-
 .../java/org/apache/beam/sdk/io/PubsubIO.java   | 783 +--
 .../apache/beam/sdk/io/PubsubUnboundedSink.java |  72 +-
 .../beam/sdk/io/PubsubUnboundedSource.java  |  67 +-
 .../apache/beam/sdk/transforms/GroupByKey.java  |   2 +-
 .../org/apache/beam/sdk/util/PropertyNames.java |   1 +
 .../org/apache/beam/sdk/util/PubsubClient.java  |  28 +-
 .../apache/beam/sdk/util/PubsubGrpcClient.java  |   6 +-
 .../apache/beam/sdk/util/PubsubJsonClient.java  |   4 +-
 .../apache/beam/sdk/util/PubsubTestClient.java  |   6 +-
 .../org/apache/beam/sdk/io/PubsubIOTest.java|  41 +-
 .../beam/sdk/io/PubsubUnboundedSinkTest.java|  41 +-
 .../beam/sdk/io/PubsubUnboundedSourceTest.java  |  10 +-
 .../beam/sdk/util/PubsubGrpcClientTest.java |   8 +-
 .../beam/sdk/util/PubsubJsonClientTest.java |   3 +-
 .../beam/sdk/util/PubsubTestClientTest.java |   4 +-
 18 files changed, 643 insertions(+), 473 deletions(-)
--

[GitHub] beam pull request #1358: [BEAM-974] Add PubSub attributes support to PubsubI...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1358


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam git commit: Add PubSub attributes support to PubsubIO.

[dhalperi]

Add PubSub attributes support to PubsubIO.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f032facb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f032facb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f032facb

Branch: refs/heads/master
Commit: f032facb4889f9183f8c6bfeda9fee9c0e4b7979
Parents: a681037
Author: Reuven Lax 
Authored: Sun Nov 13 20:29:31 2016 -0800
Committer: Dan Halperin 
Committed: Fri Jan 20 14:57:56 2017 -0800

--
 .../beam/examples/complete/game/GameStats.java  |   5 +-
 .../examples/complete/game/LeaderBoard.java |   5 +-
 .../beam/runners/dataflow/DataflowRunner.java   |  30 +-
 .../java/org/apache/beam/sdk/io/PubsubIO.java   | 783 +--
 .../apache/beam/sdk/io/PubsubUnboundedSink.java |  72 +-
 .../beam/sdk/io/PubsubUnboundedSource.java  |  67 +-
 .../apache/beam/sdk/transforms/GroupByKey.java  |   2 +-
 .../org/apache/beam/sdk/util/PropertyNames.java |   1 +
 .../org/apache/beam/sdk/util/PubsubClient.java  |  28 +-
 .../apache/beam/sdk/util/PubsubGrpcClient.java  |   6 +-
 .../apache/beam/sdk/util/PubsubJsonClient.java  |   4 +-
 .../apache/beam/sdk/util/PubsubTestClient.java  |   6 +-
 .../org/apache/beam/sdk/io/PubsubIOTest.java|  41 +-
 .../beam/sdk/io/PubsubUnboundedSinkTest.java|  41 +-
 .../beam/sdk/io/PubsubUnboundedSourceTest.java  |  10 +-
 .../beam/sdk/util/PubsubGrpcClientTest.java |   8 +-
 .../beam/sdk/util/PubsubJsonClientTest.java |   3 +-
 .../beam/sdk/util/PubsubTestClientTest.java |   4 +-
 18 files changed, 643 insertions(+), 473 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f032facb/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
--
diff --git 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
index 74f1b30..c880061 100644
--- 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
+++ 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/GameStats.java
@@ -24,6 +24,7 @@ import org.apache.beam.examples.common.ExampleUtils;
 import org.apache.beam.examples.complete.game.utils.WriteWindowedToBigQuery;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.io.PubsubIO;
 import org.apache.beam.sdk.options.Default;
 import org.apache.beam.sdk.options.Description;
@@ -252,7 +253,9 @@ public class GameStats extends LeaderBoard {
 
 // Read Events from Pub/Sub using custom timestamps
 PCollection rawEvents = pipeline
-
.apply(PubsubIO.Read.timestampLabel(TIMESTAMP_ATTRIBUTE).topic(options.getTopic()))
+.apply(PubsubIO.read()
+.timestampLabel(TIMESTAMP_ATTRIBUTE).topic(options.getTopic())
+.withCoder(StringUtf8Coder.of()))
 .apply("ParseGameEvent", ParDo.of(new ParseEventFn()));
 
 // Extract username/score pairs from the event stream

http://git-wip-us.apache.org/repos/asf/beam/blob/f032facb/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
--
diff --git 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
index 519bd5f..35b586b 100644
--- 
a/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
+++ 
b/examples/java8/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java
@@ -27,6 +27,7 @@ import 
org.apache.beam.examples.complete.game.utils.WriteToBigQuery;
 import org.apache.beam.examples.complete.game.utils.WriteWindowedToBigQuery;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.io.PubsubIO;
 import org.apache.beam.sdk.options.Default;
 import org.apache.beam.sdk.options.Description;
@@ -190,7 +191,9 @@ public class LeaderBoard extends HourlyTeamScore {
 // Read game events from Pub/Sub using custom timestamps, which are 
extracted from the pubsub
 // data elements, and parse the data.
 PCollection gameEvents = pipeline
-
.apply(PubsubIO.Read.timestampLabel(TIMESTAMP_ATTRIBUTE).topic(options.getTopic()))
+.apply(PubsubIO.read()
+.timestampLabel(TIMESTAMP_ATTRIBUTE).topic(options.getTopic())
+.withCoder(StringUtf8Coder.of()))
 .apply("ParseGameEvent", ParDo.of(new ParseEventFn()));
 
 gameEvents.apply("CalculateTeamS

[1/4] beam git commit: Add tests for standard beam coder types.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/python-sdk 5588db8d1 -> c57c66ed4


Add tests for standard beam coder types.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1503682c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1503682c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1503682c

Branch: refs/heads/python-sdk
Commit: 1503682c26db74bd6e84b6042a4227a4c2af5846
Parents: 5588db8
Author: Robert Bradshaw 
Authored: Tue Jan 10 15:47:58 2017 -0800
Committer: Robert Bradshaw 
Committed: Fri Jan 20 14:48:05 2017 -0800

--
 .../apache_beam/coders/standard_coders.yaml |  25 +
 .../apache_beam/coders/standard_coders_test.py  | 108 +++
 2 files changed, 133 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1503682c/sdks/python/apache_beam/coders/standard_coders.yaml
--
diff --git a/sdks/python/apache_beam/coders/standard_coders.yaml 
b/sdks/python/apache_beam/coders/standard_coders.yaml
new file mode 100644
index 000..77db379
--- /dev/null
+++ b/sdks/python/apache_beam/coders/standard_coders.yaml
@@ -0,0 +1,25 @@
+coder:
+  urn: "beam:coders:bytes:0.1"
+nested: false
+examples:
+  abc: abc
+  "ab\0c": "ab\0c"
+
+---
+
+coder:
+  urn: "beam:coders:varint:0.1"
+examples:
+  "\0": 0
+  "\u0001": 1
+  "\u000A": 10
+
+---
+
+coder:
+  urn: "beam:coders:kv:0.1"
+  components: [{urn: "beam:coders:bytes:0.1"},
+   {urn: "beam:coders:varint:0.1"}]
+examples:
+  "\u0003abc\0": {key: abc, value: 0}
+  "\u0004ab\0c\u000A": {key: "ab\0c", value: 10}

http://git-wip-us.apache.org/repos/asf/beam/blob/1503682c/sdks/python/apache_beam/coders/standard_coders_test.py
--
diff --git a/sdks/python/apache_beam/coders/standard_coders_test.py 
b/sdks/python/apache_beam/coders/standard_coders_test.py
new file mode 100644
index 000..18dc0a7
--- /dev/null
+++ b/sdks/python/apache_beam/coders/standard_coders_test.py
@@ -0,0 +1,108 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for coders that must be consistent across all beam SDKs.
+"""
+
+import collections
+import os.path
+import pickle
+import unittest
+
+import yaml
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.coders import coder_impl
+
+
+class StandardCodersTest(unittest.TestCase):
+
+  _urn_to_coder_class = {
+'beam:coders:bytes:0.1': coders.BytesCoder,
+'beam:coders:varint:0.1': coders.VarIntCoder,
+'beam:coders:kv:0.1': lambda k, v: coders.TupleCoder((k, v))
+  }
+
+  _urn_to_json_value_parser = {
+'beam:coders:bytes:0.1': lambda x: x,
+'beam:coders:varint:0.1': lambda x: x,
+'beam:coders:kv:0.1':
+lambda x, key_parser, value_parser: (key_parser(x['key']),
+ value_parser(x['value']))
+  }
+
+  # We must prepend an underscore to this name so that the open-source unittest
+  # runner does not execute this method directly as a test.
+  @classmethod
+  def _create_test(cls, spec):
+counter = 0
+name = spec.get('name', spec['coder']['urn'].split(':')[-2])
+unique_name = 'test_' + name
+while hasattr(cls, unique_name):
+  counter += 1
+  unique_name = 'test_%s_%d' % (name, counter)
+setattr(cls, unique_name, lambda self: self._run_coder_test(spec))
+
+  # We must prepend an underscore to this name so that the open-source unittest
+  # runner does not execute this method directly as a test.
+  @classmethod
+  def _create_tests(cls, coder_test_specs):
+for spec in yaml.load_all(open(coder_test_specs)):
+  cls._create_test(spec)
+
+  def _run_coder_test(self, spec):
+coder = self.parse_coder(spec['coder'])
+parse_value = self.json_value_parser(spec['coder'])
+for encoded, json_value in spec['examples'].items():
+  value = parse_value(json_value)
+  if spec.get('nested', True):
+self.assertEqual(decode_nested(coder, encoded), value)
+s

[3/4] beam git commit: A couple more examples.

[robertwb]

A couple more examples.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7e5dc400
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7e5dc400
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7e5dc400

Branch: refs/heads/python-sdk
Commit: 7e5dc40018520fd727c29df81396d7029e0a3358
Parents: 5c054ae
Author: Robert Bradshaw 
Authored: Tue Jan 10 16:49:33 2017 -0800
Committer: Robert Bradshaw 
Committed: Fri Jan 20 14:48:07 2017 -0800

--
 sdks/python/apache_beam/coders/standard_coders.yaml | 16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/7e5dc400/sdks/python/apache_beam/coders/standard_coders.yaml
--
diff --git a/sdks/python/apache_beam/coders/standard_coders.yaml 
b/sdks/python/apache_beam/coders/standard_coders.yaml
index 3193883..34e4336 100644
--- a/sdks/python/apache_beam/coders/standard_coders.yaml
+++ b/sdks/python/apache_beam/coders/standard_coders.yaml
@@ -20,17 +20,31 @@ coder:
   urn: "beam:coders:bytes:0.1"
 nested: false
 examples:
-  abc: abc
+  "abc": abc
   "ab\0c": "ab\0c"
 
 ---
 
 coder:
+  urn: "beam:coders:bytes:0.1"
+nested: true
+examples:
+  "\u0003abc": abc
+  "\u0004ab\0c": "ab\0c"
+  "\u00c8\u0001   10|   20|   30|   40|   50|   60|
   70|   80|   90|  100|  110|  120|  130|  140|
  150|  160|  170|  180|  190|  200|":
+  "   10|   20|   30|   40|   50|   60|
   70|   80|   90|  100|  110|  120|  130|  140|
  150|  160|  170|  180|  190|  200|"
+
+---
+
+coder:
   urn: "beam:coders:varint:0.1"
 examples:
   "\0": 0
   "\u0001": 1
   "\u000A": 10
+  "\u00c8\u0001": 200
+  "\u00e8\u0007": 1000
+  "\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u00ff\u0001": -1
 
 ---
 

[4/4] beam git commit: Closes #1764

[robertwb]

Closes #1764


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c57c66ed
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c57c66ed
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c57c66ed

Branch: refs/heads/python-sdk
Commit: c57c66ed4f1252388c13abb0d8a6cd02a2c803df
Parents: 5588db8 7e5dc40
Author: Robert Bradshaw 
Authored: Fri Jan 20 14:48:20 2017 -0800
Committer: Robert Bradshaw 
Committed: Fri Jan 20 14:48:20 2017 -0800

--
 sdks/python/apache_beam/coders/coder_impl.py|   3 +
 .../apache_beam/coders/standard_coders.yaml |  67 +
 .../apache_beam/coders/standard_coders_test.py  | 136 +++
 3 files changed, 206 insertions(+)
--

[2/4] beam git commit: Add a --fix option to the standard coder test that populates encodings.

[robertwb]

Add a --fix option to the standard coder test that populates encodings.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5c054aef
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5c054aef
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5c054aef

Branch: refs/heads/python-sdk
Commit: 5c054aeffd6b5eba8cb62065dd626d33cae0312a
Parents: 1503682
Author: Robert Bradshaw 
Authored: Tue Jan 10 16:33:23 2017 -0800
Committer: Robert Bradshaw 
Committed: Fri Jan 20 14:48:07 2017 -0800

--
 sdks/python/apache_beam/coders/coder_impl.py|  3 +
 .../apache_beam/coders/standard_coders.yaml | 28 +++
 .../apache_beam/coders/standard_coders_test.py  | 78 +---
 3 files changed, 84 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/5c054aef/sdks/python/apache_beam/coders/coder_impl.py
--
diff --git a/sdks/python/apache_beam/coders/coder_impl.py 
b/sdks/python/apache_beam/coders/coder_impl.py
index fcdc441..96d0a3e 100644
--- a/sdks/python/apache_beam/coders/coder_impl.py
+++ b/sdks/python/apache_beam/coders/coder_impl.py
@@ -37,6 +37,9 @@ try:
   from stream import OutputStream as create_OutputStream
   from stream import ByteCountingOutputStream
   from stream import get_varint_size
+  globals()['create_InputStream'] = create_InputStream
+  globals()['create_OutputStream'] = create_OutputStream
+  globals()['ByteCountingOutputStream'] = ByteCountingOutputStream
 except ImportError:
   from slow_stream import InputStream as create_InputStream
   from slow_stream import OutputStream as create_OutputStream

http://git-wip-us.apache.org/repos/asf/beam/blob/5c054aef/sdks/python/apache_beam/coders/standard_coders.yaml
--
diff --git a/sdks/python/apache_beam/coders/standard_coders.yaml 
b/sdks/python/apache_beam/coders/standard_coders.yaml
index 77db379..3193883 100644
--- a/sdks/python/apache_beam/coders/standard_coders.yaml
+++ b/sdks/python/apache_beam/coders/standard_coders.yaml
@@ -1,3 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
 coder:
   urn: "beam:coders:bytes:0.1"
 nested: false
@@ -23,3 +41,13 @@ coder:
 examples:
   "\u0003abc\0": {key: abc, value: 0}
   "\u0004ab\0c\u000A": {key: "ab\0c", value: 10}
+
+---
+
+coder:
+  urn: "beam:coders:kv:0.1"
+  components: [{urn: "beam:coders:bytes:0.1"},
+   {urn: "beam:coders:bytes:0.1"}]
+examples:
+  "\u0003abc\u0003def": {key: abc, value: def}
+  "\u0004ab\0c\u0004de\0f": {key: "ab\0c", value: "de\0f"}

http://git-wip-us.apache.org/repos/asf/beam/blob/5c054aef/sdks/python/apache_beam/coders/standard_coders_test.py
--
diff --git a/sdks/python/apache_beam/coders/standard_coders_test.py 
b/sdks/python/apache_beam/coders/standard_coders_test.py
index 18dc0a7..d451132 100644
--- a/sdks/python/apache_beam/coders/standard_coders_test.py
+++ b/sdks/python/apache_beam/coders/standard_coders_test.py
@@ -15,17 +15,16 @@
 # limitations under the License.
 #
 
-"""Unit tests for coders that must be consistent across all beam SDKs.
+"""Unit tests for coders that must be consistent across all Beam SDKs.
 """
 
-import collections
+import json
 import os.path
-import pickle
+import sys
 import unittest
 
 import yaml
 
-import apache_beam as beam
 from apache_beam import coders
 from apache_beam.coders import coder_impl
 
@@ -33,17 +32,17 @@ from apache_beam.coders import coder_impl
 class StandardCodersTest(unittest.TestCase):
 
   _urn_to_coder_class = {
-'beam:coders:bytes:0.1': coders.BytesCoder,
-'beam:coders:varint:0.1': coders.VarIntCoder,
-'beam:coders:kv:0.1': lambda k, v: coders.TupleCoder((k, v))
+  'beam:coders:bytes:0.1': coders.BytesCoder,
+  'beam:coders:varint:0.1': coders.VarIntCoder,
+  'beam:coders:kv:0.1': lambda k, v: coders.TupleCoder((k, v))
   }
 
   _urn_to_json_value_parser = {
-'beam:coders:bytes:0.1':

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832533#comment-15832533
 ] 

ASF GitHub Bot commented on BEAM-498:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1636


> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: 0.3.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/4] beam git commit: Moves OldDoFn to runners-core

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 77c7505b8 -> a6810372b


http://git-wip-us.apache.org/repos/asf/beam/blob/5f8b8c5b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java
deleted file mode 100644
index 7b04533..000
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java
+++ /dev/null
@@ -1,465 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.transforms;
-
-import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.base.Preconditions.checkNotNull;
-import static com.google.common.base.Preconditions.checkState;
-
-import java.io.Serializable;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.Map;
-import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.annotations.Experimental.Kind;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.runners.PipelineRunner;
-import org.apache.beam.sdk.transforms.Combine.CombineFn;
-import org.apache.beam.sdk.transforms.display.DisplayData;
-import org.apache.beam.sdk.transforms.display.HasDisplayData;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.WindowingInternals;
-import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.sdk.values.TupleTag;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-
-/**
- * The argument to {@link ParDo} providing the code to use to process
- * elements of the input
- * {@link org.apache.beam.sdk.values.PCollection}.
- *
- * See {@link ParDo} for more explanation, examples of use, and
- * discussion of constraints on {@code OldDoFn}s, including their
- * serializability, lack of access to global shared mutable state,
- * requirements for failure tolerance, and benefits of optimization.
- *
- * {@code OldDoFn}s can be tested in the context of a particular
- * {@code Pipeline} by running that {@code Pipeline} on sample input
- * and then checking its output.  Unit testing of a {@code OldDoFn},
- * separately from any {@code ParDo} transform or {@code Pipeline},
- * can be done via the {@link DoFnTester} harness.
- *
- * {@link DoFn} (currently experimental) offers an alternative
- * mechanism for accessing {@link ProcessContext#window()} without the need
- * to implement {@link RequiresWindowAccess}.
- *
- * See also {@link #processElement} for details on implementing the 
transformation
- * from {@code InputT} to {@code OutputT}.
- *
- * @param  the type of the (main) input elements
- * @param  the type of the (main) output elements
- * @deprecated Uses of {@link OldDoFn} should be replaced by the new {@link 
DoFn}.
- */
-@Deprecated
-public abstract class OldDoFn implements Serializable, 
HasDisplayData {
-  /**
-   * Information accessible to all methods in this {@code OldDoFn}.
-   * Used primarily to output elements.
-   */
-  public abstract class Context {
-
-/**
- * Returns the {@code PipelineOptions} specified with the
- * {@link org.apache.beam.sdk.runners.PipelineRunner}
- * invoking this {@code OldDoFn}.  The {@code PipelineOptions} will
- * be the default running via {@link DoFnTester}.
- */
-public abstract PipelineOptions getPipelineOptions();
-
-/**
- * Adds the given element to the main output {@code PCollection}.
- *
- * Once passed to {@code output} the element should be considered
- * immutable and not be modified in any way. It may be cached or retained
- * by a Beam runner or later steps in the pipeline, or used in
- * other unspecified ways.
- *
- * If invoked from {@link OldDoFn#processElement processElement}, the 
output
- * element will have the same timestamp and be in the same windows
- * as the input element passed to {@link OldDoFn#processElement 
processElement}.
- *

[GitHub] beam pull request #1636: [BEAM-498] Moves OldDoFn to runners-core

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1636


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/4] beam git commit: Point to new Dataflow worker

[kenn]

Point to new Dataflow worker


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a2b94eca
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a2b94eca
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a2b94eca

Branch: refs/heads/master
Commit: a2b94ecabd80ef68654ba3bc8ffa0a0c3759316b
Parents: 5f8b8c5
Author: Eugene Kirpichov 
Authored: Fri Jan 20 13:31:30 2017 -0800
Committer: Eugene Kirpichov 
Committed: Fri Jan 20 13:32:19 2017 -0800

--
 .../org/apache/beam/runners/dataflow/dataflow.properties | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a2b94eca/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
 
b/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
index 161a897..9976ed9 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
+++ 
b/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
@@ -18,6 +18,6 @@
 
 environment.major.version=6
 
-worker.image.batch=dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20170119
+worker.image.batch=dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20170120
 
-worker.image.streaming=dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20170119
+worker.image.streaming=dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20170120

[2/4] beam git commit: Moves OldDoFn to runners-core

[kenn]

Moves OldDoFn to runners-core


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5f8b8c5b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5f8b8c5b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5f8b8c5b

Branch: refs/heads/master
Commit: 5f8b8c5b06cfd49c4293a20dff2eea99f1076444
Parents: 77c7505
Author: Eugene Kirpichov 
Authored: Tue Jan 17 16:12:39 2017 -0800
Committer: Eugene Kirpichov 
Committed: Fri Jan 20 13:31:58 2017 -0800

--
 .../apex/translation/WindowBoundTranslator.java |   2 +-
 .../operators/ApexGroupByKeyOperator.java   |   2 +-
 .../operators/ApexParDoOperator.java|   2 +-
 .../beam/runners/core/AssignWindowsDoFn.java|   3 +-
 .../apache/beam/runners/core/DoFnAdapters.java  |   1 -
 .../apache/beam/runners/core/DoFnRunner.java|   1 -
 .../apache/beam/runners/core/DoFnRunners.java   |   1 -
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |   1 -
 .../runners/core/GroupAlsoByWindowsDoFn.java|   1 -
 .../core/LateDataDroppingDoFnRunner.java|   1 -
 .../org/apache/beam/runners/core/OldDoFn.java   | 472 +++
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   3 +-
 .../core/DoFnDelegatingAggregatorTest.java  | 144 ++
 .../apache/beam/runners/core/NoOpOldDoFn.java   |  72 +++
 .../beam/runners/core/OldDoFnContextTest.java   |  72 +++
 .../apache/beam/runners/core/OldDoFnTest.java   | 192 
 .../runners/core/SimpleOldDoFnRunnerTest.java   |   2 +-
 .../runners/flink/OldPerKeyCombineFnRunner.java |   2 +-
 .../flink/OldPerKeyCombineFnRunners.java|   2 +-
 .../functions/FlinkDoFnFunction.java|   2 +-
 .../FlinkMergingNonShuffleReduceFunction.java   |   2 +-
 .../functions/FlinkMultiOutputDoFnFunction.java |   2 +-
 .../FlinkMultiOutputProcessContext.java |   2 +-
 .../functions/FlinkNoElementAssignContext.java  |   2 +-
 .../functions/FlinkPartialReduceFunction.java   |   2 +-
 .../functions/FlinkProcessContextBase.java  |   2 +-
 .../functions/FlinkReduceFunction.java  |   2 +-
 .../FlinkSingleOutputProcessContext.java|   2 +-
 .../wrappers/streaming/DoFnOperator.java|   2 +-
 .../wrappers/streaming/WindowDoFnOperator.java  |   2 +-
 .../beam/runners/dataflow/util/DoFnInfo.java|   4 +-
 .../sdk/transforms/DelegatingAggregator.java|   2 +-
 .../org/apache/beam/sdk/transforms/OldDoFn.java | 465 --
 .../org/apache/beam/sdk/util/NameUtils.java |   2 +-
 .../DoFnDelegatingAggregatorTest.java   | 142 --
 .../apache/beam/sdk/transforms/NoOpOldDoFn.java |  71 ---
 .../beam/sdk/transforms/OldDoFnContextTest.java |  69 ---
 .../apache/beam/sdk/transforms/OldDoFnTest.java | 187 
 .../org/apache/beam/sdk/util/NameUtilsTest.java |  20 +-
 39 files changed, 982 insertions(+), 978 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/5f8b8c5b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
index ef049e1..50af81d 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
@@ -23,8 +23,8 @@ import org.apache.beam.runners.apex.ApexPipelineOptions;
 import org.apache.beam.runners.apex.translation.operators.ApexParDoOperator;
 import org.apache.beam.runners.core.AssignWindowsDoFn;
 import org.apache.beam.runners.core.DoFnAdapters;
+import org.apache.beam.runners.core.OldDoFn;
 import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.util.WindowedValue;

http://git-wip-us.apache.org/repos/asf/beam/blob/5f8b8c5b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
index 173434f..4c2b461 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
@@ -44,6 +44,

[4/4] beam git commit: This closes #1636: Move OldDoFn to runners-core

[kenn]

This closes #1636: Move OldDoFn to runners-core

  Point to new Dataflow worker
  Moves OldDoFn to runners-core


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a6810372
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a6810372
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a6810372

Branch: refs/heads/master
Commit: a6810372b003adf24bdbe34ed764a63841af9b99
Parents: 77c7505 a2b94ec
Author: Kenneth Knowles 
Authored: Fri Jan 20 14:22:38 2017 -0800
Committer: Kenneth Knowles 
Committed: Fri Jan 20 14:22:38 2017 -0800

--
 .../apex/translation/WindowBoundTranslator.java |   2 +-
 .../operators/ApexGroupByKeyOperator.java   |   2 +-
 .../operators/ApexParDoOperator.java|   2 +-
 .../beam/runners/core/AssignWindowsDoFn.java|   3 +-
 .../apache/beam/runners/core/DoFnAdapters.java  |   1 -
 .../apache/beam/runners/core/DoFnRunner.java|   1 -
 .../apache/beam/runners/core/DoFnRunners.java   |   1 -
 .../core/GroupAlsoByWindowViaWindowSetDoFn.java |   1 -
 .../runners/core/GroupAlsoByWindowsDoFn.java|   1 -
 .../core/LateDataDroppingDoFnRunner.java|   1 -
 .../org/apache/beam/runners/core/OldDoFn.java   | 472 +++
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   3 +-
 .../core/DoFnDelegatingAggregatorTest.java  | 144 ++
 .../apache/beam/runners/core/NoOpOldDoFn.java   |  72 +++
 .../beam/runners/core/OldDoFnContextTest.java   |  72 +++
 .../apache/beam/runners/core/OldDoFnTest.java   | 192 
 .../runners/core/SimpleOldDoFnRunnerTest.java   |   2 +-
 .../runners/flink/OldPerKeyCombineFnRunner.java |   2 +-
 .../flink/OldPerKeyCombineFnRunners.java|   2 +-
 .../functions/FlinkDoFnFunction.java|   2 +-
 .../FlinkMergingNonShuffleReduceFunction.java   |   2 +-
 .../functions/FlinkMultiOutputDoFnFunction.java |   2 +-
 .../FlinkMultiOutputProcessContext.java |   2 +-
 .../functions/FlinkNoElementAssignContext.java  |   2 +-
 .../functions/FlinkPartialReduceFunction.java   |   2 +-
 .../functions/FlinkProcessContextBase.java  |   2 +-
 .../functions/FlinkReduceFunction.java  |   2 +-
 .../FlinkSingleOutputProcessContext.java|   2 +-
 .../wrappers/streaming/DoFnOperator.java|   2 +-
 .../wrappers/streaming/WindowDoFnOperator.java  |   2 +-
 .../beam/runners/dataflow/util/DoFnInfo.java|   4 +-
 .../beam/runners/dataflow/dataflow.properties   |   4 +-
 .../sdk/transforms/DelegatingAggregator.java|   2 +-
 .../org/apache/beam/sdk/transforms/OldDoFn.java | 465 --
 .../org/apache/beam/sdk/util/NameUtils.java |   2 +-
 .../DoFnDelegatingAggregatorTest.java   | 142 --
 .../apache/beam/sdk/transforms/NoOpOldDoFn.java |  71 ---
 .../beam/sdk/transforms/OldDoFnContextTest.java |  69 ---
 .../apache/beam/sdk/transforms/OldDoFnTest.java | 187 
 .../org/apache/beam/sdk/util/NameUtilsTest.java |  20 +-
 40 files changed, 984 insertions(+), 980 deletions(-)
--

[jira] [Commented] (BEAM-886) Support new DoFn in Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832500#comment-15832500
 ] 

ASF GitHub Bot commented on BEAM-886:
-

Github user sb2nov closed the pull request at:

https://github.com/apache/beam/pull/1782


> Support new DoFn in Python SDK
> --
>
> Key: BEAM-886
> URL: https://issues.apache.org/jira/browse/BEAM-886
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: backward-incompatible, sdk-consistency
>
> Figure out what is needed for supporting new DoFns, add support and removed 
> old DoFns.
> Related Docs from Java:
> Original Proposal email:
> https://lists.apache.org/thread.html/2abf32d528dbb64b79853552c5d10c217e2194f0685af21aeb4635dd@%3Cdev.beam.apache.org%3E
> Presentation & Doc (with short Python sections):
> https://s.apache.org/presenting-a-new-dofn
> https://s.apache.org/a-new-dofn



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1782: [BEAM-886] Implement annotation based NewDoFn in Py...

[sb2nov]

Github user sb2nov closed the pull request at:

https://github.com/apache/beam/pull/1782


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-886) Support new DoFn in Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832499#comment-15832499
 ] 

ASF GitHub Bot commented on BEAM-886:
-

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/1805

[BEAM-886] Implement Annotation based NewDoFn in python SDK

- Implement the new annotation based DoFn in python see 
https://s.apache.org/a-new-dofn
- Handle typechecking in the NewDoFn
- All unittests pass
- None of the actual DoFn use the new ones yet

Future work:
- Performance test New Vs Old DoFn
- Migrate the OldDoFn to New
- Remove OldDoFn from the code
- Rename NewDoFn to DoFn
- Deprecate use of context 

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @robertwb  PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam 
BEAM-886-Add-NewDoFn-class-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1805.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1805


commit f8ae4f2ccc399c6c0952f903b5e140986c61a933
Author: Sourabh Bajaj 
Date:   2017-01-20T22:17:00Z

Implement Annotation based NewDoFn in python SDK




> Support new DoFn in Python SDK
> --
>
> Key: BEAM-886
> URL: https://issues.apache.org/jira/browse/BEAM-886
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: backward-incompatible, sdk-consistency
>
> Figure out what is needed for supporting new DoFns, add support and removed 
> old DoFns.
> Related Docs from Java:
> Original Proposal email:
> https://lists.apache.org/thread.html/2abf32d528dbb64b79853552c5d10c217e2194f0685af21aeb4635dd@%3Cdev.beam.apache.org%3E
> Presentation & Doc (with short Python sections):
> https://s.apache.org/presenting-a-new-dofn
> https://s.apache.org/a-new-dofn



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1805: [BEAM-886] Implement Annotation based NewDoFn in py...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/1805

[BEAM-886] Implement Annotation based NewDoFn in python SDK

- Implement the new annotation based DoFn in python see 
https://s.apache.org/a-new-dofn
- Handle typechecking in the NewDoFn
- All unittests pass
- None of the actual DoFn use the new ones yet

Future work:
- Performance test New Vs Old DoFn
- Migrate the OldDoFn to New
- Remove OldDoFn from the code
- Rename NewDoFn to DoFn
- Deprecate use of context 

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @robertwb  PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam 
BEAM-886-Add-NewDoFn-class-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1805.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1805


commit f8ae4f2ccc399c6c0952f903b5e140986c61a933
Author: Sourabh Bajaj 
Date:   2017-01-20T22:17:00Z

Implement Annotation based NewDoFn in python SDK




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2066

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1291) KafkaIO: don't log warning in offset fetcher while closing the reader.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832466#comment-15832466
 ] 

ASF GitHub Bot commented on BEAM-1291:
--

GitHub user rangadi opened a pull request:

https://github.com/apache/beam/pull/1804

[BEAM-1291] KafkaIO: don't log warnig in offset fetcher while closing.

Don't log a warning in offset fetcher thread if the reader is already 
closed. This avoids misleading warning in logs (especially with spark runner 
where the reader is closed after each micro-batch).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rangadi/incubator-beam no_warn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1804.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1804


commit 44f17560122a09bad4e866495037daa309fb9ea8
Author: Raghu Angadi 
Date:   2017-01-20T21:24:09Z

KafkaIO: don't log warnig in offset fetcher while closing.




> KafkaIO: don't log warning in offset fetcher while closing the reader.
> --
>
> Key: BEAM-1291
> URL: https://issues.apache.org/jira/browse/BEAM-1291
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Minor
>
> When the KafkaIO reader is closed, it wakes up two background threads used 
> for reading from Kafka. The wakeup notification results in an exception. One 
> of the threads logs a warning with full stacktrace even though it is part of 
> normal operation. It should not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1804: [BEAM-1291] KafkaIO: don't log warnig in offset fet...

[rangadi]

GitHub user rangadi opened a pull request:

https://github.com/apache/beam/pull/1804

[BEAM-1291] KafkaIO: don't log warnig in offset fetcher while closing.

Don't log a warning in offset fetcher thread if the reader is already 
closed. This avoids misleading warning in logs (especially with spark runner 
where the reader is closed after each micro-batch).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rangadi/incubator-beam no_warn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1804.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1804


commit 44f17560122a09bad4e866495037daa309fb9ea8
Author: Raghu Angadi 
Date:   2017-01-20T21:24:09Z

KafkaIO: don't log warnig in offset fetcher while closing.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1291) KafkaIO: don't log warning in offset fetcher while closing the reader.

[Raghu Angadi (JIRA)]

Raghu Angadi created BEAM-1291:
--

 Summary: KafkaIO: don't log warning in offset fetcher while 
closing the reader.
 Key: BEAM-1291
 URL: https://issues.apache.org/jira/browse/BEAM-1291
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Raghu Angadi
Assignee: Raghu Angadi
Priority: Minor


When the KafkaIO reader is closed, it wakes up two background threads used for 
reading from Kafka. The wakeup notification results in an exception. One of the 
threads logs a warning with full stacktrace even though it is part of normal 
operation. It should not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1792: [BEAM-1113] Set USES_KEYED_STATE for timers in Data...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1792


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1113) Support for new Timer API in Dataflow runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832396#comment-15832396
 ] 

ASF GitHub Bot commented on BEAM-1113:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1792


> Support for new Timer API in Dataflow runner
> 
>
> Key: BEAM-1113
> URL: https://issues.apache.org/jira/browse/BEAM-1113
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/4] beam git commit: DoFnSignature: deprecate isStateful in favor of usesState

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master eecbcc48b -> 77c7505b8


DoFnSignature: deprecate isStateful in favor of usesState


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ed434ecf
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ed434ecf
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ed434ecf

Branch: refs/heads/master
Commit: ed434ecfdbcfc339815c85aa15852124bb2d3981
Parents: eecbcc4
Author: Kenneth Knowles 
Authored: Fri Jan 20 11:28:45 2017 -0800
Committer: Kenneth Knowles 
Committed: Fri Jan 20 11:35:10 2017 -0800

--
 .../beam/runners/dataflow/DataflowPipelineTranslator.java |  2 +-
 .../apache/beam/sdk/transforms/reflect/DoFnSignature.java | 10 +++---
 2 files changed, 8 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ed434ecf/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index 7609745..cfb3d1a 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -943,7 +943,7 @@ public class DataflowPipelineTranslator {
 DoFnInfo.forFn(
 fn, windowingStrategy, sideInputs, inputCoder, mainOutput, 
outputMap;
 
-if (signature.isStateful()) {
+if (signature.usesState()) {
   stepContext.addInput(PropertyNames.USES_KEYED_STATE, "true");
 }
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/ed434ecf/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
index ccc9ac3..5255adc 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
@@ -105,13 +105,17 @@ public abstract class DoFnSignature {
   @Nullable
   public abstract Map onTimerMethods();
 
-  /**
-   * Whether the {@link DoFn} described by this signature uses state.
-   */
+  /** @deprecated use {@link #usesState()}, it's cleaner */
+  @Deprecated
   public boolean isStateful() {
 return stateDeclarations().size() > 0;
   }
 
+  /** Whether the {@link DoFn} described by this signature uses state. */
+  public boolean usesState() {
+return stateDeclarations().size() > 0;
+  }
+
   static Builder builder() {
 return new AutoValue_DoFnSignature.Builder();
   }

[4/4] beam git commit: This closes #1792: Dataflow USES_KEYED_STATE when DoFn uses timers

[kenn]

This closes #1792: Dataflow USES_KEYED_STATE when DoFn uses timers

  Set USES_KEYED_STATE in Dataflow when DoFn uses timers
  Add DoFnSignature.usesTimers() shorthand
  DoFnSignature: deprecate isStateful in favor of usesState


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/77c7505b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/77c7505b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/77c7505b

Branch: refs/heads/master
Commit: 77c7505b870a8ad7c56053fec84a28eb5b8e7dfa
Parents: eecbcc4 f535d65
Author: Kenneth Knowles 
Authored: Fri Jan 20 12:52:07 2017 -0800
Committer: Kenneth Knowles 
Committed: Fri Jan 20 12:52:07 2017 -0800

--
 .../runners/dataflow/DataflowPipelineTranslator.java |  2 +-
 .../beam/sdk/transforms/reflect/DoFnSignature.java   | 15 ---
 2 files changed, 13 insertions(+), 4 deletions(-)
--

[2/4] beam git commit: Add DoFnSignature.usesTimers() shorthand

[kenn]

Add DoFnSignature.usesTimers() shorthand


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/978ff559
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/978ff559
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/978ff559

Branch: refs/heads/master
Commit: 978ff559f68f1a122346bdbe1a44d2895b598191
Parents: ed434ec
Author: Kenneth Knowles 
Authored: Wed Jan 18 16:25:51 2017 -0800
Committer: Kenneth Knowles 
Committed: Fri Jan 20 11:36:39 2017 -0800

--
 .../org/apache/beam/sdk/transforms/reflect/DoFnSignature.java   | 5 +
 1 file changed, 5 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/978ff559/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
index 5255adc..f470782 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java
@@ -116,6 +116,11 @@ public abstract class DoFnSignature {
 return stateDeclarations().size() > 0;
   }
 
+  /** Whether the {@link DoFn} described by this signature uses timers. */
+  public boolean usesTimers() {
+return timerDeclarations().size() > 0;
+  }
+
   static Builder builder() {
 return new AutoValue_DoFnSignature.Builder();
   }

[3/4] beam git commit: Set USES_KEYED_STATE in Dataflow when DoFn uses timers

[kenn]

Set USES_KEYED_STATE in Dataflow when DoFn uses timers


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f535d658
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f535d658
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f535d658

Branch: refs/heads/master
Commit: f535d658b551f56ebe4f2c77948fef63be7f44dc
Parents: 978ff55
Author: Kenneth Knowles 
Authored: Fri Jan 20 11:36:52 2017 -0800
Committer: Kenneth Knowles 
Committed: Fri Jan 20 11:36:52 2017 -0800

--
 .../apache/beam/runners/dataflow/DataflowPipelineTranslator.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f535d658/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index cfb3d1a..babbe69 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -943,7 +943,7 @@ public class DataflowPipelineTranslator {
 DoFnInfo.forFn(
 fn, windowingStrategy, sideInputs, inputCoder, mainOutput, 
outputMap;
 
-if (signature.usesState()) {
+if (signature.usesState() || signature.usesTimers()) {
   stepContext.addInput(PropertyNames.USES_KEYED_STATE, "true");
 }
   }

[jira] [Created] (BEAM-1290) Consider rehtrowing the original exception from the waitUntilFinish

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-1290:
-

 Summary: Consider rehtrowing the original exception from the 
waitUntilFinish
 Key: BEAM-1290
 URL: https://issues.apache.org/jira/browse/BEAM-1290
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-core, sdk-py
Reporter: Ahmet Altay
Priority: Minor


Failing to check status is error prone, and for the currently supported sdk 
languages it is normal to raise an exception to signal abnormal behavior.

Could we consider changing waitUntilFinish to rethrow the original exception 
when the execution fails during the wait.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] beam git commit: [BEAM-1285] Apache rat plugin will print out files with unapproved licenses to console

[davor]

Repository: beam
Updated Branches:
  refs/heads/master 487019961 -> eecbcc48b


[BEAM-1285] Apache rat plugin will print out files with unapproved licenses to 
console


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fa3203d7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fa3203d7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fa3203d7

Branch: refs/heads/master
Commit: fa3203d735db3f53a82e1b4784fb255bae89c154
Parents: 4870199
Author: Luke Cwik 
Authored: Thu Jan 19 14:07:46 2017 -0800
Committer: Davor Bonaci 
Committed: Fri Jan 20 10:22:50 2017 -0800

--
 pom.xml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/fa3203d7/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 3e4f5e9..a96275c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -953,6 +953,7 @@
   
 
${project.build.directory}/${project.build.finalName}.rat
 false
+true
 true
 
   

[GitHub] beam pull request #1799: [BEAM-1285] Apache rat plugin will print out files ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1799


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1285) Apache Rat Plugin should list files with invalid licenses in the console

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832205#comment-15832205
 ] 

ASF GitHub Bot commented on BEAM-1285:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1799


> Apache Rat Plugin should list files with invalid licenses in the console
> 
>
> Key: BEAM-1285
> URL: https://issues.apache.org/jira/browse/BEAM-1285
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> The rat error report doesn't list the files with unapproved licenses in 
> Jenkins making it difficult to diagnose without rerunning locally. The file 
> is output to an inaccessible location on Jenkins.
> Example error message:
> 2017-01-18T23:29:03.041 [ERROR] Failed to execute goal 
> org.apache.rat:apache-rat-plugin:0.12:check (default) on project beam-parent: 
> Too many files with unapproved license: 1 See RAT report in: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/target/beam-parent-0.5.0-incubating-SNAPSHOT.rat
>  -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project 
> beam-parent: Too many files with unapproved license: 1 See RAT report in: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/target/beam-parent-0.5.0-incubating-SNAPSHOT.rat



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1289) TextIO (and others) should provide more informative messages when encountering RVP

[Ben Chambers (JIRA)]

Ben Chambers created BEAM-1289:
--

 Summary: TextIO (and others) should provide more informative 
messages when encountering RVP
 Key: BEAM-1289
 URL: https://issues.apache.org/jira/browse/BEAM-1289
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Ben Chambers
Assignee: Davor Bonaci
Priority: Minor


https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L301

When reporting failing to validate because a parameter comes from a 
RuntimeValueProvider (such as when creating templates) we report:

"Cannot validate with a RVP"

We should instead provide a message saying what went wrong and how to fix it. 
Something like:

"Unable to validate parameters that aren't available until runtime. Disable 
validation using .withoutValidation()."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] beam git commit: This closes #1799

[davor]

This closes #1799


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/eecbcc48
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/eecbcc48
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/eecbcc48

Branch: refs/heads/master
Commit: eecbcc48bb84e74b37391ba58d0bbf1cc5d9423c
Parents: 4870199 fa3203d
Author: Davor Bonaci 
Authored: Fri Jan 20 10:22:54 2017 -0800
Committer: Davor Bonaci 
Committed: Fri Jan 20 10:22:54 2017 -0800

--
 pom.xml | 1 +
 1 file changed, 1 insertion(+)
--

[jira] [Updated] (BEAM-1286) DataflowRunner handling of missing filesToStage

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-1286:
---
Assignee: (was: Davor Bonaci)

> DataflowRunner handling of missing filesToStage
> ---
>
> Key: BEAM-1286
> URL: https://issues.apache.org/jira/browse/BEAM-1286
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>  Labels: starter
>
> DataflowRunner allows filesToStage to be missing -- it logs an error and 
> moves on. Is this the right behavior? It can complicate user experience.
> At least, I guess that if nothing to be staged is found, we should fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1056) Reference beam version only once in Archetypes

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832202#comment-15832202
 ] 

Davor Bonaci commented on BEAM-1056:


Yup.

> Reference beam version only once in Archetypes
> --
>
> Key: BEAM-1056
> URL: https://issues.apache.org/jira/browse/BEAM-1056
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Thomas Groh
>Assignee: Tim Taschke
>Priority: Trivial
>  Labels: easyfix, starter
>
> Instead of referring to the version of the SDK inline, reference it once in a 
> beam.version property and use that to determine sdk and runner versions. This 
> reduces maintenance burden for updating archetypes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1286) DataflowRunner handling of missing filesToStage

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-1286:
---
Labels: starter  (was: )

> DataflowRunner handling of missing filesToStage
> ---
>
> Key: BEAM-1286
> URL: https://issues.apache.org/jira/browse/BEAM-1286
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Davor Bonaci
>  Labels: starter
>
> DataflowRunner allows filesToStage to be missing -- it logs an error and 
> moves on. Is this the right behavior? It can complicate user experience.
> At least, I guess that if nothing to be staged is found, we should fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1289) TextIO (and others) should provide more informative messages when encountering RVP

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-1289:
---
Assignee: Sam McVeety  (was: Davor Bonaci)

> TextIO (and others) should provide more informative messages when 
> encountering RVP
> --
>
> Key: BEAM-1289
> URL: https://issues.apache.org/jira/browse/BEAM-1289
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Ben Chambers
>Assignee: Sam McVeety
>Priority: Minor
>  Labels: starter
>
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L301
> When reporting failing to validate because a parameter comes from a 
> RuntimeValueProvider (such as when creating templates) we report:
> "Cannot validate with a RVP"
> We should instead provide a message saying what went wrong and how to fix it. 
> Something like:
> "Unable to validate parameters that aren't available until runtime. Disable 
> validation using .withoutValidation()."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-1056) Reference beam version only once in Archetypes

[Tim Taschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831379#comment-15831379
 ] 

Tim Taschke edited comment on BEAM-1056 at 1/20/17 8:32 AM:


Just to be sure:
This is just about the archetypes in {{/sdks/java/maven-archetypes/}}, right?

Assuming so, I just moved the property
{code:xml}

0.5.0-SNAPSHOT

{code}
to {{/sdks/java/maven-archetypes/pom.xml}}





was (Author: ttaschke):
Just to be sure:
This is just about the archetypes in /sdks/java/maven-archetypes/ right?

Assuming so, I just moved the property
{code:xml}

0.5.0-SNAPSHOT

{code}
to /sdks/java/maven-archetypes/pom.xml




> Reference beam version only once in Archetypes
> --
>
> Key: BEAM-1056
> URL: https://issues.apache.org/jira/browse/BEAM-1056
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Thomas Groh
>Assignee: Tim Taschke
>Priority: Trivial
>  Labels: easyfix, starter
>
> Instead of referring to the version of the SDK inline, reference it once in a 
> beam.version property and use that to determine sdk and runner versions. This 
> reduces maintenance burden for updating archetypes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1056) Reference beam version only once in Archetypes

[Tim Taschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831379#comment-15831379
 ] 

Tim Taschke commented on BEAM-1056:
---

Just to be sure:
This is just about the archetypes in /sdks/java/maven-archetypes/ right?

Assuming so, I just moved the property
{code:xml}

0.5.0-SNAPSHOT

{code}
to /sdks/java/maven-archetypes/pom.xml




> Reference beam version only once in Archetypes
> --
>
> Key: BEAM-1056
> URL: https://issues.apache.org/jira/browse/BEAM-1056
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Thomas Groh
>Assignee: Tim Taschke
>Priority: Trivial
>  Labels: easyfix, starter
>
> Instead of referring to the version of the SDK inline, reference it once in a 
> beam.version property and use that to determine sdk and runner versions. This 
> reduces maintenance burden for updating archetypes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-25) Add user-ready API for interacting with state

[Jingsong Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831368#comment-15831368
 ] 

Jingsong Lee edited comment on BEAM-25 at 1/20/17 8:21 AM:
---

Are there plans to support StatefulCombineFn?
eg: select itemId, count(distinct userId) from accessTable group by 
ceil(timestamp to day), itemId.
Use SetState to distinct users in CombineFn instead of use Distinct 
transform(bring another Combine.perKey, not efficient).
Related to BEAM-41


was (Author: lzljs3620320):
Are there plans to support StatefulCombineFn?
eg: select itemId, count(distinct userId) from accessTable group by 
ceil(timestamp to day), itemId.
Use SetState to distinct users in CombineFn instead of use Distinct 
transform(bring another Combine.perKey, not efficient).

> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[Jingsong Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831368#comment-15831368
 ] 

Jingsong Lee commented on BEAM-25:
--

Are there plans to support StatefulCombineFn?
eg: select itemId, count(distinct userId) from accessTable group by 
ceil(timestamp to day), itemId.
Use SetState to distinct users in CombineFn instead of use Distinct 
transform(bring another Combine.perKey, not efficient).

> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1288) `mvn verify` leaves .pyc files throughout source tree

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1288:
-

 Summary: `mvn verify` leaves .pyc files throughout source tree
 Key: BEAM-1288
 URL: https://issues.apache.org/jira/browse/BEAM-1288
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Daniel Halperin
Assignee: Ahmet Altay


even after running `mvn clean`



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1100) Consider removing pylint from mvn verify

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831361#comment-15831361
 ] 

Daniel Halperin commented on BEAM-1100:
---

Just ran into this issue now. To be included in a Beam release -- which is a 
tarball of the source code -- the build process must not assume it's in a git 
repo.

> Consider removing pylint from mvn verify
> 
>
> Key: BEAM-1100
> URL: https://issues.apache.org/jira/browse/BEAM-1100
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Critical
>
> pylint depends on git commands for getting a list of changed files to run the 
> linter on. This does not make sense in some cases (for example when in the 
> case code was copied/packaged out of git structure). A few possible remedies 
> could be considered:
> - Remove pylint step from mvn verify
> - Add an option to pylint run on whole code base and use this in mvn verify
> - Change pylint to run on the whole code base (instead of just changed files)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1100) Consider removing pylint from mvn verify

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1100:
--
Priority: Critical  (was: Minor)

> Consider removing pylint from mvn verify
> 
>
> Key: BEAM-1100
> URL: https://issues.apache.org/jira/browse/BEAM-1100
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Critical
>
> pylint depends on git commands for getting a list of changed files to run the 
> linter on. This does not make sense in some cases (for example when in the 
> case code was copied/packaged out of git structure). A few possible remedies 
> could be considered:
> - Remove pylint step from mvn verify
> - Add an option to pylint run on whole code base and use this in mvn verify
> - Change pylint to run on the whole code base (instead of just changed files)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)