Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2572

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #4317

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_Python #72

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3529

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3340

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #4316

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #2672

[Apache Jenkins Server]

See 


Changes:

[klk] Fix bad merge

[klk] Simplified ByteBuddyOnTimerInvokerFactory

--
[...truncated 583.77 KB...]
}, 
{
  "kind": "CombineValues", 
  "name": "s4", 
  "properties": {
"display_data": [], 
"encoding": {
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "monthly count/Combine.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s3"
}, 
"serialized_fn": "", 
"user_name": "monthly count/Combine"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s5", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "format.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s4"
}, 
"serialized_fn": "", 
"user_name": "format"
  }
}, 
{
  "kind": "ParallelWrite", 
  "name": "s6", 
  "properties": {
"create_disposition": "CREATE_IF_NEEDED", 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2571

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2534) KafkaIO should allow gaps in message offsets

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077614#comment-16077614
 ] 

ASF GitHub Bot commented on BEAM-2534:
--

GitHub user rangadi opened a pull request:

https://github.com/apache/beam/pull/3514

[BEAM-2534] Cherry-pick #3461 into 2.1.0.

Cherry-pick #3461 : Handle offset gaps in Kafka messages.

R: @jbonofre CC: @kennknowles.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rangadi/beam cherrypick-pr-3461

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3514.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3514


commit 9ffaf185ceefaf7de51b1197e00d2d54bdbb6760
Author: Raghu Angadi 
Date:   2017-06-28T19:07:06Z

[BEAM-2534] Handle offset gaps in Kafka messages.

KafkaIO logged a warning when there is a gap in offstes for messages.
Kafka also support 'KV' store style topics where some of the messages
are deleted leading gaps in offsets. This PR removes the log and
accounts for offset gaps in backlog estimate.




> KafkaIO should allow gaps in message offsets
> 
>
> Key: BEAM-2534
> URL: https://issues.apache.org/jira/browse/BEAM-2534
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.0.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Minor
> Fix For: 2.1.0
>
>
> KafkaIO reader logs a warning when it notices gaps in offsets for messages. 
> While such gaps are not expected for normal Kafka topics, there could be gaps 
> when log compaction is enabled (which deletes older messages for a key). 
> This warning log is not very useful. Also we should take such gaps while 
> estimating backlog.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3514: [BEAM-2534] Cherry-pick #3461 into 2.1.0.

[rangadi]

GitHub user rangadi opened a pull request:

https://github.com/apache/beam/pull/3514

[BEAM-2534] Cherry-pick #3461 into 2.1.0.

Cherry-pick #3461 : Handle offset gaps in Kafka messages.

R: @jbonofre CC: @kennknowles.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rangadi/beam cherrypick-pr-3461

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3514.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3514


commit 9ffaf185ceefaf7de51b1197e00d2d54bdbb6760
Author: Raghu Angadi 
Date:   2017-06-28T19:07:06Z

[BEAM-2534] Handle offset gaps in Kafka messages.

KafkaIO logged a warning when there is a gap in offstes for messages.
Kafka also support 'KV' store style topics where some of the messages
are deleted leading gaps in offsets. This PR removes the log and
accounts for offset gaps in backlog estimate.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2570

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3339

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #2671

[Apache Jenkins Server]

See 


Changes:

[klk] Made DataflowRunner TransformTranslator public

--
[...truncated 583.50 KB...]
}, 
{
  "kind": "CombineValues", 
  "name": "s4", 
  "properties": {
"display_data": [], 
"encoding": {
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "monthly count/Combine.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s3"
}, 
"serialized_fn": "", 
"user_name": "monthly count/Combine"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s5", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "format.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s4"
}, 
"serialized_fn": "", 
"user_name": "format"
  }
}, 
{
  "kind": "ParallelWrite", 
  "name": "s6", 
  "properties": {
"create_disposition": "CREATE_IF_NEEDED", 
"dataset": "

[jira] [Commented] (BEAM-2551) KafkaIO reader blocks indefinitely if servers are not reachable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077594#comment-16077594
 ] 

ASF GitHub Bot commented on BEAM-2551:
--

GitHub user rangadi opened a pull request:

https://github.com/apache/beam/pull/3513

[BEAM-2551] Cherrypick #3492 to 2.1.0

Cherrypick #3492 to 2.1.0
 R @kennknowles.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rangadi/beam cherrypick-timeout

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3513.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3513


commit 38fc2b2e76fd13c3edd304ffcb5378fb36ab48c0
Author: Raghu Angadi 
Date:   2017-07-04T06:54:10Z

Add timeout to initialization of partition in KafkaIO




> KafkaIO reader blocks indefinitely if servers are not reachable
> ---
>
> Key: BEAM-2551
> URL: https://issues.apache.org/jira/browse/BEAM-2551
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.0.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
> Fix For: 2.1.0
>
>
> If the KafaIO source reader on the worker can't reach the server, Kafka 
> consumer blocks forever inside {{UnboundedReader.start()}}. Users have no 
> indication of the error. It is better if start() fails with an error. 
> It is easy to reproduce in Kafka. I reported it on Kafka users list here : 
> https://lists.apache.org/thread.html/98cebefacbd65b0d6c6817fe0b5197e26bc60252e72d05fced91e628@%3Cusers.kafka.apache.org%3E
> It blocks inside Kafka client. Fortunately it can be unblocked with 
> KafkaConsumer.wakeup(). We could run initialization in another thread and 
> cancel it if takes longer than a minute.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3513: [BEAM-2551] Cherrypick #3492 to 2.1.0

[rangadi]

GitHub user rangadi opened a pull request:

https://github.com/apache/beam/pull/3513

[BEAM-2551] Cherrypick #3492 to 2.1.0

Cherrypick #3492 to 2.1.0
+R @kennknowles.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rangadi/beam cherrypick-timeout

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3513.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3513


commit 38fc2b2e76fd13c3edd304ffcb5378fb36ab48c0
Author: Raghu Angadi 
Date:   2017-07-04T06:54:10Z

Add timeout to initialization of partition in KafkaIO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2534) KafkaIO should allow gaps in message offsets

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077590#comment-16077590
 ] 

Kenneth Knowles commented on BEAM-2534:
---

Yes. [~rangadi]?

> KafkaIO should allow gaps in message offsets
> 
>
> Key: BEAM-2534
> URL: https://issues.apache.org/jira/browse/BEAM-2534
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.0.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Minor
> Fix For: 2.1.0
>
>
> KafkaIO reader logs a warning when it notices gaps in offsets for messages. 
> While such gaps are not expected for normal Kafka topics, there could be gaps 
> when log compaction is enabled (which deletes older messages for a key). 
> This warning log is not very useful. Also we should take such gaps while 
> estimating backlog.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-940) ByteBuddyOnTimerInvokerFactory: key the cache with a (Class, id) tuple or OnTimerMethod

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-940.
--
   Resolution: Fixed
Fix Version/s: 2.2.0

> ByteBuddyOnTimerInvokerFactory: key the cache with a (Class, id) tuple or 
> OnTimerMethod
> ---
>
> Key: BEAM-940
> URL: https://issues.apache.org/jira/browse/BEAM-940
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Innocent
>Priority: Trivial
> Fix For: 2.2.0
>
>
> Right now it is a {{LoadingCache>}}. 
> It is correct but just a bit less straightforward then we might like.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-940) ByteBuddyOnTimerInvokerFactory: key the cache with a (Class, id) tuple or OnTimerMethod

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077584#comment-16077584
 ] 

ASF GitHub Bot commented on BEAM-940:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3470


> ByteBuddyOnTimerInvokerFactory: key the cache with a (Class, id) tuple or 
> OnTimerMethod
> ---
>
> Key: BEAM-940
> URL: https://issues.apache.org/jira/browse/BEAM-940
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Innocent
>Priority: Trivial
>
> Right now it is a {{LoadingCache>}}. 
> It is correct but just a bit less straightforward then we might like.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[1/2] beam git commit: Simplified ByteBuddyOnTimerInvokerFactory

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 17bc3b140 -> 440c7d45b


Simplified ByteBuddyOnTimerInvokerFactory


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c8d98336
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c8d98336
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c8d98336

Branch: refs/heads/master
Commit: c8d983363efd3f3d93825ecc8e8abae2dfa4e008
Parents: 17bc3b1
Author: Innocent Djiofack 
Authored: Wed Jun 28 22:15:11 2017 -0400
Committer: Kenneth Knowles 
Committed: Thu Jul 6 21:46:53 2017 -0700

--
 .../reflect/ByteBuddyOnTimerInvokerFactory.java | 73 
 .../reflect/OnTimerMethodSpecifier.java | 37 ++
 2 files changed, 65 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c8d98336/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
index e031337..5e31f2e 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/ByteBuddyOnTimerInvokerFactory.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.sdk.transforms.reflect;
 
+
 import com.google.common.base.CharMatcher;
 import com.google.common.cache.CacheBuilder;
 import com.google.common.cache.CacheLoader;
@@ -61,13 +62,14 @@ class ByteBuddyOnTimerInvokerFactory implements 
OnTimerInvokerFactory {
 
 @SuppressWarnings("unchecked")
 Class> fnClass = (Class>) 
fn.getClass();
-
 try {
-  Constructor constructor = constructorCache.get(fnClass).get(timerId);
-  @SuppressWarnings("unchecked")
-  OnTimerInvoker invoker =
+OnTimerMethodSpecifier onTimerMethodSpecifier =
+OnTimerMethodSpecifier.forClassAndTimerId(fnClass, timerId);
+Constructor constructor = 
constructorCache.get(onTimerMethodSpecifier);
+
+OnTimerInvoker invoker =
   (OnTimerInvoker) constructor.newInstance(fn);
-  return invoker;
+return invoker;
 } catch (InstantiationException
 | IllegalAccessException
 | IllegalArgumentException
@@ -97,50 +99,31 @@ class ByteBuddyOnTimerInvokerFactory implements 
OnTimerInvokerFactory {
   private static final String FN_DELEGATE_FIELD_NAME = "delegate";
 
   /**
-   * A cache of constructors of generated {@link OnTimerInvoker} classes, 
keyed by {@link DoFn}
-   * class and then by {@link TimerId}.
+   * A cache of constructors of generated {@link OnTimerInvoker} classes,
+   * keyed by {@link OnTimerMethodSpecifier}.
*
* Needed because generating an invoker class is expensive, and to avoid 
generating an
* excessive number of classes consuming PermGen memory in Java's that still 
have PermGen.
*/
-  private final LoadingCache>, LoadingCache>>
-  constructorCache =
-  CacheBuilder.newBuilder()
-  .build(
-  new CacheLoader<
-  Class>, LoadingCache>>() {
-@Override
-public LoadingCache> load(
-final Class> fnClass) throws 
Exception {
-  return CacheBuilder.newBuilder().build(new 
OnTimerConstructorLoader(fnClass));
-}
-  });
-
-  /**
-   * A cache loader fixed to a particular {@link DoFn} class that loads 
constructors for the
-   * invokers for its {@link OnTimer @OnTimer} methods.
-   */
-  private static class OnTimerConstructorLoader extends CacheLoader> {
-
-private final DoFnSignature signature;
-
-public OnTimerConstructorLoader(Class> clazz) {
-  this.signature = DoFnSignatures.getSignature(clazz);
-}
-
-@Override
-public Constructor load(String timerId) throws Exception {
-  Class> invokerClass =
-  generateOnTimerInvokerClass(signature, timerId);
-  try {
-return invokerClass.getConstructor(signature.fnClass());
-  } catch (IllegalArgumentException | NoSuchMethodException | 
SecurityException e) {
-throw new RuntimeException(e);
-  }
-}
-  }
-
-  /**
+  private final LoadingCache> 
constructorCache =
+  CacheBuilder.newBuilder().build(
+  new CacheLoader>() {
+  @Override
+  public Constructor load(final OnTimerMethodSpecifier 
onTimerMethodSpecifier)
+  throws Exception {
+  DoFnSignature signature =
+  
DoFnSig

[GitHub] beam pull request #3470: [BEAM-940] ByteBuddyOnTimerInvokerFactory: key the ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3470


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3470: [BEAM-940] ByteBuddyOnTimerInvokerFactory: key the cache with a (Class, id) tuple or OnTimerMethod

[kenn]

This closes #3470: [BEAM-940] ByteBuddyOnTimerInvokerFactory: key the cache 
with a (Class, id) tuple or OnTimerMethod

  Simplified ByteBuddyOnTimerInvokerFactory


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/440c7d45
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/440c7d45
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/440c7d45

Branch: refs/heads/master
Commit: 440c7d45b21e64d49566f0674a2dc91f6d284257
Parents: 17bc3b1 c8d9833
Author: Kenneth Knowles 
Authored: Thu Jul 6 21:47:11 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 21:47:11 2017 -0700

--
 .../reflect/ByteBuddyOnTimerInvokerFactory.java | 73 
 .../reflect/OnTimerMethodSpecifier.java | 37 ++
 2 files changed, 65 insertions(+), 45 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3338

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2567) Port triggers design doc to a contributor technical reference

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2567:
-

 Summary: Port triggers design doc to a contributor technical 
reference
 Key: BEAM-2567
 URL: https://issues.apache.org/jira/browse/BEAM-2567
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


There is a fairly old doc at https://s.apache.org/beam-triggers that could be a 
useful reference doc for contributors. Since we don't catalog these docs 
anywhere, it should be surfaced in a useful form.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2568) Port lateness design doc to a contributor technical reference

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2568:
-

 Summary: Port lateness design doc to a contributor technical 
reference
 Key: BEAM-2568
 URL: https://issues.apache.org/jira/browse/BEAM-2568
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


There is a fairly old doc at https://s.apache.org/beam-lateness that could be a 
useful reference doc for contributors. Since we don't catalog these docs 
anywhere, it should be surfaced in a useful form.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2569

[Apache Jenkins Server]

See 

beam git commit: Fix bad merge

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 17d7e5980 -> 17bc3b140


Fix bad merge


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/17bc3b14
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/17bc3b14
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/17bc3b14

Branch: refs/heads/master
Commit: 17bc3b140c7c7315880ce18d4e15d6ac512c35d2
Parents: 17d7e59
Author: Kenneth Knowles 
Authored: Thu Jul 6 21:45:39 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 21:45:39 2017 -0700

--
 .../java/org/apache/beam/runners/dataflow/TransformTranslator.java  | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/17bc3b14/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
index 7f61b6c..06ed1e0 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
@@ -22,6 +22,7 @@ import java.util.Map;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
 import org.apache.beam.runners.dataflow.util.OutputReference;
 import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.annotations.Internal;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3337

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4315

[Apache Jenkins Server]

See 


Changes:

[klk] Made DataflowRunner TransformTranslator public

--
[...truncated 574.66 KB...]
2017-07-07T04:39:32.548 [INFO]  
   
2017-07-07T04:39:32.548 [INFO] 

2017-07-07T04:39:32.548 [INFO] Building Apache Beam :: SDKs :: Java :: IO :: 
AMQP 2.2.0-SNAPSHOT
2017-07-07T04:39:32.548 [INFO] 

2017-07-07T04:39:32.557 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/qpid/proton-j/0.13.1/proton-j-0.13.1.pom
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[INFO] I/O exception (java.net.SocketException) caught when processing request 
to {s}->https://repo.maven.apache.org:443: Connection reset
[INFO] Retrying request to {s}->https://repo.maven.apache.org:443
[JENKINS] Archiving disabled
2017-07-07T04:39:33.260 [INFO]  
   
2017-07-07T04:39:33.260 [INFO] 

2017-07-07T04:39:33.260 [INFO] Skipping Apache Beam :: Parent
2017-07-07T04:39:33.260 [INFO] This project has been banned from the build due 
to previous failures.
2017-07-07T04:39:33.260 [INFO] 

[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
2017-07-07T04:39:46.984 [INFO] 

2017-07-07T04:39:46.984 [INFO] Reactor Summary:
2017-07-07T04:39:46.984 [INFO] 
2017-07-07T04:39:46.984 [INFO] Apache Beam :: Parent 
.. SUCCESS [ 25.863 s]
2017-07-07T04:39:46.984 [INFO] Apache Beam :: SDKs :: Java :: Build Tools 
. SUCCESS [  8.411 s]
2017-07-07T04:39:46.984 [INFO] Apache Beam :: SDKs 
 SUCCESS [  4.313 s]
2017-07-07T04:39:46.984 [INFO] Apache Beam :: SDKs :: Common 
.. SUCCESS [  1.590 s]
2017-07-07T04:39:46.984 [INFO] Apache Beam :: SDKs :: Common :: Runner API 
 SUCCESS [ 15.816 s]
2017-07-07T04:39:46.985 [INFO] Apache Beam :: SDKs :: Common :: Fn API 
 SUCCESS [ 18.178 s]
2017-07-07T04:39:46.985 [INFO] Apache Beam :: SDKs :: Java 
 SUCCESS [  1.638 s]
2017-07-07T04:39:46.985 [INFO] Apache Beam :: SDKs :: Java :: Core 
 SUCCESS [02:54 min]
2017-07-07T04:39:46.985 [INFO] Apache Beam :: Runners 
. SUCCESS [  2.505 s]
2017-07-07T04:39:46.985 [INFO] Apache Beam :: Runners :: Core Construction Java 
... SUCCESS [ 31.761 s]
2017-07-07T04:39:46.985 [INFO] Apache Beam :: Runners :: Core Java 
 SUCCESS [ 53.943 s]
2017-07-07T04:39:46.985 [INFO] Apache Beam :: Runners :: Direct Java 
.. SUCCESS [04:35 min]
2017-07-07T04:39:46.985 [INFO] Apache Beam :: SDKs :: Java :: IO 
.. SUCCESS [  1.918 s]
2017-07-07T04:39:46.985 [INFO] Apache Beam :: SDKs :: Java :: IO :: AMQP 
.. FAILURE [  0.216 s]
2017-07-

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #3528

[Apache Jenkins Server]

See 


Changes:

[klk] Made DataflowRunner TransformTranslator public

--
[...truncated 2.37 MB...]
[INFO] Closing reader after reading 9 records.
[INFO] Proposing to split 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]} at 
fraction 0.4453125 (key [6b657930303030303030303402])
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Adjusting range start from [6b6579303030303030303030] to 
[6b6579303030303030303030] as position of first returned record
[INFO] Proposing to split 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]} at 
fraction 0.4453125 (key [6b657930303030303030303402])
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Adjusting range start from [6b6579303030303030303030] to 
[6b6579303030303030303030] as position of first returned record
[INFO] Closing reader after reading 9 records.
[INFO] Proposing to split 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]} at 
fraction 0.4453125 (key [6b657930303030303030303402])
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[INFO] Adjusting range start from [6b6579303030303030303030] to 
[6b6579303030303030303030] as position of first returned record
[INFO] Proposing to split 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303036]} at 
fraction 0.4453125 (key [6b657930303030303030303402])
[INFO] Closing reader after reading 9 records.
[WARNING] 
ByteKeyRangeTracker{range=ByteKeyRange{startKey=[6b6579303030303030303030], 
endKey=[6b6579303030303030303039]}, position=[6b6579303030303030303038]}: 
Rejecting split request at [6b657930303030303030303402] because it 
is not after current position [6b6579303030303030303038].
[WARNING] After 101 concurrent splitting trials at item #5, observed only 
failure, giving up on this item
[INFO] 474 total concurrent splitting trials for 9 items
[ERROR] At least 1 errors occurred writing to Bigtable. First 1 errors: 
Error mutating row KEY with mutations []: cell value missing
[INFO] About to split into bundles of size 5000 with sampleRowKeys length 10 
first element row_key: "key00100"
offset_bytes: 1

[INFO] Generated 20 splits. First split: 
BigtableSource{tableId=TEST-MANY-ROWS-SPLITS-TABLE, filter=null, 
range=ByteKeyRange{startKey=[], endKey=[35b2bc981818181818189818]}, 
estimatedSizeBytes=5000}
[INFO] Adjusting range start from [] to [6b6579303030303030303030] as position 
of first returned record
[INFO] Closing reader after reading 1000 records.
[INFO] Closing reader after reading 0 records.
[INFO] Adjusting range start from [35b2bc981818181818189818] to 
[6b6579303030303030303030] as position of first returned record
[INFO] Closing reader after reading 100 records.
[INFO] Adjusting range start from [6b6579303030303030313030] to 
[6b6579303030303030313030] as position of first returned record
[INFO] Closing reader after reading 100 records.
[INFO] Closing reader after reading 0 records.
[INFO] Adjusting range start from [6b6579303030303030323030] to 
[6b6579303030303030323030] as position of first returned record
[INFO] Closing reader after reading 100 records.
[INFO] Closing reader after reading 0 records.
[INFO] Adjusting range start from [6b6579303030303030333030] to 
[6b6579303030303030333030] as position of first returned record
[INFO] Closing reader after reading 100 records.
[INFO] Closing reader after reading 0 records.
[INFO] Adjusting range start from [6b6579303030303030343030] to 
[6b6579303030303030343030] as position of first returned record
[INFO] Closing reader after reading 100 records.
[INFO] Closing reader after reading 0 records.
[INFO] Adjusting range start from [6b6579303030303030353030] to

[jira] [Created] (BEAM-2566) Java SDK harness should not depend on any runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-2566:
-

 Summary: Java SDK harness should not depend on any runner
 Key: BEAM-2566
 URL: https://issues.apache.org/jira/browse/BEAM-2566
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Luke Cwik


Right now there is a dependency on the Dataflow runner. I believe this is 
legacy due to using {{CloudObject}} temporarily but I do not claim to 
understand the full nature of the dependency.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3467: Made TransformTranslator public.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3467


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3467: Made DataflowRunner TransformTranslator public

[kenn]

This closes #3467: Made DataflowRunner TransformTranslator public


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/17d7e598
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/17d7e598
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/17d7e598

Branch: refs/heads/master
Commit: 17d7e5980e58a6ee668f41f1518a09bed274b5b2
Parents: 628dace da92256
Author: Kenneth Knowles 
Authored: Thu Jul 6 21:18:05 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 21:18:05 2017 -0700

--
 .../org/apache/beam/runners/dataflow/TransformTranslator.java | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--

[1/2] beam git commit: Made DataflowRunner TransformTranslator public

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 628dace9c -> 17d7e5980


Made DataflowRunner TransformTranslator public


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/da92256b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/da92256b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/da92256b

Branch: refs/heads/master
Commit: da92256ba64c5f476bd6283db2484bd72293
Parents: 1e16aa2
Author: Jeremie Lenfant-Engelmann 
Authored: Wed Jun 28 16:11:21 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 21:15:05 2017 -0700

--
 .../org/apache/beam/runners/dataflow/TransformTranslator.java | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/da92256b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
index a7452b2..7f61b6c 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/TransformTranslator.java
@@ -36,7 +36,8 @@ import org.apache.beam.sdk.values.TupleTag;
  * A {@link TransformTranslator} knows how to translate a particular subclass 
of {@link PTransform}
  * for the Cloud Dataflow service. It does so by mutating the {@link 
TranslationContext}.
  */
-interface TransformTranslator {
+@Internal
+public interface TransformTranslator {
   void translate(TransformT transform, TranslationContext context);
 
   /**

Build failed in Jenkins: beam_PostCommit_Python_Verify #2670

[Apache Jenkins Server]

See 


--
[...truncated 583.73 KB...]
}, 
{
  "kind": "CombineValues", 
  "name": "s4", 
  "properties": {
"display_data": [], 
"encoding": {
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "monthly count/Combine.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s3"
}, 
"serialized_fn": "", 
"user_name": "monthly count/Combine"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s5", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "format.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s4"
}, 
"serialized_fn": "", 
"user_name": "format"
  }
}, 
{
  "kind": "ParallelWrite", 
  "name": "s6", 
  "properties": {
"create_disposition": "CREATE_IF_NEEDED", 
"dataset": "BigQueryTornadoesIT", 
"display_data": [], 
"encoding": {
   

[jira] [Created] (BEAM-2565) Add integration test for OTHER operators

[James Xu (JIRA)]

James Xu created BEAM-2565:
--

 Summary: Add integration test for OTHER operators
 Key: BEAM-2565
 URL: https://issues.apache.org/jira/browse/BEAM-2565
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: James Xu






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2564) Add integration test for string operators

[James Xu (JIRA)]

James Xu created BEAM-2564:
--

 Summary: Add integration test for string operators
 Key: BEAM-2564
 URL: https://issues.apache.org/jira/browse/BEAM-2564
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: James Xu






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2563) Add integration test for math operators

[James Xu (JIRA)]

James Xu created BEAM-2563:
--

 Summary: Add integration test for math operators
 Key: BEAM-2563
 URL: https://issues.apache.org/jira/browse/BEAM-2563
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: James Xu






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2562) Add integration test for logical operators

[James Xu (JIRA)]

James Xu created BEAM-2562:
--

 Summary: Add integration test for logical operators
 Key: BEAM-2562
 URL: https://issues.apache.org/jira/browse/BEAM-2562
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: James Xu






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2561) Add integration test for date functions

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu reassigned BEAM-2561:
--

Assignee: (was: Xu Mingmin)

> Add integration test for date functions
> ---
>
> Key: BEAM-2561
> URL: https://issues.apache.org/jira/browse/BEAM-2561
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: James Xu
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2561) Add integration test for date functions

[James Xu (JIRA)]

James Xu created BEAM-2561:
--

 Summary: Add integration test for date functions
 Key: BEAM-2561
 URL: https://issues.apache.org/jira/browse/BEAM-2561
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: James Xu
Assignee: Xu Mingmin






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2559) Add integration test for the sql functions

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu updated BEAM-2559:
---
Summary: Add integration test for the sql functions  (was: Add test for the 
sql functions)

> Add integration test for the sql functions
> --
>
> Key: BEAM-2559
> URL: https://issues.apache.org/jira/browse/BEAM-2559
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: James Xu
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2560) Add integration test for arithmetic operators

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu updated BEAM-2560:
---
Summary: Add integration test for arithmetic operators  (was: Add test for 
arithmetic operators)

> Add integration test for arithmetic operators
> -
>
> Key: BEAM-2560
> URL: https://issues.apache.org/jira/browse/BEAM-2560
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: James Xu
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2560) Add test for arithmetic operators

[James Xu (JIRA)]

James Xu created BEAM-2560:
--

 Summary: Add test for arithmetic operators
 Key: BEAM-2560
 URL: https://issues.apache.org/jira/browse/BEAM-2560
 Project: Beam
  Issue Type: Sub-task
  Components: dsl-sql
Reporter: James Xu
Assignee: James Xu






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2559) Add test for the sql functions

[James Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Xu reassigned BEAM-2559:
--

Assignee: (was: Xu Mingmin)

> Add test for the sql functions
> --
>
> Key: BEAM-2559
> URL: https://issues.apache.org/jira/browse/BEAM-2559
> Project: Beam
>  Issue Type: Task
>  Components: dsl-sql
>Reporter: James Xu
>  Labels: dsl_sql_merge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2559) Add test for the sql functions

[James Xu (JIRA)]

James Xu created BEAM-2559:
--

 Summary: Add test for the sql functions
 Key: BEAM-2559
 URL: https://issues.apache.org/jira/browse/BEAM-2559
 Project: Beam
  Issue Type: Task
  Components: dsl-sql
Reporter: James Xu
Assignee: Xu Mingmin






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2568

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #71

[Apache Jenkins Server]

See 


Changes:

[klk] Add timeout to initialization of partition in KafkaIO

[klk] Ignore processing time timers in expired windows

[klk] Process timer firings for a window together

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 628dace9c173182299f322fa6ed974c75dbff4b5 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 628dace9c173182299f322fa6ed974c75dbff4b5
 > git rev-list 85a99e29448670ae6728a8ee2e4cd3ef95877c3e # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson4251429228094718244.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson5841686889897044075.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson5139416760218626072.sh
+ pip install --user -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): python-gflags==3.1.1 
in /home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): setuptools in 
/usr/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt 
(line 16))
Requirement already satisfied (use --upgrade to upgrade): 
colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 17))
  Installing extra requirements: 'windows'
Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): numpy in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/hudson1618825673769026633.sh
+ pip install --user -e 'sdks/python/[gcp,test]'
Obtaining 
file://
  Running setup.py 
(path:
 egg_info for package from 
file://

:66:
 UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is 
recommended.
  _PIP_VERSION, REQUIRED_PIP_VERSION
no previously-included directories found matching 'doc/.build'

Installed 


warnin

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3336

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2542) Python performance test failed in Beam repo init

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077350#comment-16077350
 ] 

Mark Liu commented on BEAM-2542:


Build back to stable: 
https://builds.apache.org/view/Beam/job/beam_PerformanceTests_Python/67

Will mark this JIRA as resolved.

> Python performance test failed in Beam repo init
> 
>
> Key: BEAM-2542
> URL: https://issues.apache.org/jira/browse/BEAM-2542
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
> Fix For: Not applicable
>
>
> Jenkins link: 
> https://builds.apache.org/view/Beam/job/beam_PerformanceTests_Python/39
> Errors:
> {code}
> 2017-06-29 00:00:32,173 b368d0d5 MainThread beam_integration_benchmark(1/1) 
> ERRORError during benchmark beam_integration_benchmark
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 555, in RunBenchmark
> DoPreparePhase(spec, detailed_timer)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 450, in DoPreparePhase
> spec.BenchmarkPrepare(spec)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
>  line 91, in Prepare
> beam_benchmark_helper.InitializeBeamRepo(benchmark_spec)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/beam_benchmark_helper.py",
>  line 136, in InitializeBeamRepo
> mvn_command)
> TypeError: AddRunnerProfileMvnArgument() takes exactly 3 arguments (2 given)
> {code}
> Need to investigate on benchmark code in Perfkit.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (BEAM-2542) Python performance test failed in Beam repo init

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu resolved BEAM-2542.

   Resolution: Fixed
Fix Version/s: Not applicable

> Python performance test failed in Beam repo init
> 
>
> Key: BEAM-2542
> URL: https://issues.apache.org/jira/browse/BEAM-2542
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
> Fix For: Not applicable
>
>
> Jenkins link: 
> https://builds.apache.org/view/Beam/job/beam_PerformanceTests_Python/39
> Errors:
> {code}
> 2017-06-29 00:00:32,173 b368d0d5 MainThread beam_integration_benchmark(1/1) 
> ERRORError during benchmark beam_integration_benchmark
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 555, in RunBenchmark
> DoPreparePhase(spec, detailed_timer)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 450, in DoPreparePhase
> spec.BenchmarkPrepare(spec)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
>  line 91, in Prepare
> beam_benchmark_helper.InitializeBeamRepo(benchmark_spec)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/beam_benchmark_helper.py",
>  line 136, in InitializeBeamRepo
> mvn_command)
> TypeError: AddRunnerProfileMvnArgument() takes exactly 3 arguments (2 given)
> {code}
> Need to investigate on benchmark code in Perfkit.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2542) Python performance test failed in Beam repo init

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu closed BEAM-2542.
--

> Python performance test failed in Beam repo init
> 
>
> Key: BEAM-2542
> URL: https://issues.apache.org/jira/browse/BEAM-2542
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
> Fix For: Not applicable
>
>
> Jenkins link: 
> https://builds.apache.org/view/Beam/job/beam_PerformanceTests_Python/39
> Errors:
> {code}
> 2017-06-29 00:00:32,173 b368d0d5 MainThread beam_integration_benchmark(1/1) 
> ERRORError during benchmark beam_integration_benchmark
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 555, in RunBenchmark
> DoPreparePhase(spec, detailed_timer)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 450, in DoPreparePhase
> spec.BenchmarkPrepare(spec)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
>  line 91, in Prepare
> beam_benchmark_helper.InitializeBeamRepo(benchmark_spec)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/PerfKitBenchmarker/perfkitbenchmarker/beam_benchmark_helper.py",
>  line 136, in InitializeBeamRepo
> mvn_command)
> TypeError: AddRunnerProfileMvnArgument() takes exactly 3 arguments (2 given)
> {code}
> Need to investigate on benchmark code in Perfkit.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3512: Set the type of batch jobs to FNAPI_BATCH when beam...

[tvalentyn]

GitHub user tvalentyn opened a pull request:

https://github.com/apache/beam/pull/3512

Set the type of batch jobs to FNAPI_BATCH when beam_fn_api experiment…

… is specified.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tvalentyn/beam 
use_beam_fn_api_experiment_to_run_batch_fnapi_jobs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3512.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3512


commit b82702de281f4c534fcf2ac8744da9af1675c7df
Author: Valentyn Tymofieiev 
Date:   2017-07-06T23:20:07Z

Set the type of batch jobs to FNAPI_BATCH when beam_fn_api experiment is 
specified.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-1502) CombinePerKey returns a list of lists when run locally, but something else when run on Dataflow

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-1502:
-

Assignee: Charles Chen  (was: Ahmet Altay)

> CombinePerKey returns a list of lists when run locally, but something else 
> when run on Dataflow
> ---
>
> Key: BEAM-1502
> URL: https://issues.apache.org/jira/browse/BEAM-1502
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Charles Chen
>
> GroupByKey also has this issue. This has been seen by customers internally 
> and externally:
> http://stackoverflow.com/questions/42276520/dataflow-0-5-5-what-does-object-of-type-unwindowedvalues-has-no-len-mean/42283279#42283279



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2669

[Apache Jenkins Server]

See 


Changes:

[klk] Ignore processing time timers in expired windows

[klk] Process timer firings for a window together

--
[...truncated 583.81 KB...]
}, 
{
  "kind": "CombineValues", 
  "name": "s4", 
  "properties": {
"display_data": [], 
"encoding": {
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "monthly count/Combine.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s3"
}, 
"serialized_fn": "", 
"user_name": "monthly count/Combine"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s5", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "format.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s4"
}, 
"serialized_fn": "", 
"user_name": "format"
  }
}, 
{
  "kind": "ParallelWrite", 
  "name": "s6", 
  "properties": {
"create_di

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #3525

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4313

[Apache Jenkins Server]

See 


Changes:

[klk] Ignore processing time timers in expired windows

[klk] Process timer firings for a window together

--
[...truncated 1.12 MB...]
2017-07-06T22:05:17.211 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-protocol/1.2.6/hbase-protocol-1.2.6.jar
 (4264 KB at 5730.4 KB/sec)
2017-07-06T22:05:17.211 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jruby/joni/joni/2.1.2/joni-2.1.2.jar
2017-07-06T22:05:17.293 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jruby/joni/joni/2.1.2/joni-2.1.2.jar 
(183 KB at 221.7 KB/sec)
2017-07-06T22:05:17.293 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-prefix-tree/1.2.6/hbase-prefix-tree-1.2.6.jar
2017-07-06T22:05:17.338 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jruby/jcodings/jcodings/1.0.8/jcodings-1.0.8.jar
 (1261 KB at 1447.6 KB/sec)
2017-07-06T22:05:17.339 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
2017-07-06T22:05:17.339 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-prefix-tree/1.2.6/hbase-prefix-tree-1.2.6.jar
 (100 KB at 114.4 KB/sec)
2017-07-06T22:05:17.339 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop-compat/1.2.6/hbase-hadoop-compat-1.2.6.jar
2017-07-06T22:05:17.376 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop-compat/1.2.6/hbase-hadoop-compat-1.2.6.jar
 (37 KB at 39.8 KB/sec)
2017-07-06T22:05:17.376 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop2-compat/1.2.6/hbase-hadoop2-compat-1.2.6.jar
2017-07-06T22:05:17.429 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.jar
 (258 KB at 268.2 KB/sec)
2017-07-06T22:05:17.429 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar
2017-07-06T22:05:17.431 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop2-compat/1.2.6/hbase-hadoop2-compat-1.2.6.jar
 (99 KB at 102.1 KB/sec)
2017-07-06T22:05:17.432 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-math/2.2/commons-math-2.2.jar
2017-07-06T22:05:17.481 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar
 (81 KB at 79.2 KB/sec)
2017-07-06T22:05:17.481 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/mortbay/jetty/jetty-sslengine/6.1.26/jetty-sslengine-6.1.26.jar
2017-07-06T22:05:17.511 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/mortbay/jetty/jetty-sslengine/6.1.26/jetty-sslengine-6.1.26.jar
 (19 KB at 17.7 KB/sec)
2017-07-06T22:05:17.512 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar
2017-07-06T22:05:17.634 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-math/2.2/commons-math-2.2.jar
 (966 KB at 827.9 KB/sec)
2017-07-06T22:05:17.634 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar
2017-07-06T22:05:17.642 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar
 (399 KB at 339.5 KB/sec)
2017-07-06T22:05:17.643 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jamon/jamon-runtime/2.4.1/jamon-runtime-2.4.1.jar
2017-07-06T22:05:17.673 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar
 (76 KB at 62.3 KB/sec)
2017-07-06T22:05:17.673 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/lmax/disruptor/3.3.0/disruptor-3.3.0.jar
2017-07-06T22:05:17.677 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jamon/jamon-runtime/2.4.1/jamon-runtime-2.4.1.jar
 (24 KB at 19.8 KB/sec)
2017-07-06T22:05:17.677 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-auth/2.5.1/hadoop-auth-2.5.1.jar
2017-07-06T22:05:17.712 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/lmax/disruptor/3.3.0/disruptor-3.3.0.jar
 (78 KB at 62.5 KB/sec)
2017-07-06T22:05:17.712 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.jar
2017-07-06T22:05:17.714 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-server/1.2.6/hbase-server-1.2.6-tests.jar
 (7413 KB at 5943.9 KB/sec)
2017-07-06T22:05:17.714 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/hbase/hbase-hadoop-compat/1.2.6/hbase-hadoop-compat-1.2.6-tests.jar
2017-07-06T22:05:17.714 [INFO] Downloaded: 
https://repo.maven.apache.org/mave

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2567

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3335

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-284) Add AWS S3 support to FileBasedSink

[Roland Harangozo (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077240#comment-16077240
 ] 

Roland Harangozo commented on BEAM-284:
---

I am interested but i have no time to work on it so anyone can feel free to 
take it.

> Add AWS S3 support to FileBasedSink
> ---
>
> Key: BEAM-284
> URL: https://issues.apache.org/jira/browse/BEAM-284
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Reporter: Roland Harangozo
>Assignee: Roland Harangozo
> Fix For: Not applicable
>
>
> Task is to add AWS S3 support to FileBasedSink class.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #2668

[Apache Jenkins Server]

See 


--
[...truncated 584.04 KB...]
}, 
{
  "kind": "CombineValues", 
  "name": "s4", 
  "properties": {
"display_data": [], 
"encoding": {
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "monthly count/Combine.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s3"
}, 
"serialized_fn": "", 
"user_name": "monthly count/Combine"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s5", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "format.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s4"
}, 
"serialized_fn": "", 
"user_name": "format"
  }
}, 
{
  "kind": "ParallelWrite", 
  "name": "s6", 
  "properties": {
"create_disposition": "CREATE_IF_NEEDED", 
"dataset": "BigQueryTornadoesIT", 
"display_data": [], 
"encoding": {
   

[jira] [Commented] (BEAM-2505) When EOW != GC and the timers fire in together, the output is not marked as the final pane

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077231#comment-16077231
 ] 

ASF GitHub Bot commented on BEAM-2505:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3511

[BEAM-2505, BEAM-2502] Fixes to ReduceFnRunner.onTimers (cherry pick to 
release-2.1.0)

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @jbonofre this cherry picks #3429 to the release branch. Please compare 
and merge when satisfied.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam cherry-pick-3429

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3511.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3511


commit 3d7b00983d2ef215639c3fefc3d8df487aac7b2e
Author: Kenneth Knowles 
Date:   2017-06-23T01:09:11Z

Ignore processing time timers in expired windows

commit b1e53d6d87a14e3116f996d44b14540b6b7bfa59
Author: Kenneth Knowles 
Date:   2017-06-23T01:43:39Z

Process timer firings for a window together




> When EOW != GC and the timers fire in together, the output is not marked as 
> the final pane
> --
>
> Key: BEAM-2505
> URL: https://issues.apache.org/jira/browse/BEAM-2505
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2502) Processing time timers for expired windows are not ignored

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077232#comment-16077232
 ] 

Kenneth Knowles commented on BEAM-2502:
---

This is merged to master and awaiting cherry-pick approval to 2.1.0

> Processing time timers for expired windows are not ignored
> --
>
> Key: BEAM-2502
> URL: https://issues.apache.org/jira/browse/BEAM-2502
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.1.0
>
>
> If the ReduceFnRunner receives a processing time timer for an expired window, 
> it may produce output even though the window is expired (and may have already 
> sent a final output!)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2505) When EOW != GC and the timers fire in together, the output is not marked as the final pane

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077233#comment-16077233
 ] 

Kenneth Knowles commented on BEAM-2505:
---

This is fixed on master, awaiting cherry-pick approval to release branch

> When EOW != GC and the timers fire in together, the output is not marked as 
> the final pane
> --
>
> Key: BEAM-2505
> URL: https://issues.apache.org/jira/browse/BEAM-2505
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3511: [BEAM-2505, BEAM-2502] Fixes to ReduceFnRunner.onTi...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/3511

[BEAM-2505, BEAM-2502] Fixes to ReduceFnRunner.onTimers (cherry pick to 
release-2.1.0)

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @jbonofre this cherry picks #3429 to the release branch. Please compare 
and merge when satisfied.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam cherry-pick-3429

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3511.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3511


commit 3d7b00983d2ef215639c3fefc3d8df487aac7b2e
Author: Kenneth Knowles 
Date:   2017-06-23T01:09:11Z

Ignore processing time timers in expired windows

commit b1e53d6d87a14e3116f996d44b14540b6b7bfa59
Author: Kenneth Knowles 
Date:   2017-06-23T01:43:39Z

Process timer firings for a window together




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2502) Processing time timers for expired windows are not ignored

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077230#comment-16077230
 ] 

ASF GitHub Bot commented on BEAM-2502:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3429


> Processing time timers for expired windows are not ignored
> --
>
> Key: BEAM-2502
> URL: https://issues.apache.org/jira/browse/BEAM-2502
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.1.0
>
>
> If the ReduceFnRunner receives a processing time timer for an expired window, 
> it may produce output even though the window is expired (and may have already 
> sent a final output!)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3429: [BEAM-2502, BEAM-2505] More fixes to ReduceFnRunner...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3429


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] beam git commit: Process timer firings for a window together

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master b8f8d18ae -> 628dace9c


Process timer firings for a window together


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/935c0773
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/935c0773
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/935c0773

Branch: refs/heads/master
Commit: 935c077341de5804b29ffee3926795acf403
Parents: bd631b8
Author: Kenneth Knowles 
Authored: Thu Jun 22 18:43:39 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 14:12:39 2017 -0700

--
 .../examples/complete/game/LeaderBoardTest.java |  2 +
 .../beam/runners/core/ReduceFnRunner.java   | 98 +---
 .../beam/runners/core/ReduceFnRunnerTest.java   | 49 +-
 3 files changed, 115 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/935c0773/examples/java8/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java
--
diff --git 
a/examples/java8/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java
 
b/examples/java8/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java
index 745c210..611e2b3 100644
--- 
a/examples/java8/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java
+++ 
b/examples/java8/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java
@@ -276,6 +276,8 @@ public class LeaderBoardTest implements Serializable {
 .addElements(event(TestUser.RED_ONE, 4, Duration.standardMinutes(2)),
 event(TestUser.BLUE_TWO, 3, Duration.ZERO),
 event(TestUser.BLUE_ONE, 3, Duration.standardMinutes(3)))
+// Move the watermark to the end of the window to output on time
+.advanceWatermarkTo(baseTime.plus(TEAM_WINDOW_DURATION))
 // Move the watermark past the end of the allowed lateness plus the 
end of the window
 .advanceWatermarkTo(baseTime.plus(ALLOWED_LATENESS)
 .plus(TEAM_WINDOW_DURATION).plus(Duration.standardMinutes(1)))

http://git-wip-us.apache.org/repos/asf/beam/blob/935c0773/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
index 0632c05..634a2d1 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
@@ -29,7 +29,6 @@ import java.util.Collection;
 import java.util.Collections;
 import java.util.HashMap;
 import java.util.HashSet;
-import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
@@ -638,11 +637,9 @@ public class ReduceFnRunner {
   }
 
   /**
-   * Enriches TimerData with state necessary for processing a timer as well as
-   * common queries about a timer.
+   * A descriptor of the activation for a window based on a timer.
*/
-  private class EnrichedTimerData {
-public final Instant timestamp;
+  private class WindowActivation {
 public final ReduceFn.Context directContext;
 public final ReduceFn.Context renamedContext;
 // If this is an end-of-window timer then we may need to set a garbage 
collection timer
@@ -653,19 +650,34 @@ public class ReduceFnRunner {
 // end-of-window time to be a signal to garbage collect.
 public final boolean isGarbageCollection;
 
-EnrichedTimerData(
-TimerData timer,
+WindowActivation(
 ReduceFn.Context directContext,
 ReduceFn.Context renamedContext) {
-  this.timestamp = timer.getTimestamp();
   this.directContext = directContext;
   this.renamedContext = renamedContext;
   W window = directContext.window();
-  this.isEndOfWindow = TimeDomain.EVENT_TIME == timer.getDomain()
-  && timer.getTimestamp().equals(window.maxTimestamp());
-  Instant cleanupTime = LateDataUtils.garbageCollectionTime(window, 
windowingStrategy);
+
+  // The output watermark is before the end of the window if it is either 
unknown
+  // or it is known to be before it. If it is unknown, that means that 
there hasn't been
+  // enough data to advance it.
+  boolean outputWatermarkBeforeEOW =
+  timerInternals.currentOutputWatermarkTime() == null
+  || 
!timerInternals.currentOutputWatermarkTime().isAfter(window.maxTimestamp());
+
+  // The "end of the window" is reached when the local input watermark 
(for this key) surpasses
+  // it but the local output watermark (also for this ke

[2/3] beam git commit: Ignore processing time timers in expired windows

[kenn]

Ignore processing time timers in expired windows


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bd631b89
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bd631b89
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bd631b89

Branch: refs/heads/master
Commit: bd631b89a8434f0756e1596875e89013fb623ab5
Parents: b8f8d18
Author: Kenneth Knowles 
Authored: Thu Jun 22 18:09:11 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 14:12:39 2017 -0700

--
 .../beam/runners/core/ReduceFnRunner.java   | 10 ++
 .../beam/runners/core/ReduceFnRunnerTest.java   | 32 
 2 files changed, 42 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/bd631b89/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
index ef33bef..0632c05 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ReduceFnRunner.java
@@ -693,6 +693,11 @@ public class ReduceFnRunner {
   @SuppressWarnings("unchecked")
 WindowNamespace windowNamespace = (WindowNamespace) 
timer.getNamespace();
   W window = windowNamespace.getWindow();
+
+  if (TimeDomain.PROCESSING_TIME == timer.getDomain() && 
windowIsExpired(window)) {
+continue;
+  }
+
   ReduceFn.Context directContext =
   contextFactory.base(window, StateStyle.DIRECT);
   ReduceFn.Context renamedContext =
@@ -1090,4 +1095,9 @@ public class ReduceFnRunner {
 }
   }
 
+  private boolean windowIsExpired(BoundedWindow w) {
+return timerInternals
+.currentInputWatermarkTime()
+
.isAfter(w.maxTimestamp().plus(windowingStrategy.getAllowedLateness()));
+  }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/bd631b89/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
index 3a2c220..79ee91b 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/ReduceFnRunnerTest.java
@@ -286,6 +286,38 @@ public class ReduceFnRunnerTest {
 
   /**
* Tests that when a processing time timer comes in after a window is expired
+   * it is just ignored.
+   */
+  @Test
+  public void testLateProcessingTimeTimer() throws Exception {
+WindowingStrategy strategy =
+WindowingStrategy.of((WindowFn) 
FixedWindows.of(Duration.millis(100)))
+.withTimestampCombiner(TimestampCombiner.EARLIEST)
+.withMode(AccumulationMode.ACCUMULATING_FIRED_PANES)
+.withAllowedLateness(Duration.ZERO)
+.withTrigger(
+Repeatedly.forever(
+
AfterProcessingTime.pastFirstElementInPane().plusDelayOf(Duration.millis(10;
+
+ReduceFnTester tester =
+ReduceFnTester.combining(strategy, Sum.ofIntegers(), VarIntCoder.of());
+
+tester.advanceProcessingTime(new Instant(5000));
+injectElement(tester, 2); // processing timer @ 5000 + 10; EOW timer @ 100
+injectElement(tester, 5);
+
+// After this advancement, the window is expired and only the GC process
+// should be allowed to touch it
+tester.advanceInputWatermarkNoTimers(new Instant(100));
+
+// This should not output
+tester.advanceProcessingTime(new Instant(6000));
+
+assertThat(tester.extractOutput(), emptyIterable());
+  }
+
+  /**
+   * Tests that when a processing time timer comes in after a window is expired
* but in the same bundle it does not cause a spurious output.
*/
   @Test

[3/3] beam git commit: This closes #3429: [BEAM-2502, BEAM-2505] More fixes to ReduceFnRunner.onTimers

[kenn]

This closes #3429: [BEAM-2502, BEAM-2505] More fixes to ReduceFnRunner.onTimers

  Process timer firings for a window together
  Ignore processing time timers in expired windows


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/628dace9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/628dace9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/628dace9

Branch: refs/heads/master
Commit: 628dace9c173182299f322fa6ed974c75dbff4b5
Parents: b8f8d18 935c077
Author: Kenneth Knowles 
Authored: Thu Jul 6 14:13:56 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 14:13:56 2017 -0700

--
 .../examples/complete/game/LeaderBoardTest.java |   2 +
 .../beam/runners/core/ReduceFnRunner.java   | 106 +--
 .../beam/runners/core/ReduceFnRunnerTest.java   |  81 +-
 3 files changed, 156 insertions(+), 33 deletions(-)
--

[jira] [Assigned] (BEAM-2539) Problem running current trunk on Dataflow

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-2539:
-

Assignee: Chamikara Jayalath  (was: Ahmet Altay)

> Problem running current trunk on Dataflow
> -
>
> Key: BEAM-2539
> URL: https://issues.apache.org/jira/browse/BEAM-2539
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.1.0
> Environment: gcp dataflow
>Reporter: Endre Karlson
>Assignee: Chamikara Jayalath
>
> Hi, have anyone of you used the python sdk from trunk using "python setup.py 
> sdist" and gotten BigQuery to work? I am facing a issue where StackDriver 
> tells me that failed to synchronize container but it doesnt show any python 
> program errors from the pipeline.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2539) Problem running current trunk on Dataflow

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077223#comment-16077223
 ] 

Ahmet Altay commented on BEAM-2539:
---

[~ekarlso] sorry, we missed this earlier. Do you have a dataflow job id that we 
can investigate?

As far as I know, BQ works with the current code.

[~chamikara] could you verify this from the release branch and master branch?

> Problem running current trunk on Dataflow
> -
>
> Key: BEAM-2539
> URL: https://issues.apache.org/jira/browse/BEAM-2539
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.1.0
> Environment: gcp dataflow
>Reporter: Endre Karlson
>Assignee: Ahmet Altay
>
> Hi, have anyone of you used the python sdk from trunk using "python setup.py 
> sdist" and gotten BigQuery to work? I am facing a issue where StackDriver 
> tells me that failed to synchronize container but it doesnt show any python 
> program errors from the pipeline.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (BEAM-2539) Problem running current trunk on Dataflow

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-2539:
-

Assignee: Ahmet Altay  (was: Thomas Groh)

> Problem running current trunk on Dataflow
> -
>
> Key: BEAM-2539
> URL: https://issues.apache.org/jira/browse/BEAM-2539
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.1.0
> Environment: gcp dataflow
>Reporter: Endre Karlson
>Assignee: Ahmet Altay
>
> Hi, have anyone of you used the python sdk from trunk using "python setup.py 
> sdist" and gotten BigQuery to work? I am facing a issue where StackDriver 
> tells me that failed to synchronize container but it doesnt show any python 
> program errors from the pipeline.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (BEAM-2558) Add a CombineFnTester

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-2558:
-

 Summary: Add a CombineFnTester
 Key: BEAM-2558
 URL: https://issues.apache.org/jira/browse/BEAM-2558
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Thomas Groh
Assignee: Thomas Groh


This should be in the style of {{CoderProperties}} or {{EqualsTester}}: the 
user should provide some inputs, and it should exercise potential paths by 
which those elements may be accumulated and ensure that they all produce the 
same results.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-1939) Serialize more coders via URN + Class name

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077196#comment-16077196
 ] 

Thomas Groh commented on BEAM-1939:
---

CoderTranslation.java is the new class name.

> Serialize more coders via URN + Class name
> --
>
> Key: BEAM-1939
> URL: https://issues.apache.org/jira/browse/BEAM-1939
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Innocent
>Priority: Trivial
>
> If the size of serializing Standard Coders becomes too large, an arbitrary 
> Standard Coder can be encoded, alongside its components, via an URN and 
> looking up the class when it is to be deserialized.
> See 
> https://github.com/tgroh/beam/commit/070854845346d8e4df824e4aa374688bd095c2c6 
> as an example



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3524

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2271) Release guide or pom.xml needs update to avoid releasing Python binary artifacts

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077163#comment-16077163
 ] 

Ahmet Altay commented on BEAM-2271:
---

We could not figure out how to generate the source zip file. (Tried variations 
of {{mvn clean install -Prelease}}.)

In the release guide I _believe_ this happens at {{mvn release:prepare}} 
(https://beam.apache.org/contribute/release-guide/#build-and-stage-java-artifacts-with-maven)
 step. But this step  requires being a committer ([~sb2nov] is not) and it has 
side effects like creating tags.

What is is the mvn command for generating the source zip file?  

> Release guide or pom.xml needs update to avoid releasing Python binary 
> artifacts
> 
>
> Key: BEAM-2271
> URL: https://issues.apache.org/jira/browse/BEAM-2271
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Daniel Halperin
>Assignee: Sourabh Bajaj
> Fix For: 2.1.0
>
>
> The following directories (and children) were discovered in 2.0.0-RC2 and 
> were present in 0.6.0.
> {code}
> sdks/python: build   dist.eggs   nose-1.3.7-py2.7.egg  (and child 
> contents)
> {code}
> Ideally, these artifacts, which are created during setup and testing, would 
> get created in the {{sdks/python/target/}} subfolder where they will 
> automatically get ignored. More info below.
> For 2.0.0, we will manually remove these files from the source release RC3+. 
> This should be fixed before the next release.
> Here is a list of other paths that get excluded, should they be useful.
> {code}
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/).*${project.build.directory}.*]
> 
> 
>  
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?maven-eclipse\.xml]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.project]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.classpath]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iws]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.idea(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?out(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.ipr]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?[^/]*\.iml]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.settings(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.externalToolBuilders(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.deployables(/.*)?]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?\.wtpmodules(/.*)?]
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?cobertura\.ser]
> 
> 
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?pom\.xml\.releaseBackup]
> 
> %regex[(?!((?!${project.build.directory}/)[^/]+/)*src/)(.*/)?release\.properties]
>   
> {code}
> This list is stored inside of this jar, which you can find by tracking 
> maven-assembly-plugin from the root apache pom: 
> https://mvnrepository.com/artifact/org.apache.apache.resources/apache-source-release-assembly-descriptor/1.0.6
> http://svn.apache.org/repos/asf/maven/pom/tags/apache-18/pom.xml



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #1924

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #2667

[Apache Jenkins Server]

See 


Changes:

[klk] Add timeout to initialization of partition in KafkaIO

--
[...truncated 583.50 KB...]
}, 
{
  "kind": "CombineValues", 
  "name": "s4", 
  "properties": {
"display_data": [], 
"encoding": {
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "monthly count/Combine.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s3"
}, 
"serialized_fn": "", 
"user_name": "monthly count/Combine"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s5", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "format.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s4"
}, 
"serialized_fn": "", 
"user_name": "format"
  }
}, 
{
  "kind": "ParallelWrite", 
  "name": "s6", 
  "properties": {
"create_disposition": "CREATE_IF_NEEDED", 
"data

[jira] [Assigned] (BEAM-893) Update Mobile Gaming Example

[Hadar Hod (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hadar Hod reassigned BEAM-893:
--

Assignee: Melissa Pashniak  (was: Hadar Hod)

> Update Mobile Gaming Example
> 
>
> Key: BEAM-893
> URL: https://issues.apache.org/jira/browse/BEAM-893
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Hadar Hod
>Assignee: Melissa Pashniak
> Fix For: 2.2.0
>
>
> - In the intro: As well as offline, this doc should also mention delays that 
> can happen in the network at various points, so that data may arrive delayed 
> and out of order.
> - For Figure 1: Explain more clearly what the two axes mean and how a 
> specific point might have a given position ('ideal' line (x=y) is very 
> helpful here).
> - In section "What Does Leaderboard Do?": These aren't exact 10 minute 
> intervals - possibly because of the arrival pattern of the data. Nothing will 
> be triggered if there's no data that arrives. @kennknowles can probably 
> provide more precise phrasing.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2566

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4311

[Apache Jenkins Server]

See 


--
[...truncated 3.10 MB...]
test_file_checksum_matcher_service_error 
(apache_beam.testing.pipeline_verifiers_test.PipelineVerifiersTest) ... ok
test_file_checksum_matcher_sleep_before_verify 
(apache_beam.testing.pipeline_verifiers_test.PipelineVerifiersTest) ... ok
test_file_checksum_matcher_success 
(apache_beam.testing.pipeline_verifiers_test.PipelineVerifiersTest) ... ok
test_pipeline_state_matcher_fails 
(apache_beam.testing.pipeline_verifiers_test.PipelineVerifiersTest)
Test PipelineStateMatcher fails when using default expected state ... ok
test_pipeline_state_matcher_given_state 
(apache_beam.testing.pipeline_verifiers_test.PipelineVerifiersTest)
Test PipelineStateMatcher successes when matches given state ... ok
test_pipeline_state_matcher_success 
(apache_beam.testing.pipeline_verifiers_test.PipelineVerifiersTest)
Test PipelineStateMatcher successes when using default expected state ... ok
test_append_extra_options 
(apache_beam.testing.test_pipeline_test.TestPipelineTest) ... ok
test_append_verifier_in_extra_opt 
(apache_beam.testing.test_pipeline_test.TestPipelineTest) ... ok
test_create_test_pipeline_options 
(apache_beam.testing.test_pipeline_test.TestPipelineTest) ... ok
test_empty_option_args_parsing 
(apache_beam.testing.test_pipeline_test.TestPipelineTest) ... ok
test_get_option (apache_beam.testing.test_pipeline_test.TestPipelineTest) ... ok
test_option_args_parsing 
(apache_beam.testing.test_pipeline_test.TestPipelineTest) ... ok
test_skip_IT (apache_beam.testing.test_pipeline_test.TestPipelineTest) ... 
SKIP: IT is skipped because --test-pipeline-options is not specified
test_basic_execution (apache_beam.testing.test_stream_test.TestStreamTest) ... 
ok
test_basic_test_stream (apache_beam.testing.test_stream_test.TestStreamTest) 
... ok
test_gbk_execution (apache_beam.testing.test_stream_test.TestStreamTest) ... ok
test_test_stream_errors (apache_beam.testing.test_stream_test.TestStreamTest) 
... ok
test_assert_that_fails (apache_beam.testing.util_test.UtilTest) ... ok
test_assert_that_fails_on_empty_expected 
(apache_beam.testing.util_test.UtilTest) ... ok
test_assert_that_fails_on_empty_input (apache_beam.testing.util_test.UtilTest) 
... ok
test_assert_that_passes (apache_beam.testing.util_test.UtilTest) ... ok

--
Ran 1206 tests in 272.221s

OK (skipped=16)
___ summary 
  docs: commands succeeded
  lint: commands succeeded
  py27: commands succeeded
ERROR:   py27cython: InvocationError: 

 install 

 (see 

  py27gcp: commands succeeded
2017-07-06T19:20:52.319 [ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit 
value: 1)
at 
org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at 
org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:764)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:711)
at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:289)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at 
org.jvnet.hudson.maven3.launcher.Maven33Launcher.main(Maven33Launcher.java:129)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcc

[jira] [Commented] (BEAM-2551) KafkaIO reader blocks indefinitely if servers are not reachable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077069#comment-16077069
 ] 

ASF GitHub Bot commented on BEAM-2551:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3492


> KafkaIO reader blocks indefinitely if servers are not reachable
> ---
>
> Key: BEAM-2551
> URL: https://issues.apache.org/jira/browse/BEAM-2551
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 2.0.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
> Fix For: 2.1.0
>
>
> If the KafaIO source reader on the worker can't reach the server, Kafka 
> consumer blocks forever inside {{UnboundedReader.start()}}. Users have no 
> indication of the error. It is better if start() fails with an error. 
> It is easy to reproduce in Kafka. I reported it on Kafka users list here : 
> https://lists.apache.org/thread.html/98cebefacbd65b0d6c6817fe0b5197e26bc60252e72d05fced91e628@%3Cusers.kafka.apache.org%3E
> It blocks inside Kafka client. Fortunately it can be unblocked with 
> KafkaConsumer.wakeup(). We could run initialization in another thread and 
> cancel it if takes longer than a minute.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3492: [BEAM-2551] KafkaIO reader blocks indefinitely in c...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3492


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3492: [BEAM-2551] KafkaIO reader blocks indefinitely in case of network issues

[kenn]

This closes #3492: [BEAM-2551] KafkaIO reader blocks indefinitely in case of 
network issues

  Add timeout to initialization of partition in KafkaIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b8f8d18a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b8f8d18a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b8f8d18a

Branch: refs/heads/master
Commit: b8f8d18ae2cdbb4874d9a0f45038037ecc2381d1
Parents: 85a99e2 526037b
Author: Kenneth Knowles 
Authored: Thu Jul 6 11:59:18 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 11:59:18 2017 -0700

--
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   | 81 +++-
 .../apache/beam/sdk/io/kafka/KafkaIOTest.java   | 30 
 2 files changed, 92 insertions(+), 19 deletions(-)
--

[1/2] beam git commit: Add timeout to initialization of partition in KafkaIO

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 85a99e294 -> b8f8d18ae


Add timeout to initialization of partition in KafkaIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/526037b6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/526037b6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/526037b6

Branch: refs/heads/master
Commit: 526037b6786315b9f9fdca6edb636baeb6f83e3f
Parents: 85a99e2
Author: Raghu Angadi 
Authored: Mon Jul 3 23:54:10 2017 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 6 11:58:41 2017 -0700

--
 .../org/apache/beam/sdk/io/kafka/KafkaIO.java   | 81 +++-
 .../apache/beam/sdk/io/kafka/KafkaIOTest.java   | 30 
 2 files changed, 92 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/526037b6/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
--
diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
index e520367..026313a 100644
--- a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
+++ b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
@@ -49,9 +49,11 @@ import java.util.Random;
 import java.util.Set;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.SynchronousQueue;
 import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.AtomicBoolean;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.annotations.Experimental;
@@ -1061,8 +1063,32 @@ public class KafkaIO {
   curBatch = Iterators.cycle(nonEmpty);
 }
 
+private void setupInitialOffset(PartitionState pState) {
+  Read spec = source.spec;
+
+  if (pState.nextOffset != UNINITIALIZED_OFFSET) {
+consumer.seek(pState.topicPartition, pState.nextOffset);
+  } else {
+// nextOffset is unininitialized here, meaning start reading from 
latest record as of now
+// ('latest' is the default, and is configurable) or 'look up offset 
by startReadTime.
+// Remember the current position without waiting until the first 
record is read. This
+// ensures checkpoint is accurate even if the reader is closed before 
reading any records.
+Instant startReadTime = spec.getStartReadTime();
+if (startReadTime != null) {
+  pState.nextOffset =
+  consumerSpEL.offsetForTime(consumer, pState.topicPartition, 
spec.getStartReadTime());
+  consumer.seek(pState.topicPartition, pState.nextOffset);
+} else {
+  pState.nextOffset = consumer.position(pState.topicPartition);
+}
+  }
+}
+
 @Override
 public boolean start() throws IOException {
+  final int defaultPartitionInitTimeout = 60 * 1000;
+  final int kafkaRequestTimeoutMultiple = 2;
+
   Read spec = source.spec;
   consumer = spec.getConsumerFactoryFn().apply(spec.getConsumerConfig());
   consumerSpEL.evaluateAssign(consumer, spec.getTopicPartitions());
@@ -1077,25 +1103,38 @@ public class KafkaIO {
   keyDeserializerInstance.configure(spec.getConsumerConfig(), true);
   valueDeserializerInstance.configure(spec.getConsumerConfig(), false);
 
-  for (PartitionState p : partitionStates) {
-if (p.nextOffset != UNINITIALIZED_OFFSET) {
-  consumer.seek(p.topicPartition, p.nextOffset);
-} else {
-  // nextOffset is unininitialized here, meaning start reading from 
latest record as of now
-  // ('latest' is the default, and is configurable) or 'look up offset 
by startReadTime.
-  // Remember the current position without waiting until the first 
record is read. This
-  // ensures checkpoint is accurate even if the reader is closed 
before reading any records.
-  Instant startReadTime = spec.getStartReadTime();
-  if (startReadTime != null) {
-p.nextOffset =
-consumerSpEL.offsetForTime(consumer, p.topicPartition, 
spec.getStartReadTime());
-consumer.seek(p.topicPartition, p.nextOffset);
-  } else {
-p.nextOffset = consumer.position(p.topicPartition);
+  // Seek to start offset for each partition. This is the first 
interaction with the server.
+  // Unfortunately it can block forever in case of network issues like 
incorrect ACLs.
+  // Initialize partition in a separate thread and cancel it if takes 
longer than a minute.
+  for (final Parti

[beam-site] branch asf-site updated (a06a04f -> 8447470)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from a06a04f  Prepare repository for deployment.
 add 561191c  Update build-in IO list with AMQP
 add d74d21f  Remove AMQPIO from in-progress list
 add c8c79f8  Fix missing closing element
 add c0c7e54  This closes #264
 new 8447470  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/io/built-in/index.html | 9 +
 src/documentation/io/built-in.md | 9 +
 2 files changed, 10 insertions(+), 8 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 8447470bcef41860ab6ca2305cb47ef9601b2da4
Author: Mergebot 
AuthorDate: Thu Jul 6 18:52:40 2017 +

Prepare repository for deployment.
---
 content/documentation/io/built-in/index.html | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/content/documentation/io/built-in/index.html 
b/content/documentation/io/built-in/index.html
index d88360b..90e4c35 100644
--- a/content/documentation/io/built-in/index.html
+++ b/content/documentation/io/built-in/index.html
@@ -161,6 +161,7 @@
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/";>XML
   
   
+https://github.com/apache/beam/tree/master/sdks/java/io/amqp";>AMQP
 https://github.com/apache/beam/tree/master/sdks/java/io/jms";>JMS
 https://github.com/apache/beam/tree/master/sdks/java/io/kafka";>Apache 
Kafka
 https://github.com/apache/beam/tree/master/sdks/java/io/kinesis";>Amazon 
Kinesis
@@ -203,10 +204,6 @@
 NameLanguageJIRA
   
   
-AMQPJava
-https://issues.apache.org/jira/browse/BEAM-1237";>BEAM-1237
-  
-  
 Apache DistributedLogJava
 https://issues.apache.org/jira/browse/BEAM-607";>BEAM-607
   
@@ -227,6 +224,10 @@
 https://issues.apache.org/jira/browse/BEAM-1893";>BEAM-1893
   
   
+InfluxDBJava
+https://issues.apache.org/jira/browse/BEAM-2546";>BEAM-2546
+  
+  
 JSONJava
 https://issues.apache.org/jira/browse/BEAM-1581";>BEAM-1581
   

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3523

[Apache Jenkins Server]

See 

[beam-site] 01/04: Update build-in IO list with AMQP

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 561191c871e54e7b9c85f27e10f56539639531ce
Author: Jean-Baptiste Onofré 
AuthorDate: Sat Jul 1 06:17:48 2017 +0200

Update build-in IO list with AMQP
---
 src/documentation/io/built-in.md | 5 +
 1 file changed, 5 insertions(+)

diff --git a/src/documentation/io/built-in.md b/src/documentation/io/built-in.md
index 637d748..6b86bad 100644
--- a/src/documentation/io/built-in.md
+++ b/src/documentation/io/built-in.md
@@ -29,6 +29,7 @@ Consult the [Programming Guide I/O section]({{site.baseurl 
}}/documentation/prog
 https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/";>XML
   
   
+https://github.com/apache/beam/tree/master/sdks/java/io/amqp";>AMQP
 https://github.com/apache/beam/tree/master/sdks/java/io/jms";>JMS
 https://github.com/apache/beam/tree/master/sdks/java/io/kafka";>Apache 
Kafka
 https://github.com/apache/beam/tree/master/sdks/java/io/kinesis";>Amazon 
Kinesis
@@ -95,6 +96,10 @@ This table contains I/O transforms that are currently 
planned or in-progress. St
 https://issues.apache.org/jira/browse/BEAM-1893";>BEAM-1893
   
   
+InfluxDBJava
+https://issues.apache.org/jira/browse/BEAM-2546";>BEAM-2546
+  
+  
 JSONJava
 https://issues.apache.org/jira/browse/BEAM-1581";>BEAM-1581
   

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch mergebot updated (41061fc -> c0c7e54)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 41061fc  This closes #267
 add a06a04f  Prepare repository for deployment.
 new 561191c  Update build-in IO list with AMQP
 new d74d21f  Remove AMQPIO from in-progress list
 new c8c79f8  Fix missing closing element
 new c0c7e54  This closes #264

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/testing/index.html | 4 ++--
 src/documentation/io/built-in.md  | 9 +
 2 files changed, 7 insertions(+), 6 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 03/04: Fix missing closing element

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit c8c79f8793c14b6dd2baf4a1d89e68b8a238e91d
Author: Jean-Baptiste Onofré 
AuthorDate: Thu Jul 6 19:28:46 2017 +0200

Fix missing closing element
---
 src/documentation/io/built-in.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/documentation/io/built-in.md b/src/documentation/io/built-in.md
index 57adeb9..0927d10 100644
--- a/src/documentation/io/built-in.md
+++ b/src/documentation/io/built-in.md
@@ -92,7 +92,7 @@ This table contains I/O transforms that are currently planned 
or in-progress. St
 https://issues.apache.org/jira/browse/BEAM-1893";>BEAM-1893
   
   
-InfluxDBJava
+InfluxDBJava
 https://issues.apache.org/jira/browse/BEAM-2546";>BEAM-2546
   
   

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 04/04: This closes #264

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit c0c7e5417ce8689ad1e663e91b7b4260b8a33f24
Merge: a06a04f c8c79f8
Author: Mergebot 
AuthorDate: Thu Jul 6 18:47:24 2017 +

This closes #264

 src/documentation/io/built-in.md | 9 +
 1 file changed, 5 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 02/04: Remove AMQPIO from in-progress list

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit d74d21f4efa95f32247c655f16e556a7f29d0b9e
Author: Jean-Baptiste Onofré 
AuthorDate: Sat Jul 1 06:24:53 2017 +0200

Remove AMQPIO from in-progress list
---
 src/documentation/io/built-in.md | 4 
 1 file changed, 4 deletions(-)

diff --git a/src/documentation/io/built-in.md b/src/documentation/io/built-in.md
index 6b86bad..57adeb9 100644
--- a/src/documentation/io/built-in.md
+++ b/src/documentation/io/built-in.md
@@ -72,10 +72,6 @@ This table contains I/O transforms that are currently 
planned or in-progress. St
 NameLanguageJIRA
   
   
-AMQPJava
-https://issues.apache.org/jira/browse/BEAM-1237";>BEAM-1237
-  
-  
 Apache DistributedLogJava
 https://issues.apache.org/jira/browse/BEAM-607";>BEAM-607
   

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2565

[Apache Jenkins Server]

See 

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit a06a04f1fbf1035646afef7bc48b10e145773c3a
Author: Mergebot 
AuthorDate: Thu Jul 6 18:22:39 2017 +

Prepare repository for deployment.
---
 content/contribute/testing/index.html | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/content/contribute/testing/index.html 
b/content/contribute/testing/index.html
index 30db095..ac4aec0 100644
--- a/content/contribute/testing/index.html
+++ b/content/contribute/testing/index.html
@@ -153,7 +153,7 @@
   
   Testing 
Types
   Unit
-  ValidatesRunner (Working 
Title)
+  ValidatesRunner
   E2E
 
   
@@ -459,7 +459,7 @@ importance of testing, Beam has a robust set of unit tests, 
as well as testing
 coverage measurement tools, which protect the codebase from simple to moderate
 breakages. Beam Java unit tests are written in JUnit.
 
-ValidatesRunner (Working Title)
+ValidatesRunner
 
 ValidatesRunner tests contain components of both component and end-to-end
 tests. They fulfill the typical purpose of a component test - they are meant to

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch asf-site updated (b411438 -> a06a04f)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from b411438  Prepare repository for deployment.
 add 16dff50  No longer working title
 add 41061fc  This closes #267
 new a06a04f  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/testing/index.html | 4 ++--
 src/contribute/testing.md | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/02: No longer working title

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 16dff502360549e7857b48ce7df8946cdd0bf606
Author: jasonkuster 
AuthorDate: Thu Jul 6 11:15:21 2017 -0700

No longer working title
---
 src/contribute/testing.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/contribute/testing.md b/src/contribute/testing.md
index 69d026b..2d4019c 100644
--- a/src/contribute/testing.md
+++ b/src/contribute/testing.md
@@ -302,7 +302,7 @@ importance of testing, Beam has a robust set of unit tests, 
as well as testing
 coverage measurement tools, which protect the codebase from simple to moderate
 breakages. Beam Java unit tests are written in JUnit.
 
-### ValidatesRunner (Working Title)
+### ValidatesRunner
 
 ValidatesRunner tests contain components of both component and end-to-end
 tests. They fulfill the typical purpose of a component test - they are meant to

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] 02/02: This closes #267

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 41061fc2bff804f5170f37ec3dd895052f74e4b5
Merge: b411438 16dff50
Author: Mergebot 
AuthorDate: Thu Jul 6 18:19:40 2017 +

This closes #267

 src/contribute/testing.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch mergebot updated (681d319 -> 41061fc)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 681d319  This closes #265
 add b411438  Prepare repository for deployment.
 new 16dff50  No longer working title
 new 41061fc  This closes #267

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/testing/index.html | 10 --
 src/contribute/testing.md |  2 +-
 2 files changed, 5 insertions(+), 7 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #1923

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 85a99e29448670ae6728a8ee2e4cd3ef95877c3e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 85a99e29448670ae6728a8ee2e4cd3ef95877c3e
 > git rev-list 85a99e29448670ae6728a8ee2e4cd3ef95877c3e # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact 
http://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
Modules changed, recalculating dependency graph
Established TCP socket on 58999
maven33-agent.jar already up to date
maven33-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_PostCommit_Java_ValidatesRunner_Apex] $ 
/home/jenkins/tools/java/latest1.8/bin/java 
-Dorg.slf4j.simpleLogger.showDateTime=true 
-Dorg.slf4j.simpleLogger.dateTimeFormat=-MM-dd'T'HH:mm:ss.SSS 
-XX:+TieredCompilation -XX:TieredStopAtLevel=1 -cp 
/home/jenkins/jenkins-slave/maven33-agent.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/tools/maven/apache-maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven33Main /home/jenkins/tools/maven/apache-maven-3.3.3 
/home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven33-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 58999
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 clean verify --projects runners/apex --also-make --batch-mode --errors 
--activate-profiles validates-runner-tests --activate-profiles 
local-validates-runner-tests
2017-07-06T18:01:49.683 [INFO] Error stacktraces are turned on.
2017-07-06T18:01:49.784 [INFO] Scanning for projects...
2017-07-06T18:01:51.193 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.5.0.Final/os-maven-plugin-1.5.0.Final.pom
2017-07-06T18:01:51.991 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/kr/motd/maven/os-maven-plugin/1.5.0.Final/os-maven-plugin-1.5.0.Final.pom
 (7 KB at 7.8 KB/sec)
2017-07-06T18:01:52.000 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
2017-07-06T18:01:52.036 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom
 (7 KB at 178.2 KB/sec)
2017-07-06T18:01:52.046 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
2017-07-06T18:01:52.082 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/3.2.1/maven-plugin-api-3.2.1.pom
 (4 KB at 94.4 KB/sec)
2017-07-06T18:01:52.084 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
2017-07-06T18:01:52.138 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven/3.2.1/maven-3.2.1.pom
 (23 KB at 408.5 KB/sec)
2017-07-06T18:01:52.141 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
2017-07-06T18:01:52.182 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/23/maven-parent-23.pom
 (32 KB at 776.2 KB/sec)
2017-07-06T18:01:52.187 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom
2017-07-06T18:01:52.221 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom (14 KB 
at 401.4 KB/sec)
2017-07-06T18:01:52.228 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/3.2.1/maven-model-3.2.1.pom
2017-07-06T18:01:52.257 [INFO] Downloaded: 
https://repo.mav