[jira] [Work logged] (BEAM-4519) Artifact Retrieval Service Protocol should be able to serve multiple Manifests.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4519?focusedWorklogId=110007=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-110007
 ]

ASF GitHub Bot logged work on BEAM-4519:


Author: ASF GitHub Bot
Created on: 08/Jun/18 05:50
Start Date: 08/Jun/18 05:50
Worklog Time Spent: 10m 
  Work Description: jkff commented on issue #5582: [BEAM-4519] Add 
staging_token to GetManifestRequest
URL: https://github.com/apache/beam/pull/5582#issuecomment-395654961
 
 
   It seems that the field rename has affected Java and Python code.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 110007)
Time Spent: 1.5h  (was: 1h 20m)

> Artifact Retrieval Service Protocol should be able to serve multiple 
> Manifests.
> ---
>
> Key: BEAM-4519
> URL: https://issues.apache.org/jira/browse/BEAM-4519
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> The artifact staging service currently returns a staging_token that can be 
> used as a key to access a manifest.  However, the current protocol does not 
> have a field that accepts this token.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #696

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-2915) Java SDK support for portable user state

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2915?focusedWorklogId=110004=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-110004
 ]

ASF GitHub Bot logged work on BEAM-2915:


Author: ASF GitHub Bot
Created on: 08/Jun/18 05:18
Start Date: 08/Jun/18 05:18
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #5445: [BEAM-2915, BEAM-4844] 
Add support for handling bag user state to the java-fn-execution library to 
support runner integration.
URL: https://github.com/apache/beam/pull/5445#issuecomment-395650277
 
 
   test this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 110004)
Time Spent: 3h  (was: 2h 50m)

> Java SDK support for portable user state
> 
>
> Key: BEAM-2915
> URL: https://issues.apache.org/jira/browse/BEAM-2915
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Minor
>  Labels: portability
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4520) No messages delivered after a while with PubsubIO

[Hrish (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hrish updated BEAM-4520:

Description: 
I am running the following Beam pipeline code locally, with the FlinkRunner. 
PubsubIO is used to read messages from a topic. I have a separate thread that 
publishes messages to the topic at regular intervals (every 30 seconds) and 
also sets the "ts" attribute which is used later to derive the event time.

Custom transform to convert to KV pair -
{code:java}
private static class PubSubMessageGrouper extends DoFn> {

@ProcessElement
public void processElement(ProcessContext c) {
PubsubMessage element = c.element();
KV kv = KV.of(element.getAttribute("key"), 
element);
c.output(kv);
}
}
{code}
Note that "key" is a key set in the message attributes earlier in the publisher 
thread. The intent is to group the messages downstream by this key.

Pipeline code -
{code:java}
PCollection pubsubColl = p
.apply(PubsubIO.readMessagesWithAttributes()
.withTimestampAttribute("ts")
.fromTopic("projects/" + projectName + "/topics/beamtest")
);


PCollection> idfied =
pubsubColl.apply(ParDo.of(new PubSubMessageGrouper()));

PCollection> windowed = idfied
.apply(Window.>into(FixedWindows.of(Duration.standardSeconds(15)))
.triggering(
Repeatedly.forever(
AfterWatermark.pastEndOfWindow()
)
)
.withAllowedLateness(Duration.standardSeconds(15))
.discardingFiredPanes());

PCollection>> grouped = 
windowed.apply(GroupByKey.create());

grouped.apply(ParDo.of(new KVPrinter()));
{code}
The transforms are not chained for ease of reading. The KVPrinter transform in 
the end is just to print out the messages received from the group by, which 
will be subsequently replaced by actual code once I get this running. When I 
run this, I don't find the trigger executing for quite some time (a couple of 
minutes or longer). When it finally triggers, I see that some of the messages 
are not received (in the final step), not matter how long I keep it running. 
The Pubsub statistics in my GCP/Stackdriver dashboard show that there is a 
backlog of undelivered messages.

Is this due to the internal watermark that PubsubIO uses? My intention here is 
to make sure that all messages are processed in the groupby, including late 
ones within the allowed lateness window.

Note that if I remove the GroupByKey, and just print the messages after the 
windowing, I can see all the messages.

  was:
I am running the following Beam pipeline code locally, with the FlinkRunner. 
PubsubIO is used to read messages from a topic. I have a separate thread that 
publishes messages to the topic at regular intervals (every 30 seconds) and 
also sets the "ts" attribute which is used later to derive the event time.

Custom transform to convert to KV pair -
{code:java}
private static class PubSubMessageGrouper extends DoFn> {

@ProcessElement
public void processElement(ProcessContext c) {
PubsubMessage element = c.element();
KV kv = KV.of(element.getAttribute("key"), 
element);
c.output(kv);
}
}
{code}
Note that "key" is a key set in the message attributes earlier in the publisher 
thread. The intent is to group the messages downstream by this key.

Pipeline code -
{code:java}
PCollection pubsubColl = p
.apply(PubsubIO.readMessagesWithAttributes()
.withTimestampAttribute("ts")
.fromTopic("projects/" + projectName + "/topics/beamtest")
);


PCollection> idfied =
pubsubColl.apply(ParDo.of(new PubSubMessageGrouper()));

PCollection> windowed = idfied
.apply(Window.>into(FixedWindows.of(Duration.standardSeconds(15)))
.triggering(
Repeatedly.forever(
AfterWatermark.pastEndOfWindow()
)
)
.withAllowedLateness(Duration.standardSeconds(15))
.discardingFiredPanes());

PCollection>> grouped = 
windowed.apply(GroupByKey.create());

grouped.apply(ParDo.of(new KVPrinter()));
{code}
The transforms are not chained for ease of reading. The KVPrinter transform in 
the end is just to print out the messages received from the group by, which 
will be subsequently replaced by actual code once I get this running. When I 
run this, I don't find the trigger executing for quite some time (a couple of 
minutes or longer). When it finally triggers, I see that some of the messages 
are not received (in the final step), not matter how long I keep it running. 
The Pubsub statistics in my GCP/Stackdriver dashboard show that there is a 
backlog of undelivered messages. 

Is this due to the internal watermark that PubsubIO uses? My intention here is 
to make sure that all messages are processed in the groupby, including late 
ones within 

[jira] [Work logged] (BEAM-4517) Add release status in README

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4517?focusedWorklogId=110001=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-110001
 ]

ASF GitHub Bot logged work on BEAM-4517:


Author: ASF GitHub Bot
Created on: 08/Jun/18 04:59
Start Date: 08/Jun/18 04:59
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5587: [BEAM-4517] Add 
PyPI status badge.
URL: https://github.com/apache/beam/pull/5587
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/README.md b/README.md
index 16907679631..81753620a62 100644
--- a/README.md
+++ b/README.md
@@ -24,6 +24,7 @@
 ## Status
 
 [![Maven 
Version](https://maven-badges.herokuapp.com/maven-central/org.apache.beam/beam-sdks-java-core/badge.svg)](http://search.maven.org/#search|gav|1|g:"org.apache.beam")
+[![PyPI 
version](https://badge.fury.io/py/apache-beam.svg)](https://badge.fury.io/py/apache-beam)
 [![Build 
Status](https://builds.apache.org/buildStatus/icon?job=beam_PostCommit_Java_GradleBuild)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild)
 [![Coverage 
Status](https://coveralls.io/repos/github/apache/beam/badge.svg?branch=master)](https://coveralls.io/github/apache/beam?branch=master)
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 110001)
Time Spent: 1h 10m  (was: 1h)

> Add release status in README
> 
>
> Key: BEAM-4517
> URL: https://issues.apache.org/jira/browse/BEAM-4517
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Xin Wang
>Assignee: Xin Wang
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Add maven/pypi status in the README.md file. This is very convenient for 
> users to know the latest release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4517) Add release status in README

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4517?focusedWorklogId=11=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-11
 ]

ASF GitHub Bot logged work on BEAM-4517:


Author: ASF GitHub Bot
Created on: 08/Jun/18 04:41
Start Date: 08/Jun/18 04:41
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #5587: [BEAM-4517] Add PyPI 
status badge.
URL: https://github.com/apache/beam/pull/5587#issuecomment-395644832
 
 
   LGTM. Thanks for adding this!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 11)
Time Spent: 1h  (was: 50m)

> Add release status in README
> 
>
> Key: BEAM-4517
> URL: https://issues.apache.org/jira/browse/BEAM-4517
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Xin Wang
>Assignee: Xin Wang
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Add maven/pypi status in the README.md file. This is very convenient for 
> users to know the latest release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4520) No messages delivered after a while with PubsubIO

[Hrish (JIRA)]

Hrish created BEAM-4520:
---

 Summary: No messages delivered after a while with PubsubIO
 Key: BEAM-4520
 URL: https://issues.apache.org/jira/browse/BEAM-4520
 Project: Beam
  Issue Type: Bug
  Components: io-java-gcp, runner-flink
Affects Versions: 2.4.0
Reporter: Hrish
Assignee: Chamikara Jayalath


I am running the following Beam pipeline code locally, with the FlinkRunner. 
PubsubIO is used to read messages from a topic. I have a separate thread that 
publishes messages to the topic at regular intervals (every 30 seconds) and 
also sets the "ts" attribute which is used later to derive the event time.

Custom transform to convert to KV pair -
{code:java}
private static class PubSubMessageGrouper extends DoFn> {

@ProcessElement
public void processElement(ProcessContext c) {
PubsubMessage element = c.element();
KV kv = KV.of(element.getAttribute("key"), 
element);
c.output(kv);
}
}
{code}
Note that "key" is a key set in the message attributes earlier in the publisher 
thread. The intent is to group the messages downstream by this key.

Pipeline code -
{code:java}
PCollection pubsubColl = p
.apply(PubsubIO.readMessagesWithAttributes()
.withTimestampAttribute("ts")
.fromTopic("projects/" + projectName + "/topics/beamtest")
);


PCollection> idfied =
pubsubColl.apply(ParDo.of(new PubSubMessageGrouper()));

PCollection> windowed = idfied
.apply(Window.>into(FixedWindows.of(Duration.standardSeconds(15)))
.triggering(
Repeatedly.forever(
AfterWatermark.pastEndOfWindow()
)
)
.withAllowedLateness(Duration.standardSeconds(15))
.discardingFiredPanes());

PCollection>> grouped = 
windowed.apply(GroupByKey.create());

grouped.apply(ParDo.of(new KVPrinter()));
{code}
The transforms are not chained for ease of reading. The KVPrinter transform in 
the end is just to print out the messages received from the group by, which 
will be subsequently replaced by actual code once I get this running. When I 
run this, I don't find the trigger executing for quite some time (a couple of 
minutes or longer). When it finally triggers, I see that some of the messages 
are not received (in the final step), not matter how long I keep it running. 
The Pubsub statistics in my GCP/Stackdriver dashboard show that there is a 
backlog of undelivered messages. 

Is this due to the internal watermark that PubsubIO uses? My intention here is 
to make sure that all messages are processed in the groupby, including late 
ones within the allowed lateness window.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-775) Remove Aggregators from the Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-775?focusedWorklogId=109998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109998
 ]

ASF GitHub Bot logged work on BEAM-775:
---

Author: ASF GitHub Bot
Created on: 08/Jun/18 04:31
Start Date: 08/Jun/18 04:31
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #2184: [BEAM-775] Remove 
Aggregators from PipelineResults and Examples in Java SDK
URL: https://github.com/apache/beam/pull/2184#issuecomment-395643545
 
 
   Hey! Yes, the way to do that now is by using `Metrics.counter`. Gald to
   help further if you have questions!
   
   On Thu, Jun 7, 2018, 7:18 PM Abdul Qadeer  wrote:
   
   > @pabloem  I was using Aggregator to keep
   > counter across each InputT bundles. What should be used now? Does Counter
   > work in my case?
   >
   > —
   > You are receiving this because you were mentioned.
   >
   >
   > Reply to this email directly, view it on GitHub
   > , or mute
   > the thread
   > 

   > .
   >
   -- 
   Got feedback? go/pabloem-feedback
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109998)
Time Spent: 40m  (was: 0.5h)

> Remove Aggregators from the Java SDK
> 
>
> Key: BEAM-775
> URL: https://issues.apache.org/jira/browse/BEAM-775
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Pablo Estrada
>Priority: Major
>  Labels: backward-incompatible
> Fix For: 2.0.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4519) Artifact Retrieval Service Protocol should be able to serve multiple Manifests.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4519?focusedWorklogId=109992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109992
 ]

ASF GitHub Bot logged work on BEAM-4519:


Author: ASF GitHub Bot
Created on: 08/Jun/18 03:54
Start Date: 08/Jun/18 03:54
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #5582: 
[BEAM-4519] Add staging_token to GetManifestRequest
URL: https://github.com/apache/beam/pull/5582#discussion_r193947173
 
 

 ##
 File path: model/job-management/src/main/proto/beam_artifact_api.proto
 ##
 @@ -81,7 +81,11 @@ message ProxyManifest {
 }
 
 // A request to get the manifest of a Job.
-message GetManifestRequest {}
+message GetManifestRequest {
+  // (Required) An opaque token representing the entirety of the staged 
artifacts.
 
 Review comment:
   I missed this comment in my fixup, but I agree that now is a good time to 
rename this field.  Fixed and documented.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109992)
Time Spent: 1h 20m  (was: 1h 10m)

> Artifact Retrieval Service Protocol should be able to serve multiple 
> Manifests.
> ---
>
> Key: BEAM-4519
> URL: https://issues.apache.org/jira/browse/BEAM-4519
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> The artifact staging service currently returns a staging_token that can be 
> used as a key to access a manifest.  However, the current protocol does not 
> have a field that accepts this token.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109991=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109991
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 03:42
Start Date: 08/Jun/18 03:42
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395636993
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109991)
Time Spent: 44h 10m  (was: 44h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 44h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109989=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109989
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 03:38
Start Date: 08/Jun/18 03:38
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395636536
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109989)
Time Spent: 44h  (was: 43h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 44h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109988=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109988
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 03:32
Start Date: 08/Jun/18 03:32
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395635678
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109988)
Time Spent: 43h 50m  (was: 43h 40m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 43h 50m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109987
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 03:28
Start Date: 08/Jun/18 03:28
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395635280
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109987)
Time Spent: 43h 40m  (was: 43.5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 43h 40m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109986=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109986
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 03:21
Start Date: 08/Jun/18 03:21
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395634329
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109986)
Time Spent: 43.5h  (was: 43h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 43.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109985
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 03:18
Start Date: 08/Jun/18 03:18
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395633919
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109985)
Time Spent: 43h 20m  (was: 43h 10m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 43h 20m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109984=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109984
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 03:04
Start Date: 08/Jun/18 03:04
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395631977
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109984)
Time Spent: 43h 10m  (was: 43h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 43h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4517) Add release status in README

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4517?focusedWorklogId=109982=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109982
 ]

ASF GitHub Bot logged work on BEAM-4517:


Author: ASF GitHub Bot
Created on: 08/Jun/18 03:02
Start Date: 08/Jun/18 03:02
Worklog Time Spent: 10m 
  Work Description: lukecwik opened a new pull request #5587: [BEAM-4517] 
Add PyPI status badge.
URL: https://github.com/apache/beam/pull/5587
 
 
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109982)
Time Spent: 40m  (was: 0.5h)

> Add release status in README
> 
>
> Key: BEAM-4517
> URL: https://issues.apache.org/jira/browse/BEAM-4517
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Xin Wang
>Assignee: Xin Wang
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Add maven/pypi status in the README.md file. This is very convenient for 
> users to know the latest release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4517) Add release status in README

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4517?focusedWorklogId=109983=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109983
 ]

ASF GitHub Bot logged work on BEAM-4517:


Author: ASF GitHub Bot
Created on: 08/Jun/18 03:02
Start Date: 08/Jun/18 03:02
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5587: [BEAM-4517] Add PyPI 
status badge.
URL: https://github.com/apache/beam/pull/5587#issuecomment-395631680
 
 
   R: @boyuanzz 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109983)
Time Spent: 50m  (was: 40m)

> Add release status in README
> 
>
> Key: BEAM-4517
> URL: https://issues.apache.org/jira/browse/BEAM-4517
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Xin Wang
>Assignee: Xin Wang
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Add maven/pypi status in the README.md file. This is very convenient for 
> users to know the latest release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4517) Add release status in README

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-4517:

Summary: Add release status in README  (was: Add maven status in README)

> Add release status in README
> 
>
> Key: BEAM-4517
> URL: https://issues.apache.org/jira/browse/BEAM-4517
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Xin Wang
>Assignee: Xin Wang
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Add maven status in the README.md file. This is very convenient for users to 
> know the latest release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4517) Add release status in README

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-4517:

Description: Add maven/pypi status in the README.md file. This is very 
convenient for users to know the latest release.  (was: Add maven status in the 
README.md file. This is very convenient for users to know the latest release.)

> Add release status in README
> 
>
> Key: BEAM-4517
> URL: https://issues.apache.org/jira/browse/BEAM-4517
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Xin Wang
>Assignee: Xin Wang
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Add maven/pypi status in the README.md file. This is very convenient for 
> users to know the latest release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109981
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 03:01
Start Date: 08/Jun/18 03:01
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395631457
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109981)
Time Spent: 43h  (was: 42h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 43h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4517) Add maven status in README

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4517.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Add maven status in README
> --
>
> Key: BEAM-4517
> URL: https://issues.apache.org/jira/browse/BEAM-4517
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Xin Wang
>Assignee: Xin Wang
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Add maven status in the README.md file. This is very convenient for users to 
> know the latest release.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109980
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 02:54
Start Date: 08/Jun/18 02:54
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395630391
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109980)
Time Spent: 42h 50m  (was: 42h 40m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 42h 50m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2915) Java SDK support for portable user state

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2915?focusedWorklogId=109978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109978
 ]

ASF GitHub Bot logged work on BEAM-2915:


Author: ASF GitHub Bot
Created on: 08/Jun/18 02:50
Start Date: 08/Jun/18 02:50
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5445: [BEAM-2915, 
BEAM-4844] Add support for handling bag user state to the java-fn-execution 
library to support runner integration.
URL: https://github.com/apache/beam/pull/5445#issuecomment-395629785
 
 
   It could be flaky, cut bugs for failing tests and assign to authors. Disable 
the test if it is consistently failing and update the bug saying that it has 
been disabled.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109978)
Time Spent: 2h 50m  (was: 2h 40m)

> Java SDK support for portable user state
> 
>
> Key: BEAM-2915
> URL: https://issues.apache.org/jira/browse/BEAM-2915
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Minor
>  Labels: portability
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109977
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 02:46
Start Date: 08/Jun/18 02:46
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395629046
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109977)
Time Spent: 42h 40m  (was: 42.5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 42h 40m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2915) Java SDK support for portable user state

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2915?focusedWorklogId=109973=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109973
 ]

ASF GitHub Bot logged work on BEAM-2915:


Author: ASF GitHub Bot
Created on: 08/Jun/18 02:31
Start Date: 08/Jun/18 02:31
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #5445: [BEAM-2915] Add 
support for handling bag user state to the java-fn-execution library to support 
runner integration.
URL: https://github.com/apache/beam/pull/5445#issuecomment-395626847
 
 
   Not clear why some PR pre-commit builds pass and others not. I wasn't able 
to build master locally. Flaky build?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109973)
Time Spent: 2h 40m  (was: 2.5h)

> Java SDK support for portable user state
> 
>
> Key: BEAM-2915
> URL: https://issues.apache.org/jira/browse/BEAM-2915
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Minor
>  Labels: portability
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109972
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 02:24
Start Date: 08/Jun/18 02:24
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395625795
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109972)
Time Spent: 42.5h  (was: 42h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 42.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2915) Java SDK support for portable user state

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2915?focusedWorklogId=109971=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109971
 ]

ASF GitHub Bot logged work on BEAM-2915:


Author: ASF GitHub Bot
Created on: 08/Jun/18 02:20
Start Date: 08/Jun/18 02:20
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5445: 
[BEAM-2915] Add support for handling bag user state to the java-fn-execution 
library to support runner integration.
URL: https://github.com/apache/beam/pull/5445#discussion_r19393
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
 ##
 @@ -181,6 +181,10 @@ private static boolean parDoCompatibility(
 // side inputs can be fused with other transforms in the same environment 
which are not
 // upstream of any of the side inputs.
 return pipeline.getSideInputs(parDo).isEmpty()
+// Since we lack the ability to mark upstream transforms as key 
preserving, we
+// purposefully break fusion here to provide runners the opportunity 
to insert a
+// grouping operation
+&& pipeline.getUserStates(parDo).isEmpty()
 
 Review comment:
   Runners need to look at the state spec and see if it exists, that would 
signal insertion of anything which does a partition by key (e.g. a reshuffle). 
Most runners (maybe all) only support user state via key partitioning.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109971)
Time Spent: 2.5h  (was: 2h 20m)

> Java SDK support for portable user state
> 
>
> Key: BEAM-2915
> URL: https://issues.apache.org/jira/browse/BEAM-2915
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Minor
>  Labels: portability
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-775) Remove Aggregators from the Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-775?focusedWorklogId=109969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109969
 ]

ASF GitHub Bot logged work on BEAM-775:
---

Author: ASF GitHub Bot
Created on: 08/Jun/18 02:19
Start Date: 08/Jun/18 02:19
Worklog Time Spent: 10m 
  Work Description: zorro786 commented on issue #2184: [BEAM-775] Remove 
Aggregators from PipelineResults and Examples in Java SDK
URL: https://github.com/apache/beam/pull/2184#issuecomment-395624706
 
 
   @pabloem I was using `Aggregator` to keep counter across each InputT 
bundles. What should be used now? Does `Counter` work in my case? If so what 
should be the implementing class of it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109969)
Time Spent: 20m  (was: 10m)

> Remove Aggregators from the Java SDK
> 
>
> Key: BEAM-775
> URL: https://issues.apache.org/jira/browse/BEAM-775
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Pablo Estrada
>Priority: Major
>  Labels: backward-incompatible
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-775) Remove Aggregators from the Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-775?focusedWorklogId=109970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109970
 ]

ASF GitHub Bot logged work on BEAM-775:
---

Author: ASF GitHub Bot
Created on: 08/Jun/18 02:19
Start Date: 08/Jun/18 02:19
Worklog Time Spent: 10m 
  Work Description: zorro786 commented on issue #2184: [BEAM-775] Remove 
Aggregators from PipelineResults and Examples in Java SDK
URL: https://github.com/apache/beam/pull/2184#issuecomment-395624706
 
 
   @pabloem I was using `Aggregator` to keep counter across each `InputT` 
bundles. What should be used now? Does `Counter` work in my case? If so what 
should be the implementing class of it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109970)
Time Spent: 0.5h  (was: 20m)

> Remove Aggregators from the Java SDK
> 
>
> Key: BEAM-775
> URL: https://issues.apache.org/jira/browse/BEAM-775
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Pablo Estrada
>Priority: Major
>  Labels: backward-incompatible
> Fix For: 2.0.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-775) Remove Aggregators from the Java SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-775?focusedWorklogId=109968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109968
 ]

ASF GitHub Bot logged work on BEAM-775:
---

Author: ASF GitHub Bot
Created on: 08/Jun/18 02:18
Start Date: 08/Jun/18 02:18
Worklog Time Spent: 10m 
  Work Description: zorro786 commented on issue #2184: [BEAM-775] Remove 
Aggregators from PipelineResults and Examples in Java SDK
URL: https://github.com/apache/beam/pull/2184#issuecomment-395624706
 
 
   @pabloem I was using `Aggregator` to keep counter across each InputT 
bundles. What should be used now? Does `Counter` work in my case?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109968)
Time Spent: 10m
Remaining Estimate: 0h

> Remove Aggregators from the Java SDK
> 
>
> Key: BEAM-775
> URL: https://issues.apache.org/jira/browse/BEAM-775
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Assignee: Pablo Estrada
>Priority: Major
>  Labels: backward-incompatible
> Fix For: 2.0.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109965
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 02:07
Start Date: 08/Jun/18 02:07
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395622926
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109965)
Time Spent: 42h 20m  (was: 42h 10m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 42h 20m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109963=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109963
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 01:58
Start Date: 08/Jun/18 01:58
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395621284
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109963)
Time Spent: 42h  (was: 41h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 42h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109964
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 01:58
Start Date: 08/Jun/18 01:58
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395621526
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109964)
Time Spent: 42h 10m  (was: 42h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 42h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109962=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109962
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 01:58
Start Date: 08/Jun/18 01:58
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395621373
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109962)
Time Spent: 41h 50m  (was: 41h 40m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 41h 50m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109961=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109961
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 01:57
Start Date: 08/Jun/18 01:57
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395620445
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109961)
Time Spent: 41h 40m  (was: 41.5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 41h 40m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109960
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 01:57
Start Date: 08/Jun/18 01:57
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395621284
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109960)
Time Spent: 41.5h  (was: 41h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 41.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109959=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109959
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 01:52
Start Date: 08/Jun/18 01:52
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395620445
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109959)
Time Spent: 41h 20m  (was: 41h 10m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 41h 20m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109958=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109958
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 01:48
Start Date: 08/Jun/18 01:48
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395619809
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109958)
Time Spent: 41h 10m  (was: 41h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 41h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109956=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109956
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 01:40
Start Date: 08/Jun/18 01:40
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395618708
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109956)
Time Spent: 41h  (was: 40h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 41h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109955=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109955
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 08/Jun/18 01:37
Start Date: 08/Jun/18 01:37
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395618225
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109955)
Time Spent: 40h 50m  (was: 40h 40m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 40h 50m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4388) Support optimized logical plan

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4388?focusedWorklogId=109953=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109953
 ]

ASF GitHub Bot logged work on BEAM-4388:


Author: ASF GitHub Bot
Created on: 08/Jun/18 01:30
Start Date: 08/Jun/18 01:30
Worklog Time Spent: 10m 
  Work Description: vectorijk commented on issue #5481: [BEAM-4388] Support 
optimized logical plan
URL: https://github.com/apache/beam/pull/5481#issuecomment-395617183
 
 
   ohhh. my mistake. I forgot to do spotlessApply again.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109953)
Time Spent: 3h  (was: 2h 50m)

> Support optimized logical plan
> --
>
> Key: BEAM-4388
> URL: https://issues.apache.org/jira/browse/BEAM-4388
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Kai Jiang
>Assignee: Kai Jiang
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Before converting into Beam Pipeline physical plan, logical plan should be 
> optimized and it will be super helpful for efficiently executing Beam 
> PTransforms pipeline. 
> Calcite has two ways for optimizing logical plan (HepPlanner and 
> VolcanoPlanner). We can support VolcanoPlanner first and apply calcite 
> builtin optimize rules (like 
> FilterJoinRule.FILTER_ON_JOIN) to sql query optimize plans.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=109948=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109948
 ]

ASF GitHub Bot logged work on BEAM-3981:


Author: ASF GitHub Bot
Created on: 08/Jun/18 01:19
Start Date: 08/Jun/18 01:19
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #5586: [BEAM-3981] Cleanup 
of coders futurization.
URL: https://github.com/apache/beam/pull/5586#issuecomment-395615484
 
 
   R: @RobbeSneyders, @tvalentyn 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109948)
Time Spent: 20h  (was: 19h 50m)

> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 20h
>  Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix 
> python 2 compatibility. This prepares the subpackage for python 3 support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=109947=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109947
 ]

ASF GitHub Bot logged work on BEAM-3981:


Author: ASF GitHub Bot
Created on: 08/Jun/18 01:18
Start Date: 08/Jun/18 01:18
Worklog Time Spent: 10m 
  Work Description: robertwb opened a new pull request #5586: [BEAM-3981] 
Cleanup of coders futurization.
URL: https://github.com/apache/beam/pull/5586
 
 
   Specifying language_level=3 allows the dict.items() optimization.
   
   This is about a 40% speedup for 1-element dicts as well as cleaner code.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109947)
Time Spent: 19h 50m  (was: 19h 40m)

> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 19h 50m
>  Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix 
> python 2 compatibility. This prepares the subpackage for python 3 support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #695

[Apache Jenkins Server]

See 


--
[...truncated 17.96 MB...]
org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
Jun 08, 2018 12:59:59 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-06-07_17_59_57-17865615850904472487
Jun 08, 2018 12:59:59 AM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-06-07_17_59_57-17865615850904472487 with 0 
expected assertions.
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T00:59:57.945Z: Autoscaling is enabled for job 
2018-06-07_17_59_57-17865615850904472487. The number of workers will be between 
1 and 1000.
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T00:59:57.968Z: Autoscaling was automatically enabled for 
job 2018-06-07_17_59_57-17865615850904472487.
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:01.734Z: Checking required Cloud APIs are enabled.
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:01.932Z: Checking permissions granted to controller 
Service Account.
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:06.281Z: Worker configuration: n1-standard-1 in 
us-central1-b.
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:06.764Z: Expanding CoGroupByKey operations into 
optimizable parts.
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:07.047Z: Expanding GroupByKey operations into 
optimizable parts.
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:07.089Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:07.438Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:07.630Z: Elided trivial flatten 
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:07.681Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:07.739Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:07.782Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:07.818Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:07.865Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Jun 08, 2018 1:00:14 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-08T01:00:07.907Z: Fusing consumer 

Jenkins build is back to normal : beam_PerformanceTests_MongoDBIO_IT #281

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #267

[Apache Jenkins Server]

See 


Changes:

[ehudm] Add a custom _url_dirname for local filesystems.

[github] Fix flaky comparison in log_handler_test.py

[jiangkai] add expression boolean casting

[jiangkai] add boolean type for agg function

[jiangkai] support EXISTS operator

[github] Update Environments.java

[github] Remove unneeded collection import.

[github] Combine immutability type fixes. (#3)

[github] fixup!

[lcwik] BEAM-3876 avoid NPE if checkpoint is null in an unbounded source

[lcwik] testing npe fix and exception rethrow in unit tests

[lcwik] Update https://github.com/apache/beam/pull/4894 to correspond with

--
[...truncated 435.21 KB...]
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:288)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
xmlioit0writethenreadall--06071724-73bg-harness-rl38.c.apache-beam-testing.internal/10.128.0.5
 to 190.239.211.130.bc.googleusercontent.com:9000 failed on connection 
exception: java.net.ConnectException: Connection refused; For more details see: 
 http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy64.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #276

[Apache Jenkins Server]

See 


Changes:

[ehudm] Add a custom _url_dirname for local filesystems.

[github] Fix flaky comparison in log_handler_test.py

[jiangkai] add expression boolean casting

[jiangkai] add boolean type for agg function

[jiangkai] support EXISTS operator

[github] Update Environments.java

[github] Remove unneeded collection import.

[github] Combine immutability type fixes. (#3)

[github] fixup!

[lcwik] BEAM-3876 avoid NPE if checkpoint is null in an unbounded source

[lcwik] testing npe fix and exception rethrow in unit tests

[lcwik] Update https://github.com/apache/beam/pull/4894 to correspond with

--
[...truncated 378.23 KB...]
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
textioit0writethenreadall-06071725-8tqj-harness-ptvh.c.apache-beam-testing.internal/10.128.0.35
 to 247.190.188.35.bc.googleusercontent.com:9000 failed on connection 
exception: java.net.ConnectException: Connection refused; For more details see: 
 http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy64.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #269

[Apache Jenkins Server]

See 


Changes:

[ehudm] Add a custom _url_dirname for local filesystems.

[github] Fix flaky comparison in log_handler_test.py

[jiangkai] add expression boolean casting

[jiangkai] add boolean type for agg function

[jiangkai] support EXISTS operator

[github] Update Environments.java

[github] Remove unneeded collection import.

[github] Combine immutability type fixes. (#3)

[github] fixup!

[lcwik] BEAM-3876 avoid NPE if checkpoint is null in an unbounded source

[lcwik] testing npe fix and exception rethrow in unit tests

[lcwik] Update https://github.com/apache/beam/pull/4894 to correspond with

--
[...truncated 379.88 KB...]
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
textioit0writethenreadall-06071725-zsj0-harness-04wl.c.apache-beam-testing.internal/10.128.0.33
 to 104.76.188.35.bc.googleusercontent.com:9000 failed on connection exception: 
java.net.ConnectException: Connection refused; For more details see:  
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy64.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #268

[Apache Jenkins Server]

See 


Changes:

[ehudm] Add a custom _url_dirname for local filesystems.

[github] Fix flaky comparison in log_handler_test.py

[jiangkai] add expression boolean casting

[jiangkai] add boolean type for agg function

[jiangkai] support EXISTS operator

[github] Update Environments.java

[github] Remove unneeded collection import.

[github] Combine immutability type fixes. (#3)

[github] fixup!

[lcwik] BEAM-3876 avoid NPE if checkpoint is null in an unbounded source

[lcwik] testing npe fix and exception rethrow in unit tests

[lcwik] Update https://github.com/apache/beam/pull/4894 to correspond with

--
[...truncated 457.62 KB...]
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy64.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy64.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 

[jira] [Work logged] (BEAM-4519) Artifact Retrieval Service Protocol should be able to serve multiple Manifests.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4519?focusedWorklogId=109940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109940
 ]

ASF GitHub Bot logged work on BEAM-4519:


Author: ASF GitHub Bot
Created on: 08/Jun/18 00:29
Start Date: 08/Jun/18 00:29
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #5582: [BEAM-4519] Add 
staging_token to GetManifestRequest
URL: https://github.com/apache/beam/pull/5582#issuecomment-395607701
 
 
   Protobufs regenerated.
   
   @jkff PTAL


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109940)
Time Spent: 1h 10m  (was: 1h)

> Artifact Retrieval Service Protocol should be able to serve multiple 
> Manifests.
> ---
>
> Key: BEAM-4519
> URL: https://issues.apache.org/jira/browse/BEAM-4519
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> The artifact staging service currently returns a staging_token that can be 
> used as a key to access a manifest.  However, the current protocol does not 
> have a field that accepts this token.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4519) Artifact Retrieval Service Protocol should be able to serve multiple Manifests.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4519?focusedWorklogId=109939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109939
 ]

ASF GitHub Bot logged work on BEAM-4519:


Author: ASF GitHub Bot
Created on: 08/Jun/18 00:28
Start Date: 08/Jun/18 00:28
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5582: 
[BEAM-4519] Add staging_token to GetManifestRequest
URL: https://github.com/apache/beam/pull/5582#discussion_r193924835
 
 

 ##
 File path: model/job-management/src/main/proto/beam_artifact_api.proto
 ##
 @@ -81,7 +81,11 @@ message ProxyManifest {
 }
 
 // A request to get the manifest of a Job.
-message GetManifestRequest {}
+message GetManifestRequest {
+  // (Required) An opaque token representing the entirety of the staged 
artifacts.
 
 Review comment:
   Maybe while we're at it let's rename it to retrieval_token? (here, in 
GetArtifactRequest, and in CommitManifestResponse; and also document in 
CommitManifestResponse that this token can be passed to GetManifestRequest and 
GetArtifactRequest)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109939)
Time Spent: 1h  (was: 50m)

> Artifact Retrieval Service Protocol should be able to serve multiple 
> Manifests.
> ---
>
> Key: BEAM-4519
> URL: https://issues.apache.org/jira/browse/BEAM-4519
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The artifact staging service currently returns a staging_token that can be 
> used as a key to access a manifest.  However, the current protocol does not 
> have a field that accepts this token.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #695

[Apache Jenkins Server]

See 


Changes:

[ehudm] Add a custom _url_dirname for local filesystems.

[github] Fix flaky comparison in log_handler_test.py

[jiangkai] add expression boolean casting

[jiangkai] add boolean type for agg function

[jiangkai] support EXISTS operator

[github] Update Environments.java

[github] Remove unneeded collection import.

[github] Combine immutability type fixes. (#3)

[github] fixup!

[lcwik] BEAM-3876 avoid NPE if checkpoint is null in an unbounded source

[lcwik] testing npe fix and exception rethrow in unit tests

[lcwik] Update https://github.com/apache/beam/pull/4894 to correspond with

--
[...truncated 122.67 KB...]
at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 
org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
at java.lang.Thread.run(Thread.java:748)

Jun 08, 2018 12:24:40 AM org.postgresql.Driver connect
SEVERE: Connection error: 
org.postgresql.util.PSQLException: The connection attempt failed.
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:257)
at 
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:452)
at org.postgresql.Driver.connect(Driver.java:254)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:94)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:79)
at 
org.apache.beam.sdk.io.common.DatabaseTestHelper.deleteTable(DatabaseTestHelper.java:57)
at org.apache.beam.sdk.io.jdbc.JdbcIOIT.tearDown(JdbcIOIT.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 

[jira] [Work logged] (BEAM-4479) Fixed document for Coder

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4479?focusedWorklogId=109937=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109937
 ]

ASF GitHub Bot logged work on BEAM-4479:


Author: ASF GitHub Bot
Created on: 08/Jun/18 00:21
Start Date: 08/Jun/18 00:21
Worklog Time Spent: 10m 
  Work Description: vesense commented on issue #459: [BEAM-4479] Fixed 
document for Coder
URL: https://github.com/apache/beam-site/pull/459#issuecomment-395606314
 
 
   @lukecwik Fixed. PTAL Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109937)
Time Spent: 2.5h  (was: 2h 20m)

> Fixed document for Coder
> 
>
> Key: BEAM-4479
> URL: https://issues.apache.org/jira/browse/BEAM-4479
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Xin Wang
>Assignee: Xin Wang
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> {code:java}CoderRegistry.getDefaultCoder{code} had been removed since 
> release-2.0.0, however, the document wasn't updated. This patch fixed that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=109931=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109931
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 08/Jun/18 00:06
Start Date: 08/Jun/18 00:06
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #5584: 
[BEAM-4291] Add distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584#discussion_r193922336
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/DfsArtifactRetrievalService.java
 ##
 @@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.artifact;
+
+
+import com.google.protobuf.ByteString;
+import io.grpc.stub.StreamObserver;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.fs.ResolveOptions;
+import org.apache.beam.sdk.io.fs.ResourceId;
+
+/**
+ * An {@link ArtifactRetrievalService} that uses distributed file systems as 
its backing storage.
+ */
+public class DfsArtifactRetrievalService
+extends ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceImplBase
+implements ArtifactRetrievalService {
+
+  private static int ARTIFACT_CHUNK_SIZE_BYTES = 1024 * 4;
+  // TODO: use the same constant as the corresponding ArtifactStagingService 
once it gets merged
+  private static String MANIFEST_NAME = "MANIFEST";
+
+  public static DfsArtifactRetrievalService create(ResourceId 
manifestResourceId) {
 
 Review comment:
   Thanks.  Will fix.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109931)
Time Spent: 1h 40m  (was: 1.5h)

> ArtifactRetrievalService that retrieves artifacts from a distributed 
> filesystem
> ---
>
> Key: BEAM-4291
> URL: https://issues.apache.org/jira/browse/BEAM-4291
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=109929=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109929
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 08/Jun/18 00:06
Start Date: 08/Jun/18 00:06
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #5584: 
[BEAM-4291] Add distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584#discussion_r193922254
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/DfsArtifactRetrievalService.java
 ##
 @@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.artifact;
+
+
+import com.google.protobuf.ByteString;
+import io.grpc.stub.StreamObserver;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.fs.ResolveOptions;
+import org.apache.beam.sdk.io.fs.ResourceId;
+
+/**
+ * An {@link ArtifactRetrievalService} that uses distributed file systems as 
its backing storage.
+ */
+public class DfsArtifactRetrievalService
+extends ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceImplBase
+implements ArtifactRetrievalService {
+
+  private static int ARTIFACT_CHUNK_SIZE_BYTES = 1024 * 4;
 
 Review comment:
   since my estimate was clearly low, what ballpark do you think is a more 
reasonable one?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109929)
Time Spent: 1.5h  (was: 1h 20m)

> ArtifactRetrievalService that retrieves artifacts from a distributed 
> filesystem
> ---
>
> Key: BEAM-4291
> URL: https://issues.apache.org/jira/browse/BEAM-4291
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=109930=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109930
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 08/Jun/18 00:06
Start Date: 08/Jun/18 00:06
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #5584: 
[BEAM-4291] Add distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584#discussion_r193922178
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/DfsArtifactRetrievalService.java
 ##
 @@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.artifact;
+
+
+import com.google.protobuf.ByteString;
+import io.grpc.stub.StreamObserver;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.fs.ResolveOptions;
+import org.apache.beam.sdk.io.fs.ResourceId;
+
+/**
+ * An {@link ArtifactRetrievalService} that uses distributed file systems as 
its backing storage.
 
 Review comment:
   As discussed in the other PR, I agree that this is a far better approach.  
I'll change this PR to integrate your suggestions.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109930)
Time Spent: 1h 40m  (was: 1.5h)

> ArtifactRetrievalService that retrieves artifacts from a distributed 
> filesystem
> ---
>
> Key: BEAM-4291
> URL: https://issues.apache.org/jira/browse/BEAM-4291
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4484) Shading model-pipeline / model-fn-execution / model-job-management produces corrupted classes

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4484:
---

Assignee: (was: Luke Cwik)

> Shading model-pipeline / model-fn-execution / model-job-management produces 
> corrupted classes
> -
>
> Key: BEAM-4484
> URL: https://issues.apache.org/jira/browse/BEAM-4484
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Luke Cwik
>Priority: Major
>
> During investigation for test failures on 
> [https://github.com/apache/beam/pull/5445,] I discovered that the the post 
> shaded beam-model-pipeline proto filedescriptors were corrupted. It turns out 
> that during the shading process, an over eager string replacement inside a 
> class is corrupting an internal field, in this case it modifies the RunnerApi 
> file descriptor storing
> {code:java}
> org.apache.beam.model.pipeline.v1.AccumulationMode.Enum{code}
> and changing it to
> {code:java}
> org.apache.beam.repackaged.beam_runners_direct_java.model.pipeline.v1.AccumulationMode.Enum{code}
> This problem exists because the proto package name and the java package name 
> collide and use org.apache.beam.model.pipeline.v1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4519) Artifact Retrieval Service Protocol should be able to serve multiple Manifests.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4519?focusedWorklogId=109926=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109926
 ]

ASF GitHub Bot logged work on BEAM-4519:


Author: ASF GitHub Bot
Created on: 07/Jun/18 23:27
Start Date: 07/Jun/18 23:27
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #5582: 
[BEAM-4519] Add staging_token to GetManifestRequest
URL: https://github.com/apache/beam/pull/5582#discussion_r193916818
 
 

 ##
 File path: model/job-management/src/main/proto/beam_artifact_api.proto
 ##
 @@ -81,7 +81,11 @@ message ProxyManifest {
 }
 
 // A request to get the manifest of a Job.
-message GetManifestRequest {}
+message GetManifestRequest {
+// (Required) An opaque token representing the entirety of the staged 
artifacts.
 
 Review comment:
   Okay, sounds good.  That would be a better arrangement.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109926)
Time Spent: 50m  (was: 40m)

> Artifact Retrieval Service Protocol should be able to serve multiple 
> Manifests.
> ---
>
> Key: BEAM-4519
> URL: https://issues.apache.org/jira/browse/BEAM-4519
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The artifact staging service currently returns a staging_token that can be 
> used as a key to access a manifest.  However, the current protocol does not 
> have a field that accepts this token.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=109921=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109921
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 07/Jun/18 23:13
Start Date: 07/Jun/18 23:13
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5584: 
[BEAM-4291] Add distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584#discussion_r193913642
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/DfsArtifactRetrievalService.java
 ##
 @@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.artifact;
+
+
+import com.google.protobuf.ByteString;
+import io.grpc.stub.StreamObserver;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.fs.ResolveOptions;
+import org.apache.beam.sdk.io.fs.ResourceId;
+
+/**
+ * An {@link ArtifactRetrievalService} that uses distributed file systems as 
its backing storage.
+ */
+public class DfsArtifactRetrievalService
+extends ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceImplBase
+implements ArtifactRetrievalService {
+
+  private static int ARTIFACT_CHUNK_SIZE_BYTES = 1024 * 4;
+  // TODO: use the same constant as the corresponding ArtifactStagingService 
once it gets merged
+  private static String MANIFEST_NAME = "MANIFEST";
+
+  public static DfsArtifactRetrievalService create(ResourceId 
manifestResourceId) {
 
 Review comment:
   The argument is unused


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109921)
Time Spent: 1h  (was: 50m)

> ArtifactRetrievalService that retrieves artifacts from a distributed 
> filesystem
> ---
>
> Key: BEAM-4291
> URL: https://issues.apache.org/jira/browse/BEAM-4291
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=109922=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109922
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 07/Jun/18 23:13
Start Date: 07/Jun/18 23:13
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5584: 
[BEAM-4291] Add distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584#discussion_r193914606
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/DfsArtifactRetrievalService.java
 ##
 @@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.artifact;
+
+
+import com.google.protobuf.ByteString;
+import io.grpc.stub.StreamObserver;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.fs.ResolveOptions;
+import org.apache.beam.sdk.io.fs.ResourceId;
+
+/**
+ * An {@link ArtifactRetrievalService} that uses distributed file systems as 
its backing storage.
 
 Review comment:
   So if I understand correctly this works like this, please correct me if I'm 
wrong:
   
   - It uses staging_token as the directory where the job's staging stuff is 
stored - both the artifacts and the manifest
   - The manifest is stored in a file $token/MANIFEST
   - Artifacts are stored in $token/$artifactName
   - The manifest is expected to have .metadata.name be full paths to the 
artifacts, i.e. $token/$originalName
   
   I would propose to change this a bit, considering that manifest should 
reflect what artifacts do we have rather than where they are stored; i.e. I 
think the name of the artifact in the manifest must be identical to its name in 
PutArtifactRequest, so it can not contain the staging token.
   
   I suggest something like the following:
   
   - Still use staging_token as the directory where the job's staging stuff is 
stored - both the artifacts and the manifest
   - Still store manifest in $token/MANIFEST
   - Manifest contains the original artifact names from PutArtifactRequest
   - Have an "artifact map" file e.g. $token/MAP, which contains a mapping from 
original artifact name to path on the storage, e.g.:
   ```
   {
 "foo.jar": "hdfs://path/to/staging/foo.jar",
 "some/data.pb": "hdfs://path/to/staging/some/data.pb",
 ...
   }
   ```
   
   Then when retrieving the artifact by name, you look it up in the map and 
download the file from where it points to.
   This flexible mapping will also allow to later store files in something like 
path/to/$md5, allowing for deduplication of artifacts (avoiding uploading 
artifacts we already have).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109922)
Time Spent: 1h 10m  (was: 1h)

> ArtifactRetrievalService that retrieves artifacts from a distributed 
> filesystem
> ---
>
> Key: BEAM-4291
> URL: https://issues.apache.org/jira/browse/BEAM-4291
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=109923=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109923
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 07/Jun/18 23:13
Start Date: 07/Jun/18 23:13
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5584: 
[BEAM-4291] Add distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584#discussion_r193913662
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/DfsArtifactRetrievalService.java
 ##
 @@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.fnexecution.artifact;
+
+
+import com.google.protobuf.ByteString;
+import io.grpc.stub.StreamObserver;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import org.apache.beam.model.jobmanagement.v1.ArtifactApi;
+import org.apache.beam.model.jobmanagement.v1.ArtifactRetrievalServiceGrpc;
+import org.apache.beam.sdk.io.FileSystems;
+import org.apache.beam.sdk.io.fs.ResolveOptions;
+import org.apache.beam.sdk.io.fs.ResourceId;
+
+/**
+ * An {@link ArtifactRetrievalService} that uses distributed file systems as 
its backing storage.
+ */
+public class DfsArtifactRetrievalService
+extends ArtifactRetrievalServiceGrpc.ArtifactRetrievalServiceImplBase
+implements ArtifactRetrievalService {
+
+  private static int ARTIFACT_CHUNK_SIZE_BYTES = 1024 * 4;
 
 Review comment:
   Should probably be quite a bit bigger for performance


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109923)
Time Spent: 1h 20m  (was: 1h 10m)

> ArtifactRetrievalService that retrieves artifacts from a distributed 
> filesystem
> ---
>
> Key: BEAM-4291
> URL: https://issues.apache.org/jira/browse/BEAM-4291
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4519) Artifact Retrieval Service Protocol should be able to serve multiple Manifests.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4519?focusedWorklogId=109916=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109916
 ]

ASF GitHub Bot logged work on BEAM-4519:


Author: ASF GitHub Bot
Created on: 07/Jun/18 23:05
Start Date: 07/Jun/18 23:05
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5582: 
[BEAM-4519] Add staging_token to GetManifestRequest
URL: https://github.com/apache/beam/pull/5582#discussion_r193913479
 
 

 ##
 File path: model/job-management/src/main/proto/beam_artifact_api.proto
 ##
 @@ -81,7 +81,11 @@ message ProxyManifest {
 }
 
 // A request to get the manifest of a Job.
-message GetManifestRequest {}
+message GetManifestRequest {
+// (Required) An opaque token representing the entirety of the staged 
artifacts.
 
 Review comment:
   Also: this token should be in GetArtifactRequest as well - the token 
basically identifies the job whose artifacts we're talking about.
   
   I see in the other PR that you've been using the manifest itself for that, 
but I don't think that's a good idea: in my view, the manifest contains 
information about artifacts that's independent about how they're stored, e.g. 
the artifact name in the manifest must NOT be a full path to the file on a 
distributed filesystem, it must be the original name of the file.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109916)
Time Spent: 40m  (was: 0.5h)

> Artifact Retrieval Service Protocol should be able to serve multiple 
> Manifests.
> ---
>
> Key: BEAM-4519
> URL: https://issues.apache.org/jira/browse/BEAM-4519
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The artifact staging service currently returns a staging_token that can be 
> used as a key to access a manifest.  However, the current protocol does not 
> have a field that accepts this token.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=109917=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109917
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 07/Jun/18 23:05
Start Date: 07/Jun/18 23:05
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #5584: [BEAM-4291] Add 
distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584#issuecomment-395593162
 
 
   Also, I would welcome any advice on how to properly test a service like this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109917)
Time Spent: 40m  (was: 0.5h)

> ArtifactRetrievalService that retrieves artifacts from a distributed 
> filesystem
> ---
>
> Key: BEAM-4291
> URL: https://issues.apache.org/jira/browse/BEAM-4291
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=109918=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109918
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 07/Jun/18 23:05
Start Date: 07/Jun/18 23:05
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #5584: [BEAM-4291] Add 
distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584#issuecomment-395593162
 
 
   Also, I would welcome any advice on how to properly author unit tests for a 
service like this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109918)
Time Spent: 50m  (was: 40m)

> ArtifactRetrievalService that retrieves artifacts from a distributed 
> filesystem
> ---
>
> Key: BEAM-4291
> URL: https://issues.apache.org/jira/browse/BEAM-4291
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4519) Artifact Retrieval Service Protocol should be able to serve multiple Manifests.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4519?focusedWorklogId=109915=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109915
 ]

ASF GitHub Bot logged work on BEAM-4519:


Author: ASF GitHub Bot
Created on: 07/Jun/18 23:03
Start Date: 07/Jun/18 23:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5582: 
[BEAM-4519] Add staging_token to GetManifestRequest
URL: https://github.com/apache/beam/pull/5582#discussion_r193913154
 
 

 ##
 File path: model/job-management/src/main/proto/beam_artifact_api.proto
 ##
 @@ -81,7 +81,11 @@ message ProxyManifest {
 }
 
 // A request to get the manifest of a Job.
-message GetManifestRequest {}
+message GetManifestRequest {
+// (Required) An opaque token representing the entirety of the staged 
artifacts.
 
 Review comment:
   Reindent and regenerate the Go protos? (sync with Ankur about how to do 
that; I think Python protos don't need to be regenerated since they're not 
checked in)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109915)
Time Spent: 0.5h  (was: 20m)

> Artifact Retrieval Service Protocol should be able to serve multiple 
> Manifests.
> ---
>
> Key: BEAM-4519
> URL: https://issues.apache.org/jira/browse/BEAM-4519
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The artifact staging service currently returns a staging_token that can be 
> used as a key to access a manifest.  However, the current protocol does not 
> have a field that accepts this token.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=109914=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109914
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 07/Jun/18 23:01
Start Date: 07/Jun/18 23:01
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #5584: [BEAM-4291] Add 
distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584#issuecomment-395592395
 
 
   This PR is kept separate from the protocol changes just because I don't want 
to get into the habit of sneaking protocol changes into PRs that implement 
features.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109914)
Time Spent: 0.5h  (was: 20m)

> ArtifactRetrievalService that retrieves artifacts from a distributed 
> filesystem
> ---
>
> Key: BEAM-4291
> URL: https://issues.apache.org/jira/browse/BEAM-4291
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #445

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109907=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109907
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 07/Jun/18 22:42
Start Date: 07/Jun/18 22:42
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395588563
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109907)
Time Spent: 40h 40m  (was: 40.5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 40h 40m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4507) Late elements windowed into expired fixed windows get dropped in Directrunner

[Batkhuyag Batsaikhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Batkhuyag Batsaikhan updated BEAM-4507:
---
Summary: Late elements windowed into expired fixed windows get dropped in 
Directrunner  (was: Objects that have Fixed Window after a Global Window 
disappear in Directrunner)

> Late elements windowed into expired fixed windows get dropped in Directrunner
> -
>
> Key: BEAM-4507
> URL: https://issues.apache.org/jira/browse/BEAM-4507
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Affects Versions: 2.4.0
>Reporter: Reza ardeshir rokni
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 2.5.0
>
>
> 1- Create a elements that go into a combinar and then drops into a  Keyed 
> Global Window, on entry setup a timer.
> Have the OnTime function:
> 1- c.output the original element with a timestamp == to the original timestamp
> 2- Create a new timestamp with element in the next window
> 2- LOG the output in a DoFn after the keyed state code, you should see all 
> the values.
> 3- Now apply a FixedWindow... and then try and log the output.. the original 
> elements wont appear, but the new OnTime generated ones will.
>  
> Tested in Dataflow runner and things work ok, the direct runner fails. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4388) Support optimized logical plan

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4388?focusedWorklogId=109893=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109893
 ]

ASF GitHub Bot logged work on BEAM-4388:


Author: ASF GitHub Bot
Created on: 07/Jun/18 22:18
Start Date: 07/Jun/18 22:18
Worklog Time Spent: 10m 
  Work Description: XuMingmin commented on issue #5481: [BEAM-4388] Support 
optimized logical plan
URL: https://github.com/apache/beam/pull/5481#issuecomment-395583612
 
 
   there's a format error in new added lines, can you fix it?
   ```
   FAILURE: Build completed with 2 failures.
   
   1: Task failed with an exception.
   ---
   * What went wrong:
   Execution failed for task ':beam-sdks-java-extensions-sql:spotlessJava'.
   > The following files had format violations:
 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlExplainTest.java
 @@ -28,80 +28,78 @@
  import·org.junit.Before;
  import·org.junit.Test;
  
 -/**
 -·*·UnitTest·for·{@link·BeamSqlParserImpl}.
 -·*/
 +/**·UnitTest·for·{@link·BeamSqlParserImpl}.·*/
  public·class·BeamSqlExplainTest·{
 -private·InMemoryMetaStore·metaStore;
 -private·BeamSqlCli·cli;
 +··private·InMemoryMetaStore·metaStore;
 +··private·BeamSqlCli·cli;
  
 -@Before
 
-public·void·setUp()·throws·SqlParseException,·RelConversionException,·ValidationException·{
 -metaStore·=·new·InMemoryMetaStore();
 -metaStore.registerProvider(new·TextTableProvider());
 -cli·=·new·BeamSqlCli().metaStore(metaStore);
 +··@Before
 
+··public·void·setUp()·throws·SqlParseException,·RelConversionException,·ValidationException·{
 +metaStore·=·new·InMemoryMetaStore();
 +metaStore.registerProvider(new·TextTableProvider());
 +cli·=·new·BeamSqlCli().metaStore(metaStore);
  
 -cli.execute(
 -"create·table·person·(\n"
 -+·"id·int·COMMENT·'id',·\n"
 -+·"name·varchar·COMMENT·'name',·\n"
 -+·"age·int·COMMENT·'age')·\n"
 -+·"TYPE·'text'·\n"
 -+·"COMMENT·''·");
 +cli.execute(
 +"create·table·person·(\n"
 ++·"id·int·COMMENT·'id',·\n"
 ++·"name·varchar·COMMENT·'name',·\n"
 ++·"age·int·COMMENT·'age')·\n"
 ++·"TYPE·'text'·\n"
 ++·"COMMENT·''·");
  
 -cli.execute(
 -"create·table·A·(\n"
 -+·"c1·int·COMMENT·'c1',\n"
 -+·"c2·int·COMMENT·'c2')\n"
 -+·"TYPE·'text'\n"
 -+·"COMMENT·''·");
 +cli.execute(
 +"create·table·A·(\n"
 ++·"c1·int·COMMENT·'c1',\n"
 ... (95 more lines that didn't fit)
 Run 'gradlew spotlessApply' to fix these violations.
   
   * Try:
   Run with --stacktrace option to get the stack trace. Run with --debug option 
to get more log output. Run with --scan to get full insights.
   
==
   
   2: Task failed with an exception.
   ---
   * What went wrong:
   Execution failed for task ':beam-sdks-java-extensions-sql:checkstyleTest'.
   > Checkstyle rule violations were found. See the report at: 
file:///home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_GradleBuild/src/sdks/java/extensions/sql/build/reports/checkstyle/test.html
 Checkstyle files with violations: 1
 Checkstyle violations by severity: [error:2]
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109893)
Time Spent: 2h 50m  (was: 2h 40m)

> Support optimized logical plan
> --
>
> Key: BEAM-4388
> URL: https://issues.apache.org/jira/browse/BEAM-4388
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Kai Jiang
>Assignee: Kai Jiang
> 

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=109892=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109892
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 07/Jun/18 22:18
Start Date: 07/Jun/18 22:18
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #5584: [BEAM-4291] Add 
distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584#issuecomment-395583550
 
 
   Merge Blocked by #5582 
   CC: @bsidhom @jkff @angoenka 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109892)
Time Spent: 20m  (was: 10m)

> ArtifactRetrievalService that retrieves artifacts from a distributed 
> filesystem
> ---
>
> Key: BEAM-4291
> URL: https://issues.apache.org/jira/browse/BEAM-4291
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4507) Objects that have Fixed Window after a Global Window disappear in Directrunner

[Batkhuyag Batsaikhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Batkhuyag Batsaikhan closed BEAM-4507.
--
   Resolution: Won't Fix
Fix Version/s: 2.5.0

> Objects that have Fixed Window after a Global Window disappear in Directrunner
> --
>
> Key: BEAM-4507
> URL: https://issues.apache.org/jira/browse/BEAM-4507
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Affects Versions: 2.4.0
>Reporter: Reza ardeshir rokni
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
> Fix For: 2.5.0
>
>
> 1- Create a elements that go into a combinar and then drops into a  Keyed 
> Global Window, on entry setup a timer.
> Have the OnTime function:
> 1- c.output the original element with a timestamp == to the original timestamp
> 2- Create a new timestamp with element in the next window
> 2- LOG the output in a DoFn after the keyed state code, you should see all 
> the values.
> 3- Now apply a FixedWindow... and then try and log the output.. the original 
> elements wont appear, but the new OnTime generated ones will.
>  
> Tested in Dataflow runner and things work ok, the direct runner fails. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4291?focusedWorklogId=109891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109891
 ]

ASF GitHub Bot logged work on BEAM-4291:


Author: ASF GitHub Bot
Created on: 07/Jun/18 22:17
Start Date: 07/Jun/18 22:17
Worklog Time Spent: 10m 
  Work Description: axelmagn opened a new pull request #5584: [BEAM-4291] 
Add distributed artifact retrieval
URL: https://github.com/apache/beam/pull/5584
 
 
   
   Implement an ArtifactRetrievalService that uses the cloud storage
   solutions supported by sdk.io.FileSystems.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109891)
Time Spent: 10m
Remaining Estimate: 0h

> ArtifactRetrievalService that retrieves artifacts from a distributed 
> filesystem
> ---
>
> Key: BEAM-4291
> URL: https://issues.apache.org/jira/browse/BEAM-4291
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4507) Objects that have Fixed Window after a Global Window disappear in Directrunner

[Batkhuyag Batsaikhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505385#comment-16505385
 ] 

Batkhuyag Batsaikhan commented on BEAM-4507:


The solution around this problem was to add enough allowed lateness to the new 
fixed windows, so that the windows don't expire when a late element arrived. 
Closing this Jira now.

> Objects that have Fixed Window after a Global Window disappear in Directrunner
> --
>
> Key: BEAM-4507
> URL: https://issues.apache.org/jira/browse/BEAM-4507
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Affects Versions: 2.4.0
>Reporter: Reza ardeshir rokni
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
>
> 1- Create a elements that go into a combinar and then drops into a  Keyed 
> Global Window, on entry setup a timer.
> Have the OnTime function:
> 1- c.output the original element with a timestamp == to the original timestamp
> 2- Create a new timestamp with element in the next window
> 2- LOG the output in a DoFn after the keyed state code, you should see all 
> the values.
> 3- Now apply a FixedWindow... and then try and log the output.. the original 
> elements wont appear, but the new OnTime generated ones will.
>  
> Tested in Dataflow runner and things work ok, the direct runner fails. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109890=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109890
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 07/Jun/18 22:16
Start Date: 07/Jun/18 22:16
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395583098
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109890)
Time Spent: 40.5h  (was: 40h 20m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 40.5h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109882
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 07/Jun/18 22:03
Start Date: 07/Jun/18 22:03
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-395579939
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109882)
Time Spent: 40h 10m  (was: 40h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 40h 10m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=109884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109884
 ]

ASF GitHub Bot logged work on BEAM-3981:


Author: ASF GitHub Bot
Created on: 07/Jun/18 22:05
Start Date: 07/Jun/18 22:05
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #5053: [BEAM-3981] 
Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#issuecomment-395580375
 
 
   Hmm. I am pretty sure my microbenchmark uses a Cython codepath since in 
order for any code change to take effect I have to run  `python setup.py 
build_ext --inplace` to recompile associated C extensions.  I checked once 
again and I do see a 2x slowdown with `items()` once the size of the dictionary 
goes more than 10 elements. Here's code generated by Cython: a 
https://docs.google.com/document/d/1S-oeqJGiMHt_L3iCgr9dYfQdR0_ukQE25mcvK-BqudU/edit#heading=h.drcukhvo4hd6.
 Perhaps the slowdown is related to materializing the list of keys? 
   
   My microbenchmark setup is here: 
https://github.com/apache/beam/compare/master...tvalentyn:coders_dict_microbencmark?expand=1


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109884)
Time Spent: 19h 40m  (was: 19.5h)

> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 19h 40m
>  Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix 
> python 2 compatibility. This prepares the subpackage for python 3 support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=109883=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109883
 ]

ASF GitHub Bot logged work on BEAM-3981:


Author: ASF GitHub Bot
Created on: 07/Jun/18 22:05
Start Date: 07/Jun/18 22:05
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #5053: [BEAM-3981] 
Futurize coders subpackage
URL: https://github.com/apache/beam/pull/5053#issuecomment-395580375
 
 
   Hmm. I am pretty sure my microbenchmark uses a Cython codepath since in 
order for any code change to take effect I have to run  `python setup.py 
build_ext --inplace` to recompile associated C extensions.  I checked once 
again and I do see a 2x slowdown with `items()` once the size of the dictionary 
goes more than 10 elements. Here's code generated by Cython: a 
https://docs.google.com/document/d/1S-oeqJGiMHt_L3iCgr9dYfQdR0_ukQE25mcvK-BqudU/edit#heading=h.drcukhvo4hd6.
 Perhaps the slowdown is related to materializing the list? 
   
   My microbenchmark setup is here: 
https://github.com/apache/beam/compare/master...tvalentyn:coders_dict_microbencmark?expand=1


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109883)
Time Spent: 19.5h  (was: 19h 20m)

> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 19.5h
>  Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix 
> python 2 compatibility. This prepares the subpackage for python 3 support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #694

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=109879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109879
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 07/Jun/18 21:52
Start Date: 07/Jun/18 21:52
Worklog Time Spent: 10m 
  Work Description: lostluck opened a new pull request #5581: [BEAM-4276] 
Add missing scope line
URL: https://github.com/apache/beam/pull/5581
 
 
   Without this line, no transforms can get lifted. It was missed from the 
previous PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109879)
Time Spent: 7h 50m  (was: 7h 40m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 7h 50m
>  Remaining Estimate: 0h
>
> Specifically add the necessary code to produce a Combine Composite with the 
> correct URN, and permit the SDK harness to understand the lifted parts when 
> receiving a bundle plan from the worker.
> Not expected as part of this issue is:
> Additional performance tweaks to the in memory cache (See 
> [BEAM-4468|https://issues.apache.org/jira/browse/BEAM-4468])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=109877=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109877
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 07/Jun/18 21:52
Start Date: 07/Jun/18 21:52
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #5581: [BEAM-4276] Add 
missing scope line
URL: https://github.com/apache/beam/pull/5581#issuecomment-395577396
 
 
   R: @lukecwik Please merge, thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109877)
Time Spent: 7.5h  (was: 7h 20m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> Specifically add the necessary code to produce a Combine Composite with the 
> correct URN, and permit the SDK harness to understand the lifted parts when 
> receiving a bundle plan from the worker.
> Not expected as part of this issue is:
> Additional performance tweaks to the in memory cache (See 
> [BEAM-4468|https://issues.apache.org/jira/browse/BEAM-4468])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4519) Artifact Retrieval Service Protocol should be able to serve multiple Manifests.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4519?focusedWorklogId=109876=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109876
 ]

ASF GitHub Bot logged work on BEAM-4519:


Author: ASF GitHub Bot
Created on: 07/Jun/18 21:52
Start Date: 07/Jun/18 21:52
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #5582: [BEAM-4519] Add 
staging_token to GetManifestRequest
URL: https://github.com/apache/beam/pull/5582#issuecomment-395577377
 
 
   R: @jkff 
   CC: @angoenka @bsidhom 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109876)
Time Spent: 20m  (was: 10m)

> Artifact Retrieval Service Protocol should be able to serve multiple 
> Manifests.
> ---
>
> Key: BEAM-4519
> URL: https://issues.apache.org/jira/browse/BEAM-4519
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The artifact staging service currently returns a staging_token that can be 
> used as a key to access a manifest.  However, the current protocol does not 
> have a field that accepts this token.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=109878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109878
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 07/Jun/18 21:52
Start Date: 07/Jun/18 21:52
Worklog Time Spent: 10m 
  Work Description: lostluck closed pull request #5581: [BEAM-4276] Add 
missing scope line
URL: https://github.com/apache/beam/pull/5581
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/go/pkg/beam/combine.go b/sdks/go/pkg/beam/combine.go
index d4508c7379b..85d99464a1f 100644
--- a/sdks/go/pkg/beam/combine.go
+++ b/sdks/go/pkg/beam/combine.go
@@ -50,6 +50,7 @@ func TryCombine(s Scope, combinefn interface{}, col 
PCollection) (PCollection, e
 // for multiple reasons, notably that the combinefn is not valid or cannot be 
bound
 // -- due to type mismatch, say -- to the incoming PCollection.
 func TryCombinePerKey(s Scope, combinefn interface{}, col PCollection) 
(PCollection, error) {
+   s = s.Scope(graph.CombinePerKeyScope)
ValidateKVType(col)
col, err := TryGroupByKey(s, col)
if err != nil {


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109878)
Time Spent: 7h 40m  (was: 7.5h)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> Specifically add the necessary code to produce a Combine Composite with the 
> correct URN, and permit the SDK harness to understand the lifted parts when 
> receiving a bundle plan from the worker.
> Not expected as part of this issue is:
> Additional performance tweaks to the in memory cache (See 
> [BEAM-4468|https://issues.apache.org/jira/browse/BEAM-4468])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4519) Artifact Retrieval Service Protocol should be able to serve multiple Manifests.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4519?focusedWorklogId=109875=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109875
 ]

ASF GitHub Bot logged work on BEAM-4519:


Author: ASF GitHub Bot
Created on: 07/Jun/18 21:52
Start Date: 07/Jun/18 21:52
Worklog Time Spent: 10m 
  Work Description: axelmagn opened a new pull request #5582: [BEAM-4519] 
Add staging_token to GetManifestRequest
URL: https://github.com/apache/beam/pull/5582
 
 
   This allows a single ArtifactRetrievalService to serve the artifacts
   from multiple manifests, indexed by staging token.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109875)
Time Spent: 10m
Remaining Estimate: 0h

> Artifact Retrieval Service Protocol should be able to serve multiple 
> Manifests.
> ---
>
> Key: BEAM-4519
> URL: https://issues.apache.org/jira/browse/BEAM-4519
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The artifact staging service currently returns a staging_token that can be 
> used as a key to access a manifest.  However, the current protocol does not 
> have a field that accepts this token.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4519) Artifact Retrieval Service Protocol should be able to serve multiple Manifests.

[Axel Magnuson (JIRA)]

Axel Magnuson created BEAM-4519:
---

 Summary: Artifact Retrieval Service Protocol should be able to 
serve multiple Manifests.
 Key: BEAM-4519
 URL: https://issues.apache.org/jira/browse/BEAM-4519
 Project: Beam
  Issue Type: Bug
  Components: runner-core
Reporter: Axel Magnuson
Assignee: Axel Magnuson


The artifact staging service currently returns a staging_token that can be used 
as a key to access a manifest.  However, the current protocol does not have a 
field that accepts this token.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2083) Develop a Go SDK for Beam

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-2083.
-
   Resolution: Fixed
Fix Version/s: 2.5.0

> Develop a Go SDK for Beam
> -
>
> Key: BEAM-2083
> URL: https://issues.apache.org/jira/browse/BEAM-2083
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Henning Rohde
>Priority: Major
> Fix For: 2.5.0
>
>
> Allow users of the Go programming language (https://golang.org/) to write 
> Beam pipelines in this language. The effort is focusing on full-fledged SDK 
> that leverages the Beam Fn API to bootstrap a native Go experience.
> Initial design:
> https://s.apache.org/beam-go-sdk-design-rfc
> Development in the master branch. Work in progress. YMMV.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=109874=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109874
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 07/Jun/18 21:48
Start Date: 07/Jun/18 21:48
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #5581: [BEAM-4276] Add 
missing scope line
URL: https://github.com/apache/beam/pull/5581#issuecomment-395576275
 
 
   R: @herohde 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109874)
Time Spent: 7h 20m  (was: 7h 10m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> Specifically add the necessary code to produce a Combine Composite with the 
> correct URN, and permit the SDK harness to understand the lifted parts when 
> receiving a bundle plan from the worker.
> Not expected as part of this issue is:
> Additional performance tweaks to the in memory cache (See 
> [BEAM-4468|https://issues.apache.org/jira/browse/BEAM-4468])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=109872=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109872
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 07/Jun/18 21:47
Start Date: 07/Jun/18 21:47
Worklog Time Spent: 10m 
  Work Description: lostluck opened a new pull request #5581: [BEAM-4276] 
Add missing scope line
URL: https://github.com/apache/beam/pull/5581
 
 
   Without this line, no transforms can get lifted. It was missed from the 
previous PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109872)
Time Spent: 7h 10m  (was: 7h)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> Specifically add the necessary code to produce a Combine Composite with the 
> correct URN, and permit the SDK harness to understand the lifted parts when 
> receiving a bundle plan from the worker.
> Not expected as part of this issue is:
> Additional performance tweaks to the in memory cache (See 
> [BEAM-4468|https://issues.apache.org/jira/browse/BEAM-4468])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4432) Performance tests need a way to generate Synthetic data

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4432?focusedWorklogId=109865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109865
 ]

ASF GitHub Bot logged work on BEAM-4432:


Author: ASF GitHub Bot
Created on: 07/Jun/18 21:34
Start Date: 07/Jun/18 21:34
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #5519: [BEAM-4432] Adding 
Sources to produce Synthetic output for Batch pipelines
URL: https://github.com/apache/beam/pull/5519#issuecomment-395572681
 
 
   Hi Pablo my excuses I have been quite busy these last days. I expect to do a 
first round of the Java part tomorrow.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109865)
Time Spent: 1h 10m  (was: 1h)

> Performance tests need a way to generate Synthetic data
> ---
>
> Key: BEAM-4432
> URL: https://issues.apache.org/jira/browse/BEAM-4432
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Minor
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> GenerateSequence fal.lls short in this regard, as we may want to generate 
> data in custom distributions, or with specific repeatability requirements / 
> and hardcoded delays for autoscaling.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4313) Enforce ErrorProne analysis in Dataflow runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4313?focusedWorklogId=109864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109864
 ]

ASF GitHub Bot logged work on BEAM-4313:


Author: ASF GitHub Bot
Created on: 07/Jun/18 21:28
Start Date: 07/Jun/18 21:28
Worklog Time Spent: 10m 
  Work Description: swegner commented on a change in pull request #5540: 
[BEAM-4313] Fix and enforce FindBugs and ErrorProne violations in Dataflow 
Runner
URL: https://github.com/apache/beam/pull/5540#discussion_r193894420
 
 

 ##
 File path: sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
 ##
 @@ -139,6 +139,33 @@
 
   
 
+  
+
+
+
 
 Review comment:
   Can you elaborate on what has changed? Here's my understanding of things:
   
   * FindBugs uses a 
[`@SuppressFBWarnings`](http://findbugs.sourceforge.net/api/edu/umd/cs/findbugs/annotations/SuppressFBWarnings.html)
 annotation for inline suppressions.
   * There is also a 
[`@edu.umd.cs.findbugs.annotations.SuppressWarnings`](http://findbugs.sourceforge.net/api/edu/umd/cs/findbugs/annotations/SuppressWarnings.html),
 but it is marked as deprecated in favor of `@SuppressFBWarnings` since it 
clashes the other `@SuppressWarnings` annotation
   * We [previously removed 
references](https://github.com/apache/beam/pull/1401) to `@SuppressFBWarnings` 
because it is only available via 
[`com.google.code.findbugs:annotations](https://mvnrepository.com/artifact/com.google.code.findbugs/annotations)
 under LGPL.
   
   Also it seems the general direction is to stop supporting FindBugs in favor 
of ErrorProne / SpotBugs. So it doesn't seem worth much to invest in migrating 
the suppressions file.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109864)
Time Spent: 2h 10m  (was: 2h)

> Enforce ErrorProne analysis in Dataflow runner project
> --
>
> Key: BEAM-4313
> URL: https://issues.apache.org/jira/browse/BEAM-4313
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-google-cloud-dataflow-java}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-google-cloud-dataflow-java:assemble}}
> # Fix each ErrorProne warning from the {{runners/google-cloud-dataflow-java}} 
> project.
> # In {{runners/google-cloud-dataflow-java/build.gradle}}, add 
> {{failOnWarning: true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Fix errorprone warnings MutableConstantField

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 492138bfe176d63ce7700cb90a2daf24deab03c3
Merge: 774ff6a eb623ce
Author: Lukasz Cwik 
AuthorDate: Thu Jun 7 14:12:15 2018 -0700

Fix errorprone warnings MutableConstantField

 .../java/org/apache/beam/runners/core/construction/Environments.java   | 3 +--
 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java   | 1 +
 2 files changed, 2 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (774ff6a -> 492138b)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 774ff6a  [BEAM-3876] avoid NPE if checkpoint is null in an unbounded 
source
 add 74f1f58  Update Environments.java
 add 0e90776  Remove unneeded collection import.
 add c4e6560  Combine immutability type fixes. (#3)
 add eb623ce  fixup!
 new 492138b  Fix errorprone warnings MutableConstantField

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/runners/core/construction/Environments.java   | 3 +--
 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java   | 1 +
 2 files changed, 2 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #693

[Apache Jenkins Server]

See 


Changes:

[lcwik] BEAM-3876 avoid NPE if checkpoint is null in an unbounded source

[lcwik] testing npe fix and exception rethrow in unit tests

[lcwik] Update https://github.com/apache/beam/pull/4894 to correspond with

--
[...truncated 17.21 MB...]
INFO: Uploading <115998 bytes, hash KuUej2s6i5XLbn-j_hUrfQ> to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testreportfailures-jenkins-0607210520-88ff9068/output/results/staging/pipeline-KuUej2s6i5XLbn-j_hUrfQ.pb

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Dataflow SDK version: 2.6.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
Jun 07, 2018 9:05:25 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-06-07_14_05_24-7887441426796664432?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-06-07_14_05_24-7887441426796664432

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
Jun 07, 2018 9:05:25 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-06-07_14_05_24-7887441426796664432
Jun 07, 2018 9:05:25 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-06-07_14_05_24-7887441426796664432 with 0 
expected assertions.
Jun 07, 2018 9:05:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-07T21:05:24.869Z: Autoscaling is enabled for job 
2018-06-07_14_05_24-7887441426796664432. The number of workers will be between 
1 and 1000.
Jun 07, 2018 9:05:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-07T21:05:24.904Z: Autoscaling was automatically enabled for 
job 2018-06-07_14_05_24-7887441426796664432.
Jun 07, 2018 9:05:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-07T21:05:28.126Z: Checking required Cloud APIs are enabled.
Jun 07, 2018 9:05:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-07T21:05:28.305Z: Checking permissions granted to controller 
Service Account.
Jun 07, 2018 9:05:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-07T21:05:32.190Z: Worker configuration: n1-standard-1 in 
us-central1-b.
Jun 07, 2018 9:05:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-07T21:05:32.610Z: Expanding CoGroupByKey operations into 
optimizable parts.
Jun 07, 2018 9:05:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-07T21:05:32.820Z: Expanding GroupByKey operations into 
optimizable parts.
Jun 07, 2018 9:05:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-07T21:05:32.856Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jun 07, 2018 9:05:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-07T21:05:33.134Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jun 07, 2018 9:05:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-07T21:05:33.168Z: Elided trivial flatten 
Jun 07, 2018 9:05:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-07T21:05:33.219Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Jun 07, 2018 9:05:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-07T21:05:33.275Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Jun 07, 2018 9:05:36 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-06-07T21:05:33.324Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 

[jira] [Work logged] (BEAM-4313) Enforce ErrorProne analysis in Dataflow runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4313?focusedWorklogId=109858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109858
 ]

ASF GitHub Bot logged work on BEAM-4313:


Author: ASF GitHub Bot
Created on: 07/Jun/18 21:06
Start Date: 07/Jun/18 21:06
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #5540: 
[BEAM-4313] Fix and enforce FindBugs and ErrorProne violations in Dataflow 
Runner
URL: https://github.com/apache/beam/pull/5540#discussion_r193889100
 
 

 ##
 File path: sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
 ##
 @@ -139,6 +139,33 @@
 
   
 
+  
+
+
+
 
 Review comment:
   But aren't we trying also in the long term to get rid of findbugs? If this 
is the goal, better do this in the xml and we get of it and that's it, no?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109858)
Time Spent: 1h 50m  (was: 1h 40m)

> Enforce ErrorProne analysis in Dataflow runner project
> --
>
> Key: BEAM-4313
> URL: https://issues.apache.org/jira/browse/BEAM-4313
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-google-cloud-dataflow-java}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-google-cloud-dataflow-java:assemble}}
> # Fix each ErrorProne warning from the {{runners/google-cloud-dataflow-java}} 
> project.
> # In {{runners/google-cloud-dataflow-java/build.gradle}}, add 
> {{failOnWarning: true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4313) Enforce ErrorProne analysis in Dataflow runner project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4313?focusedWorklogId=109859=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109859
 ]

ASF GitHub Bot logged work on BEAM-4313:


Author: ASF GitHub Bot
Created on: 07/Jun/18 21:06
Start Date: 07/Jun/18 21:06
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #5540: [BEAM-4313] Fix and 
enforce FindBugs and ErrorProne violations in Dataflow Runner
URL: https://github.com/apache/beam/pull/5540#issuecomment-395565259
 
 
   @swegner sorry was quite busy these last days, since @kennknowles started I 
will let him finish the review. Please ping me in case you consider worth a 
second check.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109859)
Time Spent: 2h  (was: 1h 50m)

> Enforce ErrorProne analysis in Dataflow runner project
> --
>
> Key: BEAM-4313
> URL: https://issues.apache.org/jira/browse/BEAM-4313
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-google-cloud-dataflow-java}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-google-cloud-dataflow-java:assemble}}
> # Fix each ErrorProne warning from the {{runners/google-cloud-dataflow-java}} 
> project.
> # In {{runners/google-cloud-dataflow-java/build.gradle}}, add 
> {{failOnWarning: true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4448) --sdk_location seems to be a required parameter now in the Apache Beam Python SDK when using the DataflowRunner

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4448.
-
   Resolution: Fixed
Fix Version/s: 2.5.0

> --sdk_location seems to be a required parameter now in the Apache Beam Python 
> SDK when using the DataflowRunner
> ---
>
> Key: BEAM-4448
> URL: https://issues.apache.org/jira/browse/BEAM-4448
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-py-core
>Reporter: Luke Cwik
>Assignee: Ankur Goenka
>Priority: Major
> Fix For: 2.5.0
>
>
> During the code import process of Apache Beam code into Google, it was 
> discovered that the --sdk_location is a required parameter. Tests would fail 
> with:  
> {code:java}
> apache_beam/runners/portability/stager.py", line 513, in 
> _download_pypi_sdk_package.format(package_name))
> RuntimeError: Please set --sdk_location command-line option or install a 
> valid apache-beam distribution. 
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4448) --sdk_location seems to be a required parameter now in the Apache Beam Python SDK when using the DataflowRunner

[Ankur Goenka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505250#comment-16505250
 ] 

Ankur Goenka commented on BEAM-4448:


This was fixed internally.

[~lcwik] Do we need to keep this bug open?

> --sdk_location seems to be a required parameter now in the Apache Beam Python 
> SDK when using the DataflowRunner
> ---
>
> Key: BEAM-4448
> URL: https://issues.apache.org/jira/browse/BEAM-4448
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-py-core
>Reporter: Luke Cwik
>Assignee: Ankur Goenka
>Priority: Major
>
> During the code import process of Apache Beam code into Google, it was 
> discovered that the --sdk_location is a required parameter. Tests would fail 
> with:  
> {code:java}
> apache_beam/runners/portability/stager.py", line 513, in 
> _download_pypi_sdk_package.format(package_name))
> RuntimeError: Please set --sdk_location command-line option or install a 
> valid apache-beam distribution. 
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4488) Contributor's guide needs to document how to install Python SDK from source

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4488?focusedWorklogId=109855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109855
 ]

ASF GitHub Bot logged work on BEAM-4488:


Author: ASF GitHub Bot
Created on: 07/Jun/18 20:46
Start Date: 07/Jun/18 20:46
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request #464: 
[BEAM-4488], [BEAM-4489], [BEAM-4490] Improvements to contribution guide
URL: https://github.com/apache/beam-site/pull/464#discussion_r193883951
 
 

 ##
 File path: src/contribute/index.md
 ##
 @@ -78,29 +97,90 @@ necessary things to run those tests. For example:
 $ ./gradlew -p sdks/java/io/cassandra check
 $ ./gradlew -p runners/flink check
 
-### Testing the Python SDK
+You can see what build tasks are available with
+
+$ ./gradlew tasks
+
+or for a module,
+
+$ ./gradlew sdks/java/io/cassandra tasks
+
+### Developing with an IDE
+
+Generate an IDEA project .ipr file with:
+
+$ ./gradlew idea
+
+### Pull requests
+
+When your change is ready to be reviewed and merged, create a pull request.
+Format the pull request title like [BEAM-XXX] Fixes bug in 
ApproximateQuantiles,
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109855)
Time Spent: 1h 10m  (was: 1h)

> Contributor's guide needs to document how to install Python SDK from source
> ---
>
> Key: BEAM-4488
> URL: https://issues.apache.org/jira/browse/BEAM-4488
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Currently it is not documented, though it does appear in the quickstart



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4488) Contributor's guide needs to document how to install Python SDK from source

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4488?focusedWorklogId=109856=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109856
 ]

ASF GitHub Bot logged work on BEAM-4488:


Author: ASF GitHub Bot
Created on: 07/Jun/18 20:46
Start Date: 07/Jun/18 20:46
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request #464: 
[BEAM-4488], [BEAM-4489], [BEAM-4490] Improvements to contribution guide
URL: https://github.com/apache/beam-site/pull/464#discussion_r193883969
 
 

 ##
 File path: src/contribute/index.md
 ##
 @@ -78,29 +97,90 @@ necessary things to run those tests. For example:
 $ ./gradlew -p sdks/java/io/cassandra check
 $ ./gradlew -p runners/flink check
 
-### Testing the Python SDK
+You can see what build tasks are available with
+
+$ ./gradlew tasks
+
+or for a module,
+
+$ ./gradlew sdks/java/io/cassandra tasks
+
+### Developing with an IDE
+
+Generate an IDEA project .ipr file with:
+
+$ ./gradlew idea
+
+### Pull requests
+
+When your change is ready to be reviewed and merged, create a pull request.
+Format the pull request title like [BEAM-XXX] Fixes bug in 
ApproximateQuantiles,
+where you replace BEAM-XXX with the appropriate JIRA issue.
+This will automatically link the pull request to the issue.
+
+Pull requests can only be merged by a [beam committer]
+(http://people.apache.org/phonebook.html?unix=beam).
+To find a committer for your area, look for similar code merges or ask on 
+[d...@beam.apache.org]({{ site.baseurl }}/community/contact-us/)
+
+Use @mention in the pull request to notify the reviewer.
+
+The pull request and any changes pushed to it will trigger precommit jobs.
+If a test fails and appears unrelated to your change, you can cause tests
+to be re-run by adding a single line comment on your PR
+
+ retest this please
+
+There are other trigger phrases for post-commit tests found in
+.testinfra/jenkins, but use these sparingly because postcommit
+tests consume shared development resources.
+
+### Developing with the Python SDK
+
+Gradle can build and test python, and is used by the Jenkins jobs, so needs to
+be maintained.
 
 You can directly use the Python toolchain instead of having Gradle orchestrate
-it. This may be faster for you. We recommend setting up a virtual environment
-before testing your code.
+it, which may be faster for you, but it is your preference.
+If you do want to use Python tools directly, we recommend setting up a virtual
+environment before testing your code.
 
 If you update any of the [cythonized](http://cython.org) files in Python SDK,
 you must install the `cython` package before running following command to
 properly test your code. 
 
 The following commands should be run in the `sdks/python` directory.
-This command runs all Python tests.
+This installs Python from source and includes the test and gcp dependencies.
+
+On macOS/Linix:
+
+$ virtualenv env
+$ . ./env/bin/activate
+(env) $ pip install .[gcp,test]
 
-$ python setup.py nosetests
+On Windows:
+
+> c:\Python27\python.exe -m virtualenv
+> env\Scripts\activate
+(env) > pip install .[gcp,test]
+
+This command runs all Python tests. The nose dependency is installed by [test] 
in pip install.
+
+(env) $ python setup.py nosetests
 
 You can use following command to run a single test method.
 
-$ python setup.py nosetests --tests :.
+(env) $ python setup.py nosetests --tests :.
+
+For example:
+(env) $ python setup.py nosetests --tests 
apache_beam.io.textio_test:TextSourceTest.test_progress
 
-Example:
-$ python setup.py nosetests --tests 
apache_beam.io.textio_test:TextSourceTest.test_progress
+You can deactivate the virtualenv when done.
 
-To check just for lint errors, run the following command.
+(env) $ deactivate
+$
+
+To check just for Python lint errors, run the following command.
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109856)
Time Spent: 1h 20m  (was: 1h 10m)

> Contributor's guide needs to document how to install Python SDK from source
> ---
>
> Key: BEAM-4488
> URL: https://issues.apache.org/jira/browse/BEAM-4488
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining