[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146723=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146723
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 05:19
Start Date: 22/Sep/18 05:19
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #6431: [BEAM-4496] 
Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423717792
 
 
   Run Website Publish


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146723)
Time Spent: 6h 10m  (was: 6h)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146722=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146722
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 05:18
Start Date: 22/Sep/18 05:18
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #6431: [BEAM-4496] 
Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423717759
 
 
   Run Website Publish
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146722)
Time Spent: 6h  (was: 5h 50m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 6h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146721=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146721
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 05:14
Start Date: 22/Sep/18 05:14
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #6431: [BEAM-4496] 
Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423717619
 
 
   Run Website Publish


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146721)
Time Spent: 5h 50m  (was: 5h 40m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146720=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146720
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:57
Start Date: 22/Sep/18 04:57
Worklog Time Spent: 10m 
  Work Description: alanmyrvold removed a comment on issue #6431: 
[BEAM-4496] Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423716148
 
 
   Run Website Publish
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146720)
Time Spent: 5h 40m  (was: 5.5h)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146718=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146718
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:57
Start Date: 22/Sep/18 04:57
Worklog Time Spent: 10m 
  Work Description: alanmyrvold removed a comment on issue #6431: 
[BEAM-4496] Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423716298
 
 
   Run Website Publish
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146718)
Time Spent: 5h 20m  (was: 5h 10m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146719=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146719
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:57
Start Date: 22/Sep/18 04:57
Worklog Time Spent: 10m 
  Work Description: alanmyrvold removed a comment on issue #6431: 
[BEAM-4496] Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423716359
 
 
   Run Seed Job
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146719)
Time Spent: 5.5h  (was: 5h 20m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146716=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146716
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:48
Start Date: 22/Sep/18 04:48
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #6431: [BEAM-4496] 
Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423716563
 
 
   Run Website Publish
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146716)
Time Spent: 5h 10m  (was: 5h)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146712
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:43
Start Date: 22/Sep/18 04:43
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #6431: [BEAM-4496] 
Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423716359
 
 
   Run Seed Job
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146712)
Time Spent: 5h  (was: 4h 50m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146710=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146710
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:42
Start Date: 22/Sep/18 04:42
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #6431: [BEAM-4496] 
Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423716298
 
 
   Run Website Publish
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146710)
Time Spent: 4h 50m  (was: 4h 40m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146709=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146709
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:41
Start Date: 22/Sep/18 04:41
Worklog Time Spent: 10m 
  Work Description: alanmyrvold removed a comment on issue #6431: 
[BEAM-4496] Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-422557902
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146709)
Time Spent: 4h 40m  (was: 4.5h)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146708
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:41
Start Date: 22/Sep/18 04:41
Worklog Time Spent: 10m 
  Work Description: alanmyrvold removed a comment on issue #6431: 
[BEAM-4496] Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-422561008
 
 
   Run Website Merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146708)
Time Spent: 4.5h  (was: 4h 20m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146706=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146706
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:39
Start Date: 22/Sep/18 04:39
Worklog Time Spent: 10m 
  Work Description: alanmyrvold removed a comment on issue #6431: 
[BEAM-4496] Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423715911
 
 
   Run Website Publish


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146706)
Time Spent: 4h 10m  (was: 4h)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146707=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146707
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:39
Start Date: 22/Sep/18 04:39
Worklog Time Spent: 10m 
  Work Description: alanmyrvold removed a comment on issue #6431: 
[BEAM-4496] Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423715601
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146707)
Time Spent: 4h 20m  (was: 4h 10m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146705=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146705
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:38
Start Date: 22/Sep/18 04:38
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #6431: [BEAM-4496] 
Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423716148
 
 
   Run Website Publish
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146705)
Time Spent: 4h  (was: 3h 50m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 4h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146674
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:33
Start Date: 22/Sep/18 04:33
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #6431: [BEAM-4496] 
Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423715911
 
 
   Run Website Publish


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146674)
Time Spent: 3h 50m  (was: 3h 40m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146672=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146672
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:32
Start Date: 22/Sep/18 04:32
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6431: [BEAM-4496] Website merge
URL: https://github.com/apache/beam/pull/6431#discussion_r219661281
 
 

 ##
 File path: website/build.gradle
 ##
 @@ -104,3 +106,32 @@ task testWebsite(type: Exec) {
 task preCommit {
   dependsOn testWebsite
 }
+
+task mergeWebsite << {
+  /* Disabled for now, for testing
+  exec {
+executable 'git'
+args 'checkout', 'asf-site'
+  }
+  copy {
+from contentBuildDir
+into contentRepoDir
+  }
+  exec {
+executable 'git'
+args 'add', contentRepoDir
+  }
+  exec {
+executable 'git'
+args 'commit', '-m', 'Update website ' + new Date().format('/MM/dd 
HH:mm:ss')
+  }
+  exec {
 
 Review comment:
   Using org.ajoberstar.git-publish which defines a :gitPublishCopy task which 
creates all the files without commit or push.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146672)
Time Spent: 3.5h  (was: 3h 20m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146671=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146671
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:30
Start Date: 22/Sep/18 04:30
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6431: [BEAM-4496] Website merge
URL: https://github.com/apache/beam/pull/6431#discussion_r219661234
 
 

 ##
 File path: website/build.gradle
 ##
 @@ -104,3 +106,32 @@ task testWebsite(type: Exec) {
 task preCommit {
   dependsOn testWebsite
 }
+
+task mergeWebsite << {
+  /* Disabled for now, for testing
+  exec {
+executable 'git'
+args 'checkout', 'asf-site'
+  }
+  copy {
+from contentBuildDir
+into contentRepoDir
+  }
+  exec {
+executable 'git'
+args 'add', contentRepoDir
+  }
+  exec {
+executable 'git'
+args 'commit', '-m', 'Update website ' + new Date().format('/MM/dd 
HH:mm:ss')
 
 Review comment:
   Done. Added the commit hash to the commit message using org.ajoberstar.grgit 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146671)
Time Spent: 3h 20m  (was: 3h 10m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146670
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:29
Start Date: 22/Sep/18 04:29
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6431: [BEAM-4496] Website merge
URL: https://github.com/apache/beam/pull/6431#discussion_r219661218
 
 

 ##
 File path: website/build.gradle
 ##
 @@ -104,3 +106,32 @@ task testWebsite(type: Exec) {
 task preCommit {
   dependsOn testWebsite
 }
+
+task mergeWebsite << {
 
 Review comment:
   The org.ajoberstar.git-publish handles moving the files now, and creates a 
new git repository with the changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146670)
Time Spent: 3h 10m  (was: 3h)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146669
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:29
Start Date: 22/Sep/18 04:29
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6431: [BEAM-4496] Website merge
URL: https://github.com/apache/beam/pull/6431#discussion_r219661201
 
 

 ##
 File path: website/build.gradle
 ##
 @@ -22,6 +22,8 @@ apply plugin: "base"
 def dockerImageTag = 'beam-website'
 def dockerWorkDir = "/repo"
 def buildDir = "$project.rootDir/build/website"
+def contentBuildDir = "$buildDir/content"
+def contentRepoDir = "$project.rootDir/website/content"
 
 Review comment:
   org.ajoberstar.git-publish has directories/files to include/exclude. Nuked 
the website/generated-content directory except for pydocs and javadocs.
   
 preserve {
   include '**'
   exclude 'website/generated-content/**'
   include 'website/generated-content/documentation/sdks/pydoc/**'
   include 'website/generated-content/documentation/sdks/javadoc/**'
 } 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146669)
Time Spent: 3h  (was: 2h 50m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146668=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146668
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:28
Start Date: 22/Sep/18 04:28
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6431: [BEAM-4496] Website merge
URL: https://github.com/apache/beam/pull/6431#discussion_r219661177
 
 

 ##
 File path: website/build.gradle
 ##
 @@ -104,3 +106,32 @@ task testWebsite(type: Exec) {
 task preCommit {
   dependsOn testWebsite
 }
+
+task mergeWebsite << {
+  /* Disabled for now, for testing
+  exec {
+executable 'git'
 
 Review comment:
   Added the org.ajoberstar.git-publish and org.ajoberstar.grgit plugins


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146668)
Time Spent: 2h 50m  (was: 2h 40m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=146667=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146667
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:26
Start Date: 22/Sep/18 04:26
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request 
#6431: [BEAM-4496] Website merge
URL: https://github.com/apache/beam/pull/6431#discussion_r219661148
 
 

 ##
 File path: .test-infra/jenkins/job_PostCommit_Website_Merge.groovy
 ##
 @@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+
+// This job builds and merges the website.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Website_Merge', 'Run 
Website Merge',
+  'Website Merge', this) {
+
+  description('Builds and merges the web site.')
 
 Review comment:
   Changed to "Publish generated website content into asf-site branch for 
hosting."


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146667)
Time Spent: 2h 40m  (was: 2.5h)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4496?focusedWorklogId=14=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-14
 ]

ASF GitHub Bot logged work on BEAM-4496:


Author: ASF GitHub Bot
Created on: 22/Sep/18 04:25
Start Date: 22/Sep/18 04:25
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #6431: [BEAM-4496] 
Website merge
URL: https://github.com/apache/beam/pull/6431#issuecomment-423715601
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 14)
Time Spent: 2.5h  (was: 2h 20m)

> Create Jenkins job to push generated HTML to asf-site branch
> 
>
> Key: BEAM-4496
> URL: https://issues.apache.org/jira/browse/BEAM-4496
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Labels: beam-site-automation-reliability
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #103

[Apache Jenkins Server]

See 


Changes:

[chamikara] Clarifies in Javadocs that SpannerIO currently does not support

--
[...truncated 53.57 MB...]
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
7595649e042264f0c15da47b15544b64.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
608ee538e6d2bd88c5e72ad8874c3a4e.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (1/16) 
(7567245eee1d7f7a9cadf15a89fe80a4) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
523c19de0128b0175e3698b252d3b30f.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (3/16) 
(cc9500a1f202691e1f784f8f9eb22a2a) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (4/16) 
(eac969a1a657b96969619ff1bf31911e) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem 
(10/16) (5fee5aaa63209ef0f0493f2ac2c98577) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16) 
(60932dec1f51d1e6588aceb421185ef4) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (8/16) 
(32daa0b5e53797f3bcc6cea6aeea11af) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (13/16) 
(8da48a3aa3aca43ac6dca03babe0d39f) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem 
(15/16) (6f7bd1f9b9ba1501be87791c69889e81) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem 
(14/16) (73360f09ac8aa79eb40dc336dcdfbbad) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (6/16) 
(775e2f539eab46b5f9fc4fac4b122543) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (2/16) 
(69329cff70d33b83fec2316fe2ffc56a) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem 
(12/16) (c25df450034e5aba6e6ce7c2b8ae9eef) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem 
(13/16) (7595649e042264f0c15da47b15544b64) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (9/16) 
(608ee538e6d2bd88c5e72ad8874c3a4e) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem 
(11/16) (523c19de0128b0175e3698b252d3b30f) switched from RUNNING to FINISHED.
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(3d7b1816a79d70cc69703c4000762479) switched from RUNNING to FINISHED.
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(3d7b1816a79d70cc69703c4000762479).
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem 
streams are closed for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(3d7b1816a79d70cc69703c4000762479) [FINISHED]
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #1099

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5445) Update SpannerIO to support unbounded writes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5445?focusedWorklogId=146662=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146662
 ]

ASF GitHub Bot logged work on BEAM-5445:


Author: ASF GitHub Bot
Created on: 22/Sep/18 02:42
Start Date: 22/Sep/18 02:42
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #6450: [BEAM-5445] 
Clarifies that SpannerIO currently does not support unbounded writes.
URL: https://github.com/apache/beam/pull/6450#issuecomment-423711142
 
 
   Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146662)
Time Spent: 0.5h  (was: 20m)

> Update SpannerIO to support unbounded writes
> 
>
> Key: BEAM-5445
> URL: https://issues.apache.org/jira/browse/BEAM-5445
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Chamikara Jayalath
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Currently, due to a known issue, streaming pipelines that use SpannerIO.Write 
> do not actually write to Spanner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6450: [BEAM-5445] Clarifies that SpannerIO currently does not support unbounded writes.

[chamikara]

This is an automated email from the ASF dual-hosted git repository.

chamikara pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 9a1f917227dfbcd23cf5e6129c2312848ceffc27
Merge: 59c85b4 e1d7280
Author: Chamikara Jayalath 
AuthorDate: Fri Sep 21 19:42:40 2018 -0700

Merge pull request #6450: [BEAM-5445] Clarifies that SpannerIO currently 
does not support unbounded writes.

 .../src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java  | 5 +
 1 file changed, 5 insertions(+)

[jira] [Work logged] (BEAM-5445) Update SpannerIO to support unbounded writes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5445?focusedWorklogId=146663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146663
 ]

ASF GitHub Bot logged work on BEAM-5445:


Author: ASF GitHub Bot
Created on: 22/Sep/18 02:42
Start Date: 22/Sep/18 02:42
Worklog Time Spent: 10m 
  Work Description: chamikaramj closed pull request #6450: [BEAM-5445] 
Clarifies that SpannerIO currently does not support unbounded writes.
URL: https://github.com/apache/beam/pull/6450
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
index f3d25733fbd..af2a3b00d1f 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java
@@ -176,6 +176,11 @@
  * guaranteed that mutations in a group are submitted in the same transaction. 
Build {@link
  * SpannerIO.Write} transform, and call {@link Write#grouped()} method. It 
will return a
  * transformation that can be applied to a PCollection of MutationGroup.
+ *
+ * Streaming Support
+ *
+ * {@link SpannerIO.Write} currently does not support unbounded writes 
hence should not be used
+ * as a sink for streaming pipelines.
  */
 @Experimental(Experimental.Kind.SOURCE_SINK)
 public class SpannerIO {


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146663)
Time Spent: 40m  (was: 0.5h)

> Update SpannerIO to support unbounded writes
> 
>
> Key: BEAM-5445
> URL: https://issues.apache.org/jira/browse/BEAM-5445
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Chamikara Jayalath
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, due to a known issue, streaming pipelines that use SpannerIO.Write 
> do not actually write to Spanner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5460) Update Dataflow Python API client

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5460?focusedWorklogId=146660=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146660
 ]

ASF GitHub Bot logged work on BEAM-5460:


Author: ASF GitHub Bot
Created on: 22/Sep/18 02:28
Start Date: 22/Sep/18 02:28
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #6460: [BEAM-5460] 
Update Python Dataflow API client
URL: https://github.com/apache/beam/pull/6460#issuecomment-423710557
 
 
   @charlesccychen I'm looking into upgrading google-api-client (not 
google-apitools): https://issues.apache.org/jira/browse/BEAM-5456
   
   Not a blocker for current release. Sorry about the confusion.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146660)
Time Spent: 1h  (was: 50m)

> Update Dataflow Python API client
> -
>
> Key: BEAM-5460
> URL: https://issues.apache.org/jira/browse/BEAM-5460
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Affects Versions: 2.7.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> We should update the Dataflow Python API client to reflect the most current 
> Dataflow API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5445) Update SpannerIO to support unbounded writes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5445?focusedWorklogId=146658=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146658
 ]

ASF GitHub Bot logged work on BEAM-5445:


Author: ASF GitHub Bot
Created on: 22/Sep/18 02:16
Start Date: 22/Sep/18 02:16
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #6450: [BEAM-5445] 
Clarifies that SpannerIO currently does not support unbounded writes.
URL: https://github.com/apache/beam/pull/6450#issuecomment-423709925
 
 
   R: @udim 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146658)
Time Spent: 20m  (was: 10m)

> Update SpannerIO to support unbounded writes
> 
>
> Key: BEAM-5445
> URL: https://issues.apache.org/jira/browse/BEAM-5445
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Chamikara Jayalath
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently, due to a known issue, streaming pipelines that use SpannerIO.Write 
> do not actually write to Spanner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #1123

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4858) Clean up _BatchSizeEstimator in element-batching transform.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4858?focusedWorklogId=146647=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146647
 ]

ASF GitHub Bot logged work on BEAM-4858:


Author: ASF GitHub Bot
Created on: 22/Sep/18 00:50
Start Date: 22/Sep/18 00:50
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #6375: [BEAM-4858] Clean 
up division in batch size estimator.
URL: https://github.com/apache/beam/pull/6375#issuecomment-423704721
 
 
   Looks like the Precommit failure is specific to this PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146647)
Time Spent: 40m  (was: 0.5h)

> Clean up _BatchSizeEstimator in element-batching transform.
> ---
>
> Key: BEAM-4858
> URL: https://issues.apache.org/jira/browse/BEAM-4858
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Robert Bradshaw
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Beam Python 3 conversion [exposed|https://github.com/apache/beam/pull/5729] 
> non-trivial performance-sensitive logic in element-batching transform. Let's 
> take a look at 
> [util.py#L271|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271].
>  
> Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on 
> the type of the keys - whether they are integers or floats. 
> The keys of key-value pairs contained in {{self._data}} are added as integers 
> [here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L260],
>  however, when we 'thin' the collected entries 
> [here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L279],
>  the keys will become floats. Surprisingly, using either integer or float 
> division consistently [in the 
> comparator|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271]
>   negatively affects the performance of a custom pipeline I was using to 
> benchmark these changes. The performance impact likely comes from changes in 
> the logic that depends on  how division is evaluated, not from the 
> performance of division operation itself.
> In terms of Python 3 conversion the best course of action that avoids 
> regression seems to be to preserve the existing Python 2 behavior using 
> {{old_div}} from {{past.utils.division}}, in the medium term we should clean 
> up the logic. We may want to add a targeted microbenchmark to evaluate 
> performance of this code, and maybe cythonize the code, since it seems to be 
> performance-sensitive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4858) Clean up _BatchSizeEstimator in element-batching transform.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4858?focusedWorklogId=146648=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146648
 ]

ASF GitHub Bot logged work on BEAM-4858:


Author: ASF GitHub Bot
Created on: 22/Sep/18 00:50
Start Date: 22/Sep/18 00:50
Worklog Time Spent: 10m 
  Work Description: tvalentyn edited a comment on issue #6375: [BEAM-4858] 
Clean up division in batch size estimator.
URL: https://github.com/apache/beam/pull/6375#issuecomment-423704721
 
 
   Looks like the Precommit failure is specific to this PR now that tests on 
the master branch are passing again.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146648)
Time Spent: 50m  (was: 40m)

> Clean up _BatchSizeEstimator in element-batching transform.
> ---
>
> Key: BEAM-4858
> URL: https://issues.apache.org/jira/browse/BEAM-4858
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Robert Bradshaw
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Beam Python 3 conversion [exposed|https://github.com/apache/beam/pull/5729] 
> non-trivial performance-sensitive logic in element-batching transform. Let's 
> take a look at 
> [util.py#L271|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271].
>  
> Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on 
> the type of the keys - whether they are integers or floats. 
> The keys of key-value pairs contained in {{self._data}} are added as integers 
> [here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L260],
>  however, when we 'thin' the collected entries 
> [here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L279],
>  the keys will become floats. Surprisingly, using either integer or float 
> division consistently [in the 
> comparator|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271]
>   negatively affects the performance of a custom pipeline I was using to 
> benchmark these changes. The performance impact likely comes from changes in 
> the logic that depends on  how division is evaluated, not from the 
> performance of division operation itself.
> In terms of Python 3 conversion the best course of action that avoids 
> regression seems to be to preserve the existing Python 2 behavior using 
> {{old_div}} from {{past.utils.division}}, in the medium term we should clean 
> up the logic. We may want to add a targeted microbenchmark to evaluate 
> performance of this code, and maybe cythonize the code, since it seems to be 
> performance-sensitive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #1098

[Apache Jenkins Server]

See 


Changes:

[scott] Upgrade gradle wrapper to latest (4.10)

[mwylde] [BEAM-5455] Log per-bundle messages at debug in python sdk

[mxm] [BEAM-5442] Add basic FlinkOptions to Python SDK

[robbe.sneyders] Fix pipeline_test

[ankurgoenka] Add Portable PostCommit test status to PR template

[robbe.sneyders] Fix hamcrest docs dependency

--
[...truncated 19.11 MB...]
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Create123/Read(CreateSource) as step s10
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding OutputSideInputs as step s11
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Window.Into()/Window.Assign as step 
s12
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
Sep 22, 2018 12:33:06 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
Sep 22, 2018 12:33:06 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0922003300-ca8d9d5f/output/results/staging/
Sep 22, 2018 12:33:06 AM 

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #1122

[Apache Jenkins Server]

See 


Changes:

[github] Revert "Adding Autocomplete IT for Java"

--
[...truncated 76.40 KB...]
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
:54:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  print('Found: %s.' % self.build_console_url(pipeline.options))
:54:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  print('Found: %s.' % self.build_console_url(pipeline.options))
:54:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  print('Found: 

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #102

[Apache Jenkins Server]

See 


--
[...truncated 53.57 MB...]
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(d734bfbc45404b93598a12213ef9c8a0) switched from RUNNING to FINISHED.
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(d734bfbc45404b93598a12213ef9c8a0).
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem 
streams are closed for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(d734bfbc45404b93598a12213ef9c8a0) [FINISHED]
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
d734bfbc45404b93598a12213ef9c8a0.
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (7/16) (f2f008e0cf7515af495415fe2d7c12ad) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (7/16) 
(f2f008e0cf7515af495415fe2d7c12ad).
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (7/16) 
(f2f008e0cf7515af495415fe2d7c12ad) [FINISHED]
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (4/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(4/16) (80897122392c9be7929a06e10525db09) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (10/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(10/16) (5d4fc1af74788aeff6eef752b01bf0c0) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (6/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(6/16) (ce904267215d0f302721529b0c81060d) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(7/16) (946b98527eb86ed064825abe3111f8d3) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(946b98527eb86ed064825abe3111f8d3).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (6/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (6/16) 
(ce904267215d0f302721529b0c81060d).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (10/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (10/16) 
(5d4fc1af74788aeff6eef752b01bf0c0).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (4/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (4/16) 
(80897122392c9be7929a06e10525db09).
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
f2f008e0cf7515af495415fe2d7c12ad.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 

Jenkins build is back to normal : beam_PreCommit_Python_Cron #374

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5427) Fix sample code (AverageFn) in Combine.java

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5427?focusedWorklogId=146641=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146641
 ]

ASF GitHub Bot logged work on BEAM-5427:


Author: ASF GitHub Bot
Created on: 22/Sep/18 00:13
Start Date: 22/Sep/18 00:13
Worklog Time Spent: 10m 
  Work Description: HuangLED commented on issue #6439: [BEAM-5427] Fix and 
update sample code for CombineFn.
URL: https://github.com/apache/beam/pull/6439#issuecomment-423701320
 
 
   @lukecwik  to check if the updated comment is accurate.
   
   Thanks!  


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146641)
Time Spent: 2h 10m  (was: 2h)

> Fix sample code (AverageFn) in Combine.java
> ---
>
> Key: BEAM-5427
> URL: https://issues.apache.org/jira/browse/BEAM-5427
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Minor
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Sample code missing coder. 
> In its current state, job run fails with Coder missing error. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #101

[Apache Jenkins Server]

See 


Changes:

[github] Revert "Adding Autocomplete IT for Java"

--
[...truncated 53.56 MB...]
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem 
(16/16) (0cb61aecc874e2ced0249f177b6b1bff) switched from RUNNING to FINISHED.
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(d1d0828d1790e95f0989e32b1b7c61ed) switched from RUNNING to FINISHED.
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(d1d0828d1790e95f0989e32b1b7c61ed).
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem 
streams are closed for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(d1d0828d1790e95f0989e32b1b7c61ed) [FINISHED]
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
d1d0828d1790e95f0989e32b1b7c61ed.
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (7/16) (ad74cf8a6013faa1dad0a33928ba1bd8) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (7/16) 
(ad74cf8a6013faa1dad0a33928ba1bd8).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (3/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(3/16) (6db712e80f66497383bb993e28071a39) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (3/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (3/16) 
(6db712e80f66497383bb993e28071a39).
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(d1d0828d1790e95f0989e32b1b7c61ed) switched from RUNNING to FINISHED.
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (7/16) 
(ad74cf8a6013faa1dad0a33928ba1bd8) [FINISHED]
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (9/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(9/16) (4f61fdbb7d610d67c6da39d37867f5ab) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (9/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (9/16) 
(4f61fdbb7d610d67c6da39d37867f5ab).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (12/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(12/16) (8110789ceb485efc7edb153c0789b62a) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
ad74cf8a6013faa1dad0a33928ba1bd8.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(11/16) (9aedcd1c3669f926caafdac3b68bd324) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16) 
(9aedcd1c3669f926caafdac3b68bd324).

[beam] 01/01: Merge pull request #6468 from apache/revert-6372-autocomplete-it

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 59c85b44d156bb7b4462d80fcb5591f860235708
Merge: ef7a79c e3ddecd
Author: Charles Chen 
AuthorDate: Fri Sep 21 16:58:22 2018 -0700

Merge pull request #6468 from apache/revert-6372-autocomplete-it

Revert "PR #6372 Adding Autocomplete IT for Java"

 .../beam/examples/complete/AutoComplete.java   | 41 +--
 .../beam/examples/complete/AutoCompleteIT.java | 59 --
 2 files changed, 2 insertions(+), 98 deletions(-)

[beam] branch master updated (ef7a79c -> 59c85b4)

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from ef7a79c  [BEAM-3711] Bug fix and improvements in Dataflow transform 
override.
 add e3ddecd  Revert "Adding Autocomplete IT for Java"
 new 59c85b4  Merge pull request #6468 from 
apache/revert-6372-autocomplete-it

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/examples/complete/AutoComplete.java   | 41 +--
 .../beam/examples/complete/AutoCompleteIT.java | 59 --
 2 files changed, 2 insertions(+), 98 deletions(-)
 delete mode 100644 
examples/java/src/test/java/org/apache/beam/examples/complete/AutoCompleteIT.java

[jira] [Work logged] (BEAM-5427) Fix sample code (AverageFn) in Combine.java

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5427?focusedWorklogId=146639=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146639
 ]

ASF GitHub Bot logged work on BEAM-5427:


Author: ASF GitHub Bot
Created on: 21/Sep/18 23:34
Start Date: 21/Sep/18 23:34
Worklog Time Spent: 10m 
  Work Description: HuangLED commented on a change in pull request #6439: 
[BEAM-5427] Fix and update sample code for CombineFn.
URL: https://github.com/apache/beam/pull/6439#discussion_r219650411
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java
 ##
 @@ -289,6 +305,21 @@ private Combine() {
* arbitrary tree structure. Commutativity is required because any order of 
the input values is
* ignored when breaking up input values into groups.
*
+   * Note on Data Encoding
+   *
+   * Data encoding is required to make CombineFn work. The sample code 
above implements {@code
+   * java.io.Serializable} so that CombineFn in consequence makes use of 
SerializableCoder to encode
+   * the data. In many cases though, relying on Serializable could be less 
preferred for efficiency
+   * considerations. In addition, Serializable may not be applicable in all 
situations. There are
+   * two alternatives to make data encoding work for AverageFn:
+   *
+   * 
+   *   Generic Coder class. Implement an encoder class explicitly and use 
{@code @DefaultCoder}
+   *   tag.
+   *   {@link CombineFn} Specific way. While extending {@link CombineFn}, 
overwrite both {@link
+   *   #getAccumulatorCoder} and {@link #getDefaultOutputCoder}.
+   * 
+   *
 
 Review comment:
   oops.  I was trying to consolidate commits for cleaner history. But looks 
like history is now gone. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146639)
Time Spent: 2h  (was: 1h 50m)

> Fix sample code (AverageFn) in Combine.java
> ---
>
> Key: BEAM-5427
> URL: https://issues.apache.org/jira/browse/BEAM-5427
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Minor
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Sample code missing coder. 
> In its current state, job run fails with Coder missing error. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5427) Fix sample code (AverageFn) in Combine.java

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5427?focusedWorklogId=146637=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146637
 ]

ASF GitHub Bot logged work on BEAM-5427:


Author: ASF GitHub Bot
Created on: 21/Sep/18 23:25
Start Date: 21/Sep/18 23:25
Worklog Time Spent: 10m 
  Work Description: HuangLED commented on a change in pull request #6439: 
[BEAM-5427] Fix and update sample code for CombineFn.
URL: https://github.com/apache/beam/pull/6439#discussion_r219648742
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java
 ##
 @@ -289,6 +305,21 @@ private Combine() {
* arbitrary tree structure. Commutativity is required because any order of 
the input values is
* ignored when breaking up input values into groups.
*
+   * Note on Data Encoding
+   *
+   * Data encoding is required to make CombineFn work. The sample code 
above implements {@code
+   * java.io.Serializable} so that CombineFn in consequence makes use of 
SerializableCoder to encode
+   * the data. In many cases though, relying on Serializable could be less 
preferred for efficiency
+   * considerations. In addition, Serializable may not be applicable in all 
situations. There are
+   * two alternatives to make data encoding work for AverageFn:
+   *
+   * 
+   *   Generic Coder class. Implement an encoder class explicitly and use 
{@code @DefaultCoder}
+   *   tag.
+   *   {@link CombineFn} Specific way. While extending {@link CombineFn}, 
overwrite both {@link
+   *   #getAccumulatorCoder} and {@link #getDefaultOutputCoder}.
+   * 
+   *
 
 Review comment:
   Thanks for the proofreading.  Updated. 
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146637)
Time Spent: 1h 50m  (was: 1h 40m)

> Fix sample code (AverageFn) in Combine.java
> ---
>
> Key: BEAM-5427
> URL: https://issues.apache.org/jira/browse/BEAM-5427
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Minor
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Sample code missing coder. 
> In its current state, job run fails with Coder missing error. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5427) Fix sample code (AverageFn) in Combine.java

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5427?focusedWorklogId=146636=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146636
 ]

ASF GitHub Bot logged work on BEAM-5427:


Author: ASF GitHub Bot
Created on: 21/Sep/18 23:21
Start Date: 21/Sep/18 23:21
Worklog Time Spent: 10m 
  Work Description: HuangLED commented on a change in pull request #6439: 
[BEAM-5427] Fix and update sample code for CombineFn.
URL: https://github.com/apache/beam/pull/6439#discussion_r219649060
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java
 ##
 @@ -254,10 +254,26 @@ private Combine() {
*
* {@code
* public class AverageFn extends CombineFn {
-   *   public static class Accum {
+   *   public static class Accum implements java.io.Serializable {
* int sum = 0;
* int count = 0;
+   *
+   * {@literal @}Override
 
 Review comment:
   removed the whitespace.  btw, ctrl+Q does not render this literal tag 
correctly. 
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146636)
Time Spent: 1h 40m  (was: 1.5h)

> Fix sample code (AverageFn) in Combine.java
> ---
>
> Key: BEAM-5427
> URL: https://issues.apache.org/jira/browse/BEAM-5427
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-java
>Reporter: Ruoyun Huang
>Assignee: Ruoyun Huang
>Priority: Minor
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Sample code missing coder. 
> In its current state, job run fails with Coder missing error. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5465) Have the sdks/go gradle tasks clean up the vendor directories on clean

[Robert Burke (JIRA)]

Robert Burke created BEAM-5465:
--

 Summary: Have the sdks/go gradle tasks clean up the vendor 
directories on clean
 Key: BEAM-5465
 URL: https://issues.apache.org/jira/browse/BEAM-5465
 Project: Beam
  Issue Type: Bug
  Components: build-system, sdk-go
Reporter: Robert Burke


The sdks/go/\{test,examples,container} directories depend on the main beam 
package, which cause gogradle to vendor the beam package in those directories.
 
The gogradle plugin doesn't clean up the [vendor 
directories|https://golang.org/cmd/go/#hdr-Vendor_Directories] that it sets up 
on builds, when clean is invoked.
 
This leads to stale copies of the Go SDK's beam package vendored in local 
directories, which can lead to build failures of other tasks that invoke the 
tests or similar, when the code in those directories uses a more recent version 
of beam than what is cached.
 
This doesn't happen for users directly using the go tool, with their git repo 
nested under Go PATH, since the go tool will correctly use the local repo copy 
of beam.
 
A workaround on a Unix machine or similar, invoked from the beam repo root, is 
to delete the vendor and gogradle directories, and retry the task.
 
rm -rf sdks/go/\{vendor,.gogradle} 
sdks/go/\{test,examples,container}/\{vendor,.gogradle}
 
This cause gogradle to fetch a more recent copy of beam for vendoring.
 
Ideally we fix the clean tasks for the go directories to delete the vendor 
directories as well, which will resolve the issue more reliably for those using 
gradle to test their changes against the go sdk.

Related: BEAM-5379 is for avoiding the vendoring & cleaning cycle all together 
and migrating to gomodules.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #6050

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #100

[Apache Jenkins Server]

See 


Changes:

[daniel.o.programmer] [BEAM-3711] Bug fix and improvements in Dataflow 
transform override.

[daniel.o.programmer] [BEAM-3711] Refactoring and testing new matchers.

--
[...truncated 53.53 MB...]
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(4016a8bcb5c4776e43f0fc188302a548) switched from RUNNING to FINISHED.
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(4016a8bcb5c4776e43f0fc188302a548).
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem 
streams are closed for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(4016a8bcb5c4776e43f0fc188302a548) [FINISHED]
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (7/16) (56d80e6be0e27f19aa8e990cc2f1461f) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (7/16) 
(56d80e6be0e27f19aa8e990cc2f1461f).
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
4016a8bcb5c4776e43f0fc188302a548.
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (7/16) 
(56d80e6be0e27f19aa8e990cc2f1461f) [FINISHED]
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (4/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(4/16) (873f59134c894a98b533de55afce23fd) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (4/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (4/16) 
(873f59134c894a98b533de55afce23fd).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (1/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(1/16) (253280b639ef5867373d6f95f65a3326) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (1/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (1/16) 
(253280b639ef5867373d6f95f65a3326).
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
56d80e6be0e27f19aa8e990cc2f1461f.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(4016a8bcb5c4776e43f0fc188302a548) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (7/16) 
(56d80e6be0e27f19aa8e990cc2f1461f) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (6/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(6/16) (28d3d8d2c451f51230981f0e8f539700) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (6/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (6/16) 
(28d3d8d2c451f51230981f0e8f539700).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (8/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 

[jira] [Work logged] (BEAM-3711) Implement portable Combiner lifting in Dataflow Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3711?focusedWorklogId=146627=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146627
 ]

ASF GitHub Bot logged work on BEAM-3711:


Author: ASF GitHub Bot
Created on: 21/Sep/18 22:46
Start Date: 21/Sep/18 22:46
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #6384: [BEAM-3711] Bug fix 
and improvements in Dataflow transform override.
URL: https://github.com/apache/beam/pull/6384
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPTransformMatchers.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPTransformMatchers.java
new file mode 100644
index 000..72534c117e0
--- /dev/null
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPTransformMatchers.java
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow;
+
+import static com.google.common.base.MoreObjects.toStringHelper;
+
+import java.util.ArrayDeque;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.runners.PTransformMatcher;
+import org.apache.beam.sdk.runners.TransformHierarchy;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.PTransform;
+
+/**
+ * A set of {@link PTransformMatcher PTransformMatchers} that are used in the 
Dataflow Runner and
+ * not general enough to be shared between runners.
+ */
+class DataflowPTransformMatchers {
+  private DataflowPTransformMatchers() {}
+
+  /**
+   * Matches {@link PTransform}s of class {@link Combine.GroupedValues} that 
have no side inputs.
+   */
+  static class CombineValuesWithoutSideInputsPTransformMatcher implements 
PTransformMatcher {
+
+@Override
+public boolean matches(AppliedPTransform application) {
+  return 
application.getTransform().getClass().equals(Combine.GroupedValues.class)
+  && ((Combine.GroupedValues) application.getTransform())
+  .getSideInputs()
+  .isEmpty();
+}
+
+@Override
+public String toString() {
+  return 
toStringHelper(CombineValuesWithoutSideInputsPTransformMatcher.class).toString();
+}
+  }
+
+  /**
+   * Matches {@link PTransform}s of class {@link Combine.GroupedValues} that 
have no side inputs and
+   * are direct subtransforms of a {@link Combine.PerKey}.
+   */
+  static class CombineValuesWithParentCheckPTransformMatcher implements 
PTransformMatcher {
+
+@Override
+public boolean matches(AppliedPTransform application) {
+  return 
application.getTransform().getClass().equals(Combine.GroupedValues.class)
+  && ((Combine.GroupedValues) 
application.getTransform()).getSideInputs().isEmpty()
+  && parentIsCombinePerKey(application);
+}
+
+private boolean parentIsCombinePerKey(AppliedPTransform 
application) {
+  // We want the PipelineVisitor below to change the parent, but the 
parent must be final to
+  // be captured in there. To work around this issue, wrap the parent in a 
one element array.
+  final TransformHierarchy.Node[] parent = new TransformHierarchy.Node[1];
+
+  // Traverse the pipeline to find the parent transform to application. Do 
this by maintaining
+  // a stack of each composite transform being entered, and grabbing the 
top transform of the
+  // stack once the target node is visited.
+  Pipeline pipeline = application.getPipeline();
+  pipeline.traverseTopologically(
+  new Pipeline.PipelineVisitor.Defaults() {
+private ArrayDeque parents = new 
ArrayDeque<>();
+
+@Override
+public CompositeBehavior 

[beam] 01/01: [BEAM-3711] Bug fix and improvements in Dataflow transform override.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit ef7a79c42927758171fd180c7042eb5d39f07218
Merge: b276db3 18b6a04
Author: Lukasz Cwik 
AuthorDate: Fri Sep 21 15:46:28 2018 -0700

[BEAM-3711] Bug fix and improvements in Dataflow transform override.

 .../dataflow/DataflowPTransformMatchers.java   | 134 +++
 .../dataflow/DataflowPipelineTranslator.java   |   8 +-
 .../beam/runners/dataflow/DataflowRunner.java  |  56 ++---
 .../dataflow/DataflowPTransformMatchersTest.java   | 250 +
 4 files changed, 413 insertions(+), 35 deletions(-)

[beam] branch master updated (b276db3 -> ef7a79c)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b276db3  Merge pull request #6459: [BEAM-5455] Log per-bundle messages 
at debug in python sdk
 add 55bea52  [BEAM-3711] Bug fix and improvements in Dataflow transform 
override.
 add 18b6a04  [BEAM-3711] Refactoring and testing new matchers.
 new ef7a79c  [BEAM-3711] Bug fix and improvements in Dataflow transform 
override.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../dataflow/DataflowPTransformMatchers.java   | 134 +++
 .../dataflow/DataflowPipelineTranslator.java   |   8 +-
 .../beam/runners/dataflow/DataflowRunner.java  |  56 ++---
 .../dataflow/DataflowPTransformMatchersTest.java   | 250 +
 4 files changed, 413 insertions(+), 35 deletions(-)
 create mode 100644 
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPTransformMatchers.java
 create mode 100644 
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPTransformMatchersTest.java

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=146622=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146622
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 21/Sep/18 22:44
Start Date: 21/Sep/18 22:44
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on a change in pull request #6382: 
[BEAM-4449] Finish splitting Aggregate and Project
URL: https://github.com/apache/beam/pull/6382#discussion_r219644569
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java
 ##
 @@ -504,21 +501,6 @@ private static BeamSqlExpression 
getBeamSqlExpression(RexNode rexNode) {
   ret = new BeamSqlIsNotNullExpression(subExps.get(0));
   break;
 
-case "HOP":
-case "TUMBLE":
-case "SESSION":
-  ret = new BeamSqlWindowExpression(subExps, 
node.type.getSqlTypeName());
 
 Review comment:
   It seems that our implementation of window start/end has never worked 
because Calcite convert window start to a column reference and window end to a 
PLUS operation.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146622)
Time Spent: 8h 20m  (was: 8h 10m)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4858) Clean up _BatchSizeEstimator in element-batching transform.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4858?focusedWorklogId=146620=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146620
 ]

ASF GitHub Bot logged work on BEAM-4858:


Author: ASF GitHub Bot
Created on: 21/Sep/18 22:35
Start Date: 21/Sep/18 22:35
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #6375: [BEAM-4858] Clean 
up division in batch size estimator.
URL: https://github.com/apache/beam/pull/6375#issuecomment-423688655
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146620)
Time Spent: 0.5h  (was: 20m)

> Clean up _BatchSizeEstimator in element-batching transform.
> ---
>
> Key: BEAM-4858
> URL: https://issues.apache.org/jira/browse/BEAM-4858
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Robert Bradshaw
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Beam Python 3 conversion [exposed|https://github.com/apache/beam/pull/5729] 
> non-trivial performance-sensitive logic in element-batching transform. Let's 
> take a look at 
> [util.py#L271|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271].
>  
> Due to Python 2 language semantics, the result of {{x2 / x1}} will depend on 
> the type of the keys - whether they are integers or floats. 
> The keys of key-value pairs contained in {{self._data}} are added as integers 
> [here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L260],
>  however, when we 'thin' the collected entries 
> [here|https://github.com/apache/beam/blob/d2ac08da2dccce8930432fae1ec7c30953880b69/sdks/python/apache_beam/transforms/util.py#L279],
>  the keys will become floats. Surprisingly, using either integer or float 
> division consistently [in the 
> comparator|https://github.com/apache/beam/blob/e98ff7c96afa2f72b3a98426dc1e9a47224da5c8/sdks/python/apache_beam/transforms/util.py#L271]
>   negatively affects the performance of a custom pipeline I was using to 
> benchmark these changes. The performance impact likely comes from changes in 
> the logic that depends on  how division is evaluated, not from the 
> performance of division operation itself.
> In terms of Python 3 conversion the best course of action that avoids 
> regression seems to be to preserve the existing Python 2 behavior using 
> {{old_div}} from {{past.utils.division}}, in the medium term we should clean 
> up the logic. We may want to add a targeted microbenchmark to evaluate 
> performance of this code, and maybe cythonize the code, since it seems to be 
> performance-sensitive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5464) Portable beam hangs while running TFX preprocessing step on a distributed cluster

[Axel Magnuson (JIRA)]

Axel Magnuson created BEAM-5464:
---

 Summary: Portable beam hangs while running TFX preprocessing step 
on a distributed cluster
 Key: BEAM-5464
 URL: https://issues.apache.org/jira/browse/BEAM-5464
 Project: Beam
  Issue Type: Bug
  Components: java-fn-execution
Reporter: Axel Magnuson
Assignee: Ankur Goenka


Recently I went through the exercise of running the TFX taxi example on a 
dataproc cluster.  However it would always hang indefinitely.  The flink UI 
indicated that the job was halfway done.  However I could not see any clear 
errors in the job driver logs, the job service logs, or the Flink logs.  The 
root cause is still a mystery to me.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5463) Artifact staging service cannot write to HDFS

[Axel Magnuson (JIRA)]

Axel Magnuson created BEAM-5463:
---

 Summary: Artifact staging service cannot write to HDFS
 Key: BEAM-5463
 URL: https://issues.apache.org/jira/browse/BEAM-5463
 Project: Beam
  Issue Type: Bug
  Components: java-fn-execution
Reporter: Axel Magnuson
Assignee: Ankur Goenka


With `--artifacts-dir=hdfs://tmp/beam-artifacts` set on the job service, Python 
wordcount fails.  The artifact staging service produces the following error:

 

java.lang.IllegalArgumentException: No filesystem found for scheme hdfs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5417) FileSystems.match behaviour diff between GCS and local file system

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5417?focusedWorklogId=146615=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146615
 ]

ASF GitHub Bot logged work on BEAM-5417:


Author: ASF GitHub Bot
Created on: 21/Sep/18 22:09
Start Date: 21/Sep/18 22:09
Worklog Time Spent: 10m 
  Work Description: joar edited a comment on issue #6423: [BEAM-5417] 
Parity between GCS and local match
URL: https://github.com/apache/beam/pull/6423#issuecomment-423683905
 
 
   Unless of course we use `pathlib.Path().glob()` rather than 
`PurePath().match()`, and in LocalFileSystem only rather than FileSystem!
   
   I've had enough for today though, I'll have to check back later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146615)
Time Spent: 2h 10m  (was: 2h)

> FileSystems.match behaviour diff between GCS and local file system
> --
>
> Key: BEAM-5417
> URL: https://issues.apache.org/jira/browse/BEAM-5417
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Joar Wandborg
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Given the directory structure:
>  
> {noformat}
> .
> ├── filesystem-match-test
> │   ├── a
> │   │   └── file.txt
> │   └── b
> │   └── file.txt
> └── filesystem-match-test.py
> {noformat}
>  
> Where {{filesystem-match-test.py}} contains:
> {code:python}
> from __future__ import print_function
> import os
> import posixpath
> from apache_beam.io.filesystem import MatchResult
> from apache_beam.io.filesystems import FileSystems
> BASES = [
> os.path.join(os.path.dirname(__file__), "./"),
> "gs://my-bucket/test/",
> ]
> pattern = "filesystem-match-test/*/file.txt"
> for base_path in BASES:
> full_pattern = posixpath.join(base_path, pattern)
> print("full_pattern: {}".format(full_pattern))
> match_result = FileSystems.match([full_pattern])[0]  # type: MatchResult
> print("metadata list: {}".format(match_result.metadata_list))
> {code}
> Running {{python filesystem-match-test.py}} does not match any files locally, 
> but does match files on GCS:
> {noformat}
> full_pattern: ./filesystem-match-test/*/file.txt
> metadata list: []
> full_pattern: gs://my-bucket/test/filesystem-match-test/*/file.txt
> metadata list: 
> [FileMetadata(gs://my-bucket/test/filesystem-match-test/a/file.txt, 6), 
> FileMetadata(gs://my-bucket/test/filesystem-match-test/b/file.txt, 6)]
> {noformat}
> The expected result is that a/file.txt and b/file.txt should be matched for 
> both patterns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5417) FileSystems.match behaviour diff between GCS and local file system

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5417?focusedWorklogId=146613=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146613
 ]

ASF GitHub Bot logged work on BEAM-5417:


Author: ASF GitHub Bot
Created on: 21/Sep/18 22:08
Start Date: 21/Sep/18 22:08
Worklog Time Spent: 10m 
  Work Description: joar edited a comment on issue #6423: [BEAM-5417] 
Parity between GCS and local match
URL: https://github.com/apache/beam/pull/6423#issuecomment-423683905
 
 
   Unless of course we use `pathlib.Path().glob()` rather than 
`PurePath().match()`, and in LocalFileSystem only rather than FileSystem!
   
   I'm done for today though, I'll have to check back later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146613)
Time Spent: 1h 50m  (was: 1h 40m)

> FileSystems.match behaviour diff between GCS and local file system
> --
>
> Key: BEAM-5417
> URL: https://issues.apache.org/jira/browse/BEAM-5417
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Joar Wandborg
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Given the directory structure:
>  
> {noformat}
> .
> ├── filesystem-match-test
> │   ├── a
> │   │   └── file.txt
> │   └── b
> │   └── file.txt
> └── filesystem-match-test.py
> {noformat}
>  
> Where {{filesystem-match-test.py}} contains:
> {code:python}
> from __future__ import print_function
> import os
> import posixpath
> from apache_beam.io.filesystem import MatchResult
> from apache_beam.io.filesystems import FileSystems
> BASES = [
> os.path.join(os.path.dirname(__file__), "./"),
> "gs://my-bucket/test/",
> ]
> pattern = "filesystem-match-test/*/file.txt"
> for base_path in BASES:
> full_pattern = posixpath.join(base_path, pattern)
> print("full_pattern: {}".format(full_pattern))
> match_result = FileSystems.match([full_pattern])[0]  # type: MatchResult
> print("metadata list: {}".format(match_result.metadata_list))
> {code}
> Running {{python filesystem-match-test.py}} does not match any files locally, 
> but does match files on GCS:
> {noformat}
> full_pattern: ./filesystem-match-test/*/file.txt
> metadata list: []
> full_pattern: gs://my-bucket/test/filesystem-match-test/*/file.txt
> metadata list: 
> [FileMetadata(gs://my-bucket/test/filesystem-match-test/a/file.txt, 6), 
> FileMetadata(gs://my-bucket/test/filesystem-match-test/b/file.txt, 6)]
> {noformat}
> The expected result is that a/file.txt and b/file.txt should be matched for 
> both patterns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5417) FileSystems.match behaviour diff between GCS and local file system

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5417?focusedWorklogId=146614=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146614
 ]

ASF GitHub Bot logged work on BEAM-5417:


Author: ASF GitHub Bot
Created on: 21/Sep/18 22:08
Start Date: 21/Sep/18 22:08
Worklog Time Spent: 10m 
  Work Description: joar edited a comment on issue #6423: [BEAM-5417] 
Parity between GCS and local match
URL: https://github.com/apache/beam/pull/6423#issuecomment-423683905
 
 
   Unless of course we use `pathlib.Path().glob()` rather than 
`PurePath().match()`, and in LocalFileSystem only rather than FileSystem!
   
   I'm had enough for today though, I'll have to check back later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146614)
Time Spent: 2h  (was: 1h 50m)

> FileSystems.match behaviour diff between GCS and local file system
> --
>
> Key: BEAM-5417
> URL: https://issues.apache.org/jira/browse/BEAM-5417
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Joar Wandborg
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Given the directory structure:
>  
> {noformat}
> .
> ├── filesystem-match-test
> │   ├── a
> │   │   └── file.txt
> │   └── b
> │   └── file.txt
> └── filesystem-match-test.py
> {noformat}
>  
> Where {{filesystem-match-test.py}} contains:
> {code:python}
> from __future__ import print_function
> import os
> import posixpath
> from apache_beam.io.filesystem import MatchResult
> from apache_beam.io.filesystems import FileSystems
> BASES = [
> os.path.join(os.path.dirname(__file__), "./"),
> "gs://my-bucket/test/",
> ]
> pattern = "filesystem-match-test/*/file.txt"
> for base_path in BASES:
> full_pattern = posixpath.join(base_path, pattern)
> print("full_pattern: {}".format(full_pattern))
> match_result = FileSystems.match([full_pattern])[0]  # type: MatchResult
> print("metadata list: {}".format(match_result.metadata_list))
> {code}
> Running {{python filesystem-match-test.py}} does not match any files locally, 
> but does match files on GCS:
> {noformat}
> full_pattern: ./filesystem-match-test/*/file.txt
> metadata list: []
> full_pattern: gs://my-bucket/test/filesystem-match-test/*/file.txt
> metadata list: 
> [FileMetadata(gs://my-bucket/test/filesystem-match-test/a/file.txt, 6), 
> FileMetadata(gs://my-bucket/test/filesystem-match-test/b/file.txt, 6)]
> {noformat}
> The expected result is that a/file.txt and b/file.txt should be matched for 
> both patterns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5417) FileSystems.match behaviour diff between GCS and local file system

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5417?focusedWorklogId=146612=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146612
 ]

ASF GitHub Bot logged work on BEAM-5417:


Author: ASF GitHub Bot
Created on: 21/Sep/18 22:07
Start Date: 21/Sep/18 22:07
Worklog Time Spent: 10m 
  Work Description: joar commented on issue #6423: [BEAM-5417] Parity 
between GCS and local match
URL: https://github.com/apache/beam/pull/6423#issuecomment-423683905
 
 
   Unless of course we use `pathlib.Path().glob()` rather than 
`PurePath().match()`, and in LocalFileSystem only rather than FileSystem. I'm 
done for today though, I'll have to check back later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146612)
Time Spent: 1h 40m  (was: 1.5h)

> FileSystems.match behaviour diff between GCS and local file system
> --
>
> Key: BEAM-5417
> URL: https://issues.apache.org/jira/browse/BEAM-5417
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Joar Wandborg
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Given the directory structure:
>  
> {noformat}
> .
> ├── filesystem-match-test
> │   ├── a
> │   │   └── file.txt
> │   └── b
> │   └── file.txt
> └── filesystem-match-test.py
> {noformat}
>  
> Where {{filesystem-match-test.py}} contains:
> {code:python}
> from __future__ import print_function
> import os
> import posixpath
> from apache_beam.io.filesystem import MatchResult
> from apache_beam.io.filesystems import FileSystems
> BASES = [
> os.path.join(os.path.dirname(__file__), "./"),
> "gs://my-bucket/test/",
> ]
> pattern = "filesystem-match-test/*/file.txt"
> for base_path in BASES:
> full_pattern = posixpath.join(base_path, pattern)
> print("full_pattern: {}".format(full_pattern))
> match_result = FileSystems.match([full_pattern])[0]  # type: MatchResult
> print("metadata list: {}".format(match_result.metadata_list))
> {code}
> Running {{python filesystem-match-test.py}} does not match any files locally, 
> but does match files on GCS:
> {noformat}
> full_pattern: ./filesystem-match-test/*/file.txt
> metadata list: []
> full_pattern: gs://my-bucket/test/filesystem-match-test/*/file.txt
> metadata list: 
> [FileMetadata(gs://my-bucket/test/filesystem-match-test/a/file.txt, 6), 
> FileMetadata(gs://my-bucket/test/filesystem-match-test/b/file.txt, 6)]
> {noformat}
> The expected result is that a/file.txt and b/file.txt should be matched for 
> both patterns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #1561

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5417) FileSystems.match behaviour diff between GCS and local file system

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5417?focusedWorklogId=146611=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146611
 ]

ASF GitHub Bot logged work on BEAM-5417:


Author: ASF GitHub Bot
Created on: 21/Sep/18 22:04
Start Date: 21/Sep/18 22:04
Worklog Time Spent: 10m 
  Work Description: joar commented on issue #6423: [BEAM-5417] Parity 
between GCS and local match
URL: https://github.com/apache/beam/pull/6423#issuecomment-423683233
 
 
   @udim It took me too long to realize thah`PurePath.match()` [also uses 
fnmatch](https://github.com/python/cpython/blob/3.7/Lib/pathlib.py#L944-L947).
   
   Maintaining our own state machine seems to be the only remaining solution.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146611)
Time Spent: 1.5h  (was: 1h 20m)

> FileSystems.match behaviour diff between GCS and local file system
> --
>
> Key: BEAM-5417
> URL: https://issues.apache.org/jira/browse/BEAM-5417
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Joar Wandborg
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Given the directory structure:
>  
> {noformat}
> .
> ├── filesystem-match-test
> │   ├── a
> │   │   └── file.txt
> │   └── b
> │   └── file.txt
> └── filesystem-match-test.py
> {noformat}
>  
> Where {{filesystem-match-test.py}} contains:
> {code:python}
> from __future__ import print_function
> import os
> import posixpath
> from apache_beam.io.filesystem import MatchResult
> from apache_beam.io.filesystems import FileSystems
> BASES = [
> os.path.join(os.path.dirname(__file__), "./"),
> "gs://my-bucket/test/",
> ]
> pattern = "filesystem-match-test/*/file.txt"
> for base_path in BASES:
> full_pattern = posixpath.join(base_path, pattern)
> print("full_pattern: {}".format(full_pattern))
> match_result = FileSystems.match([full_pattern])[0]  # type: MatchResult
> print("metadata list: {}".format(match_result.metadata_list))
> {code}
> Running {{python filesystem-match-test.py}} does not match any files locally, 
> but does match files on GCS:
> {noformat}
> full_pattern: ./filesystem-match-test/*/file.txt
> metadata list: []
> full_pattern: gs://my-bucket/test/filesystem-match-test/*/file.txt
> metadata list: 
> [FileMetadata(gs://my-bucket/test/filesystem-match-test/a/file.txt, 6), 
> FileMetadata(gs://my-bucket/test/filesystem-match-test/b/file.txt, 6)]
> {noformat}
> The expected result is that a/file.txt and b/file.txt should be matched for 
> both patterns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #99

[Apache Jenkins Server]

See 


Changes:

[mwylde] [BEAM-5455] Log per-bundle messages at debug in python sdk

--
[...truncated 53.56 MB...]
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (9/16) 
(cf85cd99f8a60679229ce0caf0fb9177) switched from RUNNING to FINISHED.
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(c8f18ec84b1a13afed9a7d06c3564193) switched from RUNNING to FINISHED.
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(c8f18ec84b1a13afed9a7d06c3564193).
[GroupByKey -> 24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem 
streams are closed for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(c8f18ec84b1a13afed9a7d06c3564193) [FINISHED]
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
c8f18ec84b1a13afed9a7d06c3564193.
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (7/16) (02a5745076755289b117c0d00c7c22e8) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (7/16) 
(02a5745076755289b117c0d00c7c22e8).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(7/16) (d7ddf67fc5ed2d233d154ffa22bc3df4) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(d7ddf67fc5ed2d233d154ffa22bc3df4).
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (7/16) 
(02a5745076755289b117c0d00c7c22e8) [FINISHED]
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (1/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(1/16) (29f9f7649062ae7b714d6e0b13b19560) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (1/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (1/16) 
(29f9f7649062ae7b714d6e0b13b19560).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (9/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(9/16) (ea60ec2b57db23779261925b5d93b5cc) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (9/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (9/16) 
(ea60ec2b57db23779261925b5d93b5cc).
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(c8f18ec84b1a13afed9a7d06c3564193) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
02a5745076755289b117c0d00c7c22e8.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (4/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(4/16) (146bee08dbee92baa08e1c74d636f074) switched from RUNNING to FINISHED.

[jira] [Work logged] (BEAM-5460) Update Dataflow Python API client

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5460?focusedWorklogId=146608=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146608
 ]

ASF GitHub Bot logged work on BEAM-5460:


Author: ASF GitHub Bot
Created on: 21/Sep/18 21:57
Start Date: 21/Sep/18 21:57
Worklog Time Spent: 10m 
  Work Description: charlesccychen removed a comment on issue #6460: 
[BEAM-5460] Update Python Dataflow API client
URL: https://github.com/apache/beam/pull/6460#issuecomment-423681876
 
 
   Run Python PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146608)
Time Spent: 0.5h  (was: 20m)

> Update Dataflow Python API client
> -
>
> Key: BEAM-5460
> URL: https://issues.apache.org/jira/browse/BEAM-5460
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Affects Versions: 2.7.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> We should update the Dataflow Python API client to reflect the most current 
> Dataflow API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5460) Update Dataflow Python API client

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5460?focusedWorklogId=146609=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146609
 ]

ASF GitHub Bot logged work on BEAM-5460:


Author: ASF GitHub Bot
Created on: 21/Sep/18 21:57
Start Date: 21/Sep/18 21:57
Worklog Time Spent: 10m 
  Work Description: charlesccychen removed a comment on issue #6460: 
[BEAM-5460] Update Python Dataflow API client
URL: https://github.com/apache/beam/pull/6460#issuecomment-423681890
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146609)
Time Spent: 40m  (was: 0.5h)

> Update Dataflow Python API client
> -
>
> Key: BEAM-5460
> URL: https://issues.apache.org/jira/browse/BEAM-5460
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Affects Versions: 2.7.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> We should update the Dataflow Python API client to reflect the most current 
> Dataflow API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #6049

[Apache Jenkins Server]

See 


Changes:

[scott] Upgrade gradle wrapper to latest (4.10)

--
[...truncated 1.21 MB...]
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 407, in run
return self.runner.run_pipeline(self)
  File 
"
 line 133, in run_pipeline
from apache_beam.runners.portability.fn_api_runner import FnApiRunner
  File 
"
 line 66, in 
beam.transforms.window.GlobalWindows.windowed_value(''))
  File 
"
 line 89, in encode_nested
self.encode_to_stream(value, out, True)
  File 
"
 line 798, in encode_to_stream
self._value_coder.encode_to_stream(wv.value, out, nested)
  File 
"
 line 363, in encode_to_stream
out.write(value, nested)
  File 
"
 line 40, in write
assert isinstance(b, bytes)
AssertionError: 
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
- >> end captured logging << -

==
FAIL: test_create (apache_beam.pipeline_test.PipelineTest)
--
Traceback (most recent call last):
  File 
"
 line 154, in test_create
pipeline.run()
  File 
"
 line 104, in run
result = super(TestPipeline, self).run(test_runner_api)
  File 
"
 line 394, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 407, in run
return self.runner.run_pipeline(self)
  File 
"
 line 133, in run_pipeline
from apache_beam.runners.portability.fn_api_runner import FnApiRunner
  File 
"
 line 66, in 
beam.transforms.window.GlobalWindows.windowed_value(''))
  File 
"
 line 89, in encode_nested
self.encode_to_stream(value, out, True)
  File 
"
 line 798, in encode_to_stream
self._value_coder.encode_to_stream(wv.value, out, nested)
  File 
"
 line 363, in encode_to_stream
out.write(value, nested)
  File 
"
 line 40, in write
assert isinstance(b, bytes)
AssertionError: 
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
- >> end captured logging << -

==
FAIL: test_create_singleton_pcollection (apache_beam.pipeline_test.PipelineTest)
--
Traceback (most recent call last):
  File 
"
 line 176, in test_create_singleton_pcollection
pipeline.run()
  File 

[jira] [Work logged] (BEAM-5460) Update Dataflow Python API client

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5460?focusedWorklogId=146606=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146606
 ]

ASF GitHub Bot logged work on BEAM-5460:


Author: ASF GitHub Bot
Created on: 21/Sep/18 21:56
Start Date: 21/Sep/18 21:56
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #6460: [BEAM-5460] 
Update Python Dataflow API client
URL: https://github.com/apache/beam/pull/6460#issuecomment-423681876
 
 
   Run Python PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146606)
Time Spent: 10m
Remaining Estimate: 0h

> Update Dataflow Python API client
> -
>
> Key: BEAM-5460
> URL: https://issues.apache.org/jira/browse/BEAM-5460
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Affects Versions: 2.7.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We should update the Dataflow Python API client to reflect the most current 
> Dataflow API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5460) Update Dataflow Python API client

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5460?focusedWorklogId=146607=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146607
 ]

ASF GitHub Bot logged work on BEAM-5460:


Author: ASF GitHub Bot
Created on: 21/Sep/18 21:56
Start Date: 21/Sep/18 21:56
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #6460: [BEAM-5460] 
Update Python Dataflow API client
URL: https://github.com/apache/beam/pull/6460#issuecomment-423681890
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146607)
Time Spent: 20m  (was: 10m)

> Update Dataflow Python API client
> -
>
> Key: BEAM-5460
> URL: https://issues.apache.org/jira/browse/BEAM-5460
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Affects Versions: 2.7.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We should update the Dataflow Python API client to reflect the most current 
> Dataflow API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (7c8c74a -> b276db3)

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7c8c74a  Merge pull request #6444: Add Portable PostCommit test status 
to PR template
 add 3f9aa26  [BEAM-5455] Log per-bundle messages at debug in python sdk
 new b276db3  Merge pull request #6459: [BEAM-5455] Log per-bundle messages 
at debug in python sdk

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/worker/bundle_processor.py | 4 ++--
 sdks/python/apache_beam/runners/worker/sdk_worker.py   | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

[beam] 01/01: Merge pull request #6459: [BEAM-5455] Log per-bundle messages at debug in python sdk

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b276db324f0fa320b7e67c48d626983778ddab04
Merge: 7c8c74a 3f9aa26
Author: Thomas Weise 
AuthorDate: Fri Sep 21 14:49:06 2018 -0700

Merge pull request #6459: [BEAM-5455] Log per-bundle messages at debug in 
python sdk

 sdks/python/apache_beam/runners/worker/bundle_processor.py | 4 ++--
 sdks/python/apache_beam/runners/worker/sdk_worker.py   | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

[jira] [Updated] (BEAM-5462) get rid of .options deprecation warnings in tests

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri updated BEAM-5462:

Description: 
Messages look like:

{{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported}}
{{pipeline.replace_all(_get_transform_overrides(pipeline.options))}}

  was:
Messages look like:

{{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported}}
{{  pipeline.replace_all(_get_transform_overrides(pipeline.options))}}


> get rid of .options deprecation warnings in tests
> ---
>
> Key: BEAM-5462
> URL: https://issues.apache.org/jira/browse/BEAM-5462
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Minor
>
> Messages look like:
> {{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
>  DeprecationWarning: options is deprecated since First stable release. 
> References to .options will not be supported}}
> {{pipeline.replace_all(_get_transform_overrides(pipeline.options))}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5462) get rid of .options deprecation warnings in tests

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri updated BEAM-5462:

Description: 
Messages look like:

{{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  pipeline.replace_all(_get_transform_overrides(pipeline.options))}}

  was:
Messages look like:
/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  pipeline.replace_all(_get_transform_overrides(pipeline.options))


> get rid of .options deprecation warnings in tests
> ---
>
> Key: BEAM-5462
> URL: https://issues.apache.org/jira/browse/BEAM-5462
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Minor
>
> Messages look like:
> {{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
>  DeprecationWarning: options is deprecated since First stable release. 
> References to .options will not be supported
>   pipeline.replace_all(_get_transform_overrides(pipeline.options))}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5462) get rid of .options deprecation warnings in tests

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri updated BEAM-5462:

Description: 
Messages look like:

{{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported}}
{{  pipeline.replace_all(_get_transform_overrides(pipeline.options))}}

  was:
Messages look like:

{{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  pipeline.replace_all(_get_transform_overrides(pipeline.options))}}


> get rid of .options deprecation warnings in tests
> ---
>
> Key: BEAM-5462
> URL: https://issues.apache.org/jira/browse/BEAM-5462
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Minor
>
> Messages look like:
> {{/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
>  DeprecationWarning: options is deprecated since First stable release. 
> References to .options will not be supported}}
> {{  pipeline.replace_all(_get_transform_overrides(pipeline.options))}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5462) get rid of .options deprecation warnings in tests

[Udi Meiri (JIRA)]

Udi Meiri created BEAM-5462:
---

 Summary: get rid of .options deprecation warnings in 
tests
 Key: BEAM-5462
 URL: https://issues.apache.org/jira/browse/BEAM-5462
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Udi Meiri
Assignee: Udi Meiri


Messages look like:
/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py:360:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  pipeline.replace_all(_get_transform_overrides(pipeline.options))



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #1560

[Apache Jenkins Server]

See 


Changes:

[robbe.sneyders] Fix pipeline_test

[ankurgoenka] Add Portable PostCommit test status to PR template

[robbe.sneyders] Fix hamcrest docs dependency

--
[...truncated 30.16 MB...]
[dag-scheduler-event-loop] INFO org.apache.spark.scheduler.DAGScheduler - 
Submitting 4 missing tasks from ResultStage 547 (MapPartitionsRDD[2773] at map 
at TranslationUtils.java:129) (first 15 tasks are for partitions Vector(0, 1, 
2, 3))
[dag-scheduler-event-loop] INFO 
org.apache.spark.scheduler.TaskSchedulerImpl - Adding task set 547.0 with 4 
tasks
[dispatcher-event-loop-0] INFO org.apache.spark.scheduler.TaskSetManager - 
Starting task 0.0 in stage 547.0 (TID 463, localhost, executor driver, 
partition 0, PROCESS_LOCAL, 8308 bytes)
[dispatcher-event-loop-0] INFO org.apache.spark.scheduler.TaskSetManager - 
Starting task 1.0 in stage 547.0 (TID 464, localhost, executor driver, 
partition 1, PROCESS_LOCAL, 8308 bytes)
[dispatcher-event-loop-0] INFO org.apache.spark.scheduler.TaskSetManager - 
Starting task 2.0 in stage 547.0 (TID 465, localhost, executor driver, 
partition 2, PROCESS_LOCAL, 8308 bytes)
[dispatcher-event-loop-0] INFO org.apache.spark.scheduler.TaskSetManager - 
Starting task 3.0 in stage 547.0 (TID 466, localhost, executor driver, 
partition 3, PROCESS_LOCAL, 8308 bytes)
[Executor task launch worker for task 463] INFO 
org.apache.spark.executor.Executor - Running task 0.0 in stage 547.0 (TID 463)
[Executor task launch worker for task 464] INFO 
org.apache.spark.executor.Executor - Running task 1.0 in stage 547.0 (TID 464)
[Executor task launch worker for task 466] INFO 
org.apache.spark.executor.Executor - Running task 3.0 in stage 547.0 (TID 466)
[Executor task launch worker for task 465] INFO 
org.apache.spark.executor.Executor - Running task 2.0 in stage 547.0 (TID 465)
[Executor task launch worker for task 464] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty 
blocks out of 5 blocks
[Executor task launch worker for task 463] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty 
blocks out of 5 blocks
[Executor task launch worker for task 464] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches 
in 0 ms
[Executor task launch worker for task 463] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches 
in 0 ms
[Executor task launch worker for task 465] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty 
blocks out of 5 blocks
[Executor task launch worker for task 465] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Started 0 remote fetches 
in 0 ms
[Executor task launch worker for task 465] INFO 
org.apache.spark.storage.BlockManager - Found block rdd_2441_2 locally
[Executor task launch worker for task 464] INFO 
org.apache.spark.storage.BlockManager - Found block rdd_2441_1 locally
[Executor task launch worker for task 463] INFO 
org.apache.spark.storage.BlockManager - Found block rdd_2441_0 locally
[Executor task launch worker for task 464] INFO 
org.apache.spark.storage.memory.MemoryStore - Block rdd_2756_1 stored as bytes 
in memory (estimated size 4.0 B, free 13.5 GB)
[Executor task launch worker for task 463] INFO 
org.apache.spark.storage.memory.MemoryStore - Block rdd_2756_0 stored as bytes 
in memory (estimated size 4.0 B, free 13.5 GB)
[Executor task launch worker for task 465] INFO 
org.apache.spark.storage.memory.MemoryStore - Block rdd_2756_2 stored as bytes 
in memory (estimated size 4.0 B, free 13.5 GB)
[dispatcher-event-loop-0] INFO org.apache.spark.storage.BlockManagerInfo - 
Added rdd_2756_0 in memory on localhost:37925 (size: 4.0 B, free: 13.5 GB)
[dispatcher-event-loop-0] INFO org.apache.spark.storage.BlockManagerInfo - 
Added rdd_2756_1 in memory on localhost:37925 (size: 4.0 B, free: 13.5 GB)
[dispatcher-event-loop-0] INFO org.apache.spark.storage.BlockManagerInfo - 
Added rdd_2756_2 in memory on localhost:37925 (size: 4.0 B, free: 13.5 GB)
[Executor task launch worker for task 465] INFO 
org.apache.spark.executor.Executor - Finished task 2.0 in stage 547.0 (TID 
465). 59881 bytes result sent to driver
[Executor task launch worker for task 464] INFO 
org.apache.spark.executor.Executor - Finished task 1.0 in stage 547.0 (TID 
464). 59881 bytes result sent to driver
[Executor task launch worker for task 463] INFO 
org.apache.spark.executor.Executor - Finished task 0.0 in stage 547.0 (TID 
463). 59881 bytes result sent to driver
[Executor task launch worker for task 466] INFO 
org.apache.spark.storage.ShuffleBlockFetcherIterator - Getting 0 non-empty 
blocks out of 5 blocks
[Executor task launch worker for task 466] INFO 

[jira] [Commented] (BEAM-5440) Add option to mount a directory inside SDK harness containers

[Henning Rohde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624222#comment-16624222
 ] 

Henning Rohde commented on BEAM-5440:
-

This is something that I think makes sense and is userful for local development 
only. For non-local, I agree with [~thw] that process is likely a better option 
or something runner or deployment specific, such as fixed mappings 
(/opt/lyft/catphotos is just always mapped in).

> Add option to mount a directory inside SDK harness containers
> -
>
> Key: BEAM-5440
> URL: https://issues.apache.org/jira/browse/BEAM-5440
> Project: Beam
>  Issue Type: New Feature
>  Components: java-fn-execution
>Reporter: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>
> While experimenting with the Python SDK locally, I found it inconvenient that 
> I can't mount a host directory to the Docker containers, i.e. the input must 
> already be in the container and the results of a Write remain inside the 
> container. For local testing, users may want to mount a host directory.
> Since BEAM-5288 the {{Environment}} carries explicit environment information, 
> we could a) add volume args to the {{DockerPayload}}, or b) provide a general 
> Docker arguments field.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #98

[Apache Jenkins Server]

See 


Changes:

[robbe.sneyders] Fix pipeline_test

[ankurgoenka] Add Portable PostCommit test status to PR template

[robbe.sneyders] Fix hamcrest docs dependency

--
[...truncated 59.87 MB...]
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (1/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (1/16) 
(24129ac76ddd68fd213b39d8bac10ea7).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(11/16) (dda0790b2ce304b324f4c13dc3134353) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16) 
(dda0790b2ce304b324f4c13dc3134353).
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (7/16) (c00aef122f648eaa4549aa4df49caf56) switched from RUNNING 
to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (4/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(4/16) (2f686817054f5b8a26c3ad4f0e21a897) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (15/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(15/16) (a7fadc8414c7cf1d30a41b072a032171) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (9/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(9/16) (6f2beb8150b0899234f23fccf532a8ff) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(7/16) (144dd471ce6d7b45a216f4bef7886220) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (10/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(10/16) (b33dc94b0d00a9683087e97fdb5e6ed9) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (10/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (10/16) 
(b33dc94b0d00a9683087e97fdb5e6ed9).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (14/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(14/16) (4b6a9ae506cebc9089f2d5af9dd53122) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (16/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(16/16) (b1e6e4d10a4a6315f9c7a48d1620ec7b) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (6/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(6/16) (9fe1fdf3c100f4c77e4ead99addf3d4e) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (16/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (16/16) 
(b1e6e4d10a4a6315f9c7a48d1620ec7b).

[jira] [Updated] (BEAM-5447) Support running user pipelines with the Universal Local Runner in Go.

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-5447:

Component/s: sdk-go

> Support running user pipelines with the Universal Local Runner in Go.
> -
>
> Key: BEAM-5447
> URL: https://issues.apache.org/jira/browse/BEAM-5447
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct, sdk-go
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>
> In order to aid testing, devs should be able to write pipelines and then 
> easily run them with the ULR. This should be generally trivial with Go as the 
> existing local runner for Go uses a build rule that should be nearly 
> identical to the build rule needed to complete this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #97

[Apache Jenkins Server]

See 


Changes:

[mxm] [BEAM-5442] Add basic FlinkOptions to Python SDK

--
[...truncated 59.95 MB...]
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (10/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (10/16) 
(917482d2a1c3d6028e72dce72845a782).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (1/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(1/16) (3b3567db6ee5276019b60c29e1e12033) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
8fc42143899661cf203ed4dbc65e7dc5.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (6/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(6/16) (e2cd536a5a7c7196eb245a0577a41ce9) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (6/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (6/16) 
(e2cd536a5a7c7196eb245a0577a41ce9).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (1/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (1/16) 
(3b3567db6ee5276019b60c29e1e12033).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (3/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(3/16) (17bdc2cd07a78917bd7fb0d75f5b4f5b) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (3/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (3/16) 
(17bdc2cd07a78917bd7fb0d75f5b4f5b).
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (7/16) 
(8fc42143899661cf203ed4dbc65e7dc5) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(11/16) (05beff746b93d9112c14557a280d641d) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16) 
(05beff746b93d9112c14557a280d641d).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (16/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(16/16) (10d54caf70c38c87c568ce756012c5b1) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (16/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (16/16) 
(10d54caf70c38c87c568ce756012c5b1).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(7/16) (50d152583fa76cea21915f778d81fd2c) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 

[beam] 01/01: Merge pull request #6444: Add Portable PostCommit test status to PR template

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7c8c74a056c9a2f788efd2439c19178532a3e59e
Merge: 9e5a725 a7a100b
Author: Thomas Weise 
AuthorDate: Fri Sep 21 14:25:58 2018 -0700

Merge pull request #6444: Add Portable PostCommit test status to PR template

 .github/PULL_REQUEST_TEMPLATE.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[beam] branch master updated (9e5a725 -> 7c8c74a)

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9e5a725  Merge pull request #6465: [BEAM-5458] Fix python pre-commit
 add a7a100b  Add Portable PostCommit test status to PR template
 new 7c8c74a  Merge pull request #6444: Add Portable PostCommit test status 
to PR template

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .github/PULL_REQUEST_TEMPLATE.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[jira] [Work logged] (BEAM-5455) Don't info log for every bundle in the python sdk

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5455?focusedWorklogId=146602=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146602
 ]

ASF GitHub Bot logged work on BEAM-5455:


Author: ASF GitHub Bot
Created on: 21/Sep/18 21:23
Start Date: 21/Sep/18 21:23
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6459: [BEAM-5455] Log 
per-bundle messages at debug in python sdk
URL: https://github.com/apache/beam/pull/6459#issuecomment-423674747
 
 
   Run Python PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146602)
Time Spent: 50m  (was: 40m)

> Don't info log for every bundle in the python sdk
> -
>
> Key: BEAM-5455
> URL: https://issues.apache.org/jira/browse/BEAM-5455
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: Micah Wylde
>Assignee: Micah Wylde
>Priority: Minor
>  Labels: portability-flink
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Currently the python sdk logs a bunch of messages at INFO level for every 
> bundle. For the portable flink runner, which uses a bundle size of 1 in 
> streaming mode, this produces huge log spew:
> {code:java}
> [grpc-default-executor-1] INFO sdk_worker.run - Got work 4
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
> 
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - start 
>  coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  coder=WindowedValueCoder[BytesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  receivers=[ConsumerSet[timestamp.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  receivers=[ConsumerSet[pair_with_one.out0, 
> coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder, 
> FastPrimitivesCoder]], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
>  coder=WindowedValueCoder[TupleCoder[FastPrimitivesCoder, 
> FastPrimitivesCoder]], len(consumers)=1]]>
> [grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 
> 
> {code}
>  
> These log messages should probably be at debug instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5458) Python PreCommit broken

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5458?focusedWorklogId=146601=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146601
 ]

ASF GitHub Bot logged work on BEAM-5458:


Author: ASF GitHub Bot
Created on: 21/Sep/18 21:22
Start Date: 21/Sep/18 21:22
Worklog Time Spent: 10m 
  Work Description: tweise closed pull request #6465: [BEAM-5458] Fix 
python tests
URL: https://github.com/apache/beam/pull/6465
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/runners/portability/fn_api_runner.py 
b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
index 32d7adef068..4b8e6dabbc0 100644
--- a/sdks/python/apache_beam/runners/portability/fn_api_runner.py
+++ b/sdks/python/apache_beam/runners/portability/fn_api_runner.py
@@ -63,9 +63,9 @@
 ENCODED_IMPULSE_VALUE = beam.coders.WindowedValueCoder(
 beam.coders.BytesCoder(),
 beam.coders.coders.GlobalWindowCoder()).get_impl().encode_nested(
-beam.transforms.window.GlobalWindows.windowed_value(''))
+beam.transforms.window.GlobalWindows.windowed_value(b''))
 
-IMPULSE_BUFFER_PREFIX = 'impulse:'
+IMPULSE_BUFFER_PREFIX = b'impulse:'
 
 
 class BeamFnControlServicer(beam_fn_api_pb2_grpc.BeamFnControlServicer):
@@ -456,7 +456,8 @@ def impulse_to_input(stages):
 unique_name=transform.unique_name,
 spec=beam_runner_api_pb2.FunctionSpec(
 urn=bundle_processor.DATA_INPUT_URN,
-payload=str(IMPULSE_BUFFER_PREFIX + impulse_pc)),
+payload=IMPULSE_BUFFER_PREFIX +
+impulse_pc.encode('utf-8')),
 outputs=transform.outputs))
 
 yield stage
diff --git a/sdks/python/scripts/generate_pydoc.sh 
b/sdks/python/scripts/generate_pydoc.sh
index 2f6f0f14cfb..e9b5c812736 100755
--- a/sdks/python/scripts/generate_pydoc.sh
+++ b/sdks/python/scripts/generate_pydoc.sh
@@ -113,7 +113,7 @@ import apache_beam as beam
 
 intersphinx_mapping = {
   'python': ('https://docs.python.org/2', None),
-  'hamcrest': ('https://pyhamcrest.readthedocs.io/en/latest/', None),
+  'hamcrest': ('https://pyhamcrest.readthedocs.io/en/stable/', None),
 }
 
 # Since private classes are skipped by sphinx, if there is any cross reference


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146601)
Time Spent: 1h  (was: 50m)

> Python PreCommit broken
> ---
>
> Key: BEAM-5458
> URL: https://issues.apache.org/jira/browse/BEAM-5458
> Project: Beam
>  Issue Type: Test
>  Components: test-failures, testing
>Reporter: Maximilian Michels
>Priority: Critical
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Python PreCommit is currently broken:
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/1308/#showFailuresLink



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2901) Containerize ULR

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde reassigned BEAM-2901:
---

Assignee: Daniel Oliveira

> Containerize ULR
> 
>
> Key: BEAM-2901
> URL: https://issues.apache.org/jira/browse/BEAM-2901
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Henning Rohde
>Assignee: Daniel Oliveira
>Priority: Minor
>  Labels: portability
>
> We should containerize ULR as a convenience options for users who have docker 
> installed, but perhaps not java.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6465: [BEAM-5458] Fix python pre-commit

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 9e5a72549fdb0bbcb333d7cdce66696613e894d3
Merge: aae50db 4b78058
Author: Thomas Weise 
AuthorDate: Fri Sep 21 14:22:10 2018 -0700

Merge pull request #6465: [BEAM-5458] Fix python pre-commit

 sdks/python/apache_beam/runners/portability/fn_api_runner.py | 7 ---
 sdks/python/scripts/generate_pydoc.sh| 2 +-
 2 files changed, 5 insertions(+), 4 deletions(-)

[jira] [Work logged] (BEAM-5458) Python PreCommit broken

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5458?focusedWorklogId=146600=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146600
 ]

ASF GitHub Bot logged work on BEAM-5458:


Author: ASF GitHub Bot
Created on: 21/Sep/18 21:21
Start Date: 21/Sep/18 21:21
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6465: [BEAM-5458] Fix python 
tests
URL: https://github.com/apache/beam/pull/6465#issuecomment-423674263
 
 
   Let's merge before it breaks again :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146600)
Time Spent: 50m  (was: 40m)

> Python PreCommit broken
> ---
>
> Key: BEAM-5458
> URL: https://issues.apache.org/jira/browse/BEAM-5458
> Project: Beam
>  Issue Type: Test
>  Components: test-failures, testing
>Reporter: Maximilian Michels
>Priority: Critical
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Python PreCommit is currently broken:
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/1308/#showFailuresLink



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2900) ULR: configurable container/process management

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-2900.
-
   Resolution: Won't Fix
Fix Version/s: Not applicable

Obsolete now that we're adding direct process support.

> ULR: configurable container/process management
> --
>
> Key: BEAM-2900
> URL: https://issues.apache.org/jira/browse/BEAM-2900
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>
> The ULR should support configurable container/process management as per 
> https://s.apache.org/beam-fn-api-container-contract. It would be convenient 
> if containers are optional for testing purposes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (aae50db -> 9e5a725)

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from aae50db  Merge pull request #6447: [BEAM-5442] Add FlinkOptions to 
Python SDK
 add 1afa7db  Fix pipeline_test
 add 4b78058  Fix hamcrest docs dependency
 new 9e5a725  Merge pull request #6465: [BEAM-5458] Fix python pre-commit

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/portability/fn_api_runner.py | 7 ---
 sdks/python/scripts/generate_pydoc.sh| 2 +-
 2 files changed, 5 insertions(+), 4 deletions(-)

[jira] [Resolved] (BEAM-2884) Dataflow runs portable pipelines

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-2884.
-
   Resolution: Fixed
Fix Version/s: 2.8.0

All SDKs have a Dataflow-specific job-submission code to work around the lack 
of job api support, but it otherwise runs portable pipelines. Closing.

> Dataflow runs portable pipelines
> 
>
> Key: BEAM-2884
> URL: https://issues.apache.org/jira/browse/BEAM-2884
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Priority: Major
>  Labels: portability
> Fix For: 2.8.0
>
>
> Dataflow should run pipelines using the full portability API as currently 
> defined:
> https://s.apache.org/beam-fn-api 
> https://s.apache.org/beam-runner-api
> https://s.apache.org/beam-job-api
> https://s.apache.org/beam-fn-api-container-contract
> This issue tracks its adoption of the portability framework. New Fn API and 
> other features will be tracked separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3514) Use portable WindowIntoPayload in DataflowRunner

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-3514:

Issue Type: Task  (was: Sub-task)
Parent: (was: BEAM-2884)

> Use portable WindowIntoPayload in DataflowRunner
> 
>
> Key: BEAM-3514
> URL: https://issues.apache.org/jira/browse/BEAM-3514
> Project: Beam
>  Issue Type: Task
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>
> The Java-specific blobs transmitted to Dataflow need more context, in the 
> form of portability framework protos.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5126) PreCommit filtering broken based upon PR contents

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624190#comment-16624190
 ] 

Luke Cwik commented on BEAM-5126:
-

I'm not certain as to whether its broken or not anymore.

> PreCommit filtering broken based upon PR contents
> -
>
> Key: BEAM-5126
> URL: https://issues.apache.org/jira/browse/BEAM-5126
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Alan Myrvold
>Priority: Minor
>
> PR precommits used to be filtered by the contents of the PR.
>  
> Example PR that should have only spawned the Java PreCommit:
> https://github.com/apache/beam/pull/6159
>  
> This broke work done in BEAM-4445



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #96

[Apache Jenkins Server]

See 


Changes:

[scott] Upgrade gradle wrapper to latest (4.10)

--
[...truncated 60.14 MB...]
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (5/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(5/16) (5b1db8c497856f658fbd17c29b97877e) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (5/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (5/16) 
(5b1db8c497856f658fbd17c29b97877e).
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16) 
(c5c32629fc9d8b37cf43e9c8a5929ffa) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (13/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(13/16) (e163ac11aaad9c58928b4053ba0e9ce6) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (13/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (13/16) 
(e163ac11aaad9c58928b4053ba0e9ce6).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (14/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(14/16) (7bb845c239229e560f5b902fef356e6c) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (12/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(12/16) (9e18a428a742b3008608f7311a8881a6) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (3/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (3/16) 
(d77c8c990dc924eb4875d946af34571e).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(11/16) (4a7c1fc8144828b21f65eb53a4a4591e) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for 
assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16) 
(4a7c1fc8144828b21f65eb53a4a4591e).
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (3/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem 
streams are closed for task assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (3/16) 
(d77c8c990dc924eb4875d946af34571e) [FINISHED]
[ToKeyedWorkItem (7/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (7/16) 
(5683639c38b59f853b30e78abde90dea) [FINISHED]
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem 
streams are closed for task assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (11/16) 
(4a7c1fc8144828b21f65eb53a4a4591e) [FINISHED]
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO org.apache.flink.runtime.taskmanager.Task - assert_that/Group/GroupByKey 
-> 42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
(7/16) (b6f28a7afa8bd22247f0800e4451a048) switched from RUNNING to FINISHED.
[assert_that/Group/GroupByKey -> 
42assert_that/Group/GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (7/16)] 
INFO 

[jira] [Work logged] (BEAM-5036) Optimize FileBasedSink's WriteOperation.moveToOutput()

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5036?focusedWorklogId=146596=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146596
 ]

ASF GitHub Bot logged work on BEAM-5036:


Author: ASF GitHub Bot
Created on: 21/Sep/18 21:10
Start Date: 21/Sep/18 21:10
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #6289: [BEAM-5036] 
Optimize the FileBasedSink WriteOperation.moveToOutput()
URL: https://github.com/apache/beam/pull/6289#issuecomment-423671589
 
 
   Luke, my comment was that definition of "right thing" might be file-system 
dependent. For example, HDFS doesn't support overwriting existing files while 
other file systems might allow this. So we either have to pass the 
responsibility of handling that complexity and providing a unified interface to 
each FileSystem implementation or have to do that at the common FileSystems 
implementation. My argument was that it might be better to handle this 
complexity at a a single common place that is developed/maintained by the Beam 
team instead of passing that to FileSystem authors. 
   
   Tim, in the spirit of the above comment. I prefer handling 
"OVERWRITE_EXISTING_FILES" option at FileSystems level instead of passing to 
FileSystem interface. I believe most file-systems will either fail or support 
overwriting but will not offer that as an option and adding this to FileSystem 
interface will add an extra burden to each FileSystem author since this option 
might have to be supported by utilizing other methods of the FileSystem 
interface. I believe this complexity has to be pushed to FileSystems interface.
   
   Also, agree that this thread has become too long for a PR comment thread. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146596)
Time Spent: 3h 40m  (was: 3.5h)

> Optimize FileBasedSink's WriteOperation.moveToOutput()
> --
>
> Key: BEAM-5036
> URL: https://issues.apache.org/jira/browse/BEAM-5036
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-files
>Affects Versions: 2.5.0
>Reporter: Jozef Vilcek
>Assignee: Tim Robertson
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> moveToOutput() methods in FileBasedSink.WriteOperation implements move by 
> copy+delete. It would be better to use a rename() which can be much more 
> effective for some filesystems.
> Filesystem must support cross-directory rename. BEAM-4861 is related to this 
> for the case of HDFS filesystem.
> Feature was discussed here:
> http://mail-archives.apache.org/mod_mbox/beam-dev/201807.mbox/%3CCAF9t7_4Mp54pQ+vRrJrBh9Vx0=uaknupzd_qdh_qdm9vxll...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2769) Java SDK support for submitting a Portable Pipeline

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde reassigned BEAM-2769:
---

Assignee: Ankur Goenka

> Java SDK support for submitting a Portable Pipeline
> ---
>
> Key: BEAM-2769
> URL: https://issues.apache.org/jira/browse/BEAM-2769
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Ankur Goenka
>Priority: Major
>  Labels: portability
>
> The Java codebase should provide a way to submit a Job to a Job Service.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2769) Java SDK support for submitting a Portable Pipeline

[Henning Rohde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624181#comment-16624181
 ] 

Henning Rohde commented on BEAM-2769:
-

[~angoenka] Isn't this done a long time ago?

> Java SDK support for submitting a Portable Pipeline
> ---
>
> Key: BEAM-2769
> URL: https://issues.apache.org/jira/browse/BEAM-2769
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Ankur Goenka
>Priority: Major
>  Labels: portability
>
> The Java codebase should provide a way to submit a Job to a Job Service.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=146591=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146591
 ]

ASF GitHub Bot logged work on BEAM-5442:


Author: ASF GitHub Bot
Created on: 21/Sep/18 20:57
Start Date: 21/Sep/18 20:57
Worklog Time Spent: 10m 
  Work Description: mxm closed pull request #6447: [BEAM-5442] Add 
FlinkOptions to Python SDK
URL: https://github.com/apache/beam/pull/6447
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/options/pipeline_options.py 
b/sdks/python/apache_beam/options/pipeline_options.py
index b75ed2ab5c3..c1bb2ed89e9 100644
--- a/sdks/python/apache_beam/options/pipeline_options.py
+++ b/sdks/python/apache_beam/options/pipeline_options.py
@@ -662,6 +662,24 @@ def _add_argparse_args(cls, parser):
  'in the pipeline when running using the Fn API.'))
 
 
+class FlinkOptions(PipelineOptions):
+
+  @classmethod
+  def _add_argparse_args(cls, parser):
+parser.add_argument('--flink_master',
+type=str,
+help=
+('Address of the Flink master where the pipeline '
+ 'should be executed. Can either be of the form '
+ '\'host:port\' or one of the special values '
+ '[local], [collection], or [auto].'))
+parser.add_argument('--parallelism',
+type=int,
+help=
+('The degree of parallelism to be used when '
+ 'distributing operations onto workers.'))
+
+
 class TestOptions(PipelineOptions):
 
   @classmethod
diff --git a/sdks/python/apache_beam/options/pipeline_options_test.py 
b/sdks/python/apache_beam/options/pipeline_options_test.py
index 55e8ee88daf..9c14c25668e 100644
--- a/sdks/python/apache_beam/options/pipeline_options_test.py
+++ b/sdks/python/apache_beam/options/pipeline_options_test.py
@@ -104,6 +104,16 @@ def tearDown(self):
'display_data': [
DisplayDataItemMatcher('mock_multi_option', ['op1', 'op2'])]
   },
+  {'flags': ['--flink_master=testmaster:8081', '--parallelism=42'],
+   'expected': {'flink_master': 'testmaster:8081',
+'parallelism': 42,
+'mock_flag': False,
+'mock_option': None,
+'mock_multi_option': None},
+   'display_data': [
+   DisplayDataItemMatcher('flink_master', 'testmaster:8081'),
+   DisplayDataItemMatcher('parallelism', 42)]
+  },
   ]
 
   # Used for testing newly added flags.


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146591)
Time Spent: 2h 50m  (was: 2h 40m)

> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing 
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the 
> PortableRunner. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5451) Add side input support to Universal Local Runner

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira closed BEAM-5451.
-
   Resolution: Duplicate
Fix Version/s: Not applicable

Closed because a JIRA for this task already exists:
https://issues.apache.org/jira/browse/BEAM-2928

> Add side input support to Universal Local Runner
> 
>
> Key: BEAM-5451
> URL: https://issues.apache.org/jira/browse/BEAM-5451
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Ruoyun Huang
>Priority: Minor
> Fix For: Not applicable
>
>
> Get side inputs working on the ULR. Since the ULR code is based on the direct 
> runner code there should already be some code that could be reused, but new 
> code will need to be written where side inputs would interact with 
> portability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=146590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146590
 ]

ASF GitHub Bot logged work on BEAM-5442:


Author: ASF GitHub Bot
Created on: 21/Sep/18 20:56
Start Date: 21/Sep/18 20:56
Worklog Time Spent: 10m 
  Work Description: mxm commented on issue #6447: [BEAM-5442] Add 
FlinkOptions to Python SDK
URL: https://github.com/apache/beam/pull/6447#issuecomment-423668316
 
 
   Merging after I ran `pythonPreCommit` locally with the fix from #6465.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146590)
Time Spent: 2h 40m  (was: 2.5h)

> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing 
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the 
> PortableRunner. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6447: [BEAM-5442] Add FlinkOptions to Python SDK

[mxm]

This is an automated email from the ASF dual-hosted git repository.

mxm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit aae50db7d9656f55cf89121e0220262d4f07089d
Merge: a5d51b0 50528d0
Author: Maximilian Michels 
AuthorDate: Fri Sep 21 22:57:20 2018 +0200

Merge pull request #6447: [BEAM-5442] Add FlinkOptions to Python SDK

 sdks/python/apache_beam/options/pipeline_options.py| 18 ++
 .../apache_beam/options/pipeline_options_test.py   | 10 ++
 2 files changed, 28 insertions(+)

[jira] [Assigned] (BEAM-2928) ULR support for portable side input

[Daniel Oliveira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Oliveira reassigned BEAM-2928:
-

Assignee: Ruoyun Huang  (was: Daniel Oliveira)

> ULR support for portable side input
> ---
>
> Key: BEAM-2928
> URL: https://issues.apache.org/jira/browse/BEAM-2928
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core, runner-direct
>Reporter: Henning Rohde
>Assignee: Ruoyun Huang
>Priority: Major
>  Labels: portability
>
> Get side inputs working on the ULR. Since the ULR code is based on the direct 
> runner code there should already be some code that could be reused, but new 
> code will need to be written where side inputs would interact with 
> portability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2880) Artifact server proxies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2880?focusedWorklogId=146589=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146589
 ]

ASF GitHub Bot logged work on BEAM-2880:


Author: ASF GitHub Bot
Created on: 21/Sep/18 20:54
Start Date: 21/Sep/18 20:54
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #6466: [BEAM-2880] Remove 
gcemd and gcsproxy container images
URL: https://github.com/apache/beam/pull/6466#issuecomment-423667600
 
 
   Run Python PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146589)
Time Spent: 1h 10m  (was: 1h)

> Artifact server proxies
> ---
>
> Key: BEAM-2880
> URL: https://issues.apache.org/jira/browse/BEAM-2880
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Minor
>  Labels: portability
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-fn-api-container-contract, we should add 
> runner-agnostic (dockerized) artifact proxies for various runner 
> environments. These proxies implement the server side of the artifact API.
> We'll likely need proxies for GCS, S3, Azure Storage, and local. If the 
> proxies are implemented in a statically-linked language like Go, say, then 
> they do not have a dependency footprint and can easily be used as native 
> binaries or in dockerized form -- at the discretion of the runner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2880) Artifact server proxies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2880?focusedWorklogId=146586=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146586
 ]

ASF GitHub Bot logged work on BEAM-2880:


Author: ASF GitHub Bot
Created on: 21/Sep/18 20:53
Start Date: 21/Sep/18 20:53
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #6466: [BEAM-2880] Remove 
gcemd and gcsproxy container images
URL: https://github.com/apache/beam/pull/6466#issuecomment-423667372
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146586)
Time Spent: 40m  (was: 0.5h)

> Artifact server proxies
> ---
>
> Key: BEAM-2880
> URL: https://issues.apache.org/jira/browse/BEAM-2880
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Minor
>  Labels: portability
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-fn-api-container-contract, we should add 
> runner-agnostic (dockerized) artifact proxies for various runner 
> environments. These proxies implement the server side of the artifact API.
> We'll likely need proxies for GCS, S3, Azure Storage, and local. If the 
> proxies are implemented in a statically-linked language like Go, say, then 
> they do not have a dependency footprint and can easily be used as native 
> binaries or in dockerized form -- at the discretion of the runner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2880) Artifact server proxies

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2880?focusedWorklogId=146588=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-146588
 ]

ASF GitHub Bot logged work on BEAM-2880:


Author: ASF GitHub Bot
Created on: 21/Sep/18 20:53
Start Date: 21/Sep/18 20:53
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #6466: [BEAM-2880] Remove 
gcemd and gcsproxy container images
URL: https://github.com/apache/beam/pull/6466#issuecomment-423667565
 
 
   Thanks Ankur. I doubt the precommit failures are related to this change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 146588)
Time Spent: 1h  (was: 50m)

> Artifact server proxies
> ---
>
> Key: BEAM-2880
> URL: https://issues.apache.org/jira/browse/BEAM-2880
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Minor
>  Labels: portability
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-fn-api-container-contract, we should add 
> runner-agnostic (dockerized) artifact proxies for various runner 
> environments. These proxies implement the server side of the artifact API.
> We'll likely need proxies for GCS, S3, Azure Storage, and local. If the 
> proxies are implemented in a statically-linked language like Go, say, then 
> they do not have a dependency footprint and can easily be used as native 
> binaries or in dockerized form -- at the discretion of the runner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)