[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233296&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233296
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:01
Start Date: 26/Apr/19 07:01
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486949531
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233296)
Time Spent: 32h 10m  (was: 32h)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 32h 10m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233295&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233295
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:01
Start Date: 26/Apr/19 07:01
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486949419
 
 
   Run Python Load Tests GBK Dataflow Batch With Reiteration
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233295)
Time Spent: 32h  (was: 31h 50m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 32h
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233299&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233299
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:17
Start Date: 26/Apr/19 07:17
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486953699
 
 
   Run Python Load Tests GBK Dataflow Batch With Reiteration
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233299)
Time Spent: 32h 20m  (was: 32h 10m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 32h 20m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233300&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233300
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:18
Start Date: 26/Apr/19 07:18
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486953929
 
 
   Run Python Load Tests GBK Dataflow Batch With Reiteration
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233300)
Time Spent: 32.5h  (was: 32h 20m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 32.5h
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233302&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233302
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:19
Start Date: 26/Apr/19 07:19
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486954316
 
 
   Run Python Load Tests GBK Dataflow Batch
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233302)
Time Spent: 32h 40m  (was: 32.5h)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 32h 40m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6627) Use Metrics API in IO performance tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6627?focusedWorklogId=233301&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233301
 ]

ASF GitHub Bot logged work on BEAM-6627:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:19
Start Date: 26/Apr/19 07:19
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #8400: [BEAM-6627] Added 
byte and item counting metrics to integration tests
URL: https://github.com/apache/beam/pull/8400#issuecomment-486954187
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233301)
Time Spent: 10h 50m  (was: 10h 40m)

> Use Metrics API in IO performance tests
> ---
>
> Key: BEAM-6627
> URL: https://issues.apache.org/jira/browse/BEAM-6627
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 10h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233310&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233310
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:40
Start Date: 26/Apr/19 07:40
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-483707522
 
 
   Run Python Load Tests Smoke
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233310)
Time Spent: 33h  (was: 32h 50m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 33h
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233308&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233308
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:40
Start Date: 26/Apr/19 07:40
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486960063
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233308)
Time Spent: 32h 50m  (was: 32h 40m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 32h 50m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233311&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233311
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:40
Start Date: 26/Apr/19 07:40
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-484070424
 
 
   Run Python Load Tests Smoke
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233311)
Time Spent: 33h 10m  (was: 33h)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 33h 10m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233323&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233323
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:41
Start Date: 26/Apr/19 07:41
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486618046
 
 
   Run Python Load Tests GBK Dataflow Batch With Reiteration
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233323)
Time Spent: 35h 10m  (was: 35h)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 35h 10m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233319&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233319
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:41
Start Date: 26/Apr/19 07:41
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486608486
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233319)
Time Spent: 34.5h  (was: 34h 20m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 34.5h
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233312&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233312
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:41
Start Date: 26/Apr/19 07:41
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-485705138
 
 
   Run Python Load Tests GBK Dataflow Batch
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233312)
Time Spent: 33h 20m  (was: 33h 10m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 33h 20m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233316&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233316
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:41
Start Date: 26/Apr/19 07:41
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486597269
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233316)
Time Spent: 34h  (was: 33h 50m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 34h
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233314&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233314
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:41
Start Date: 26/Apr/19 07:41
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486186925
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233314)
Time Spent: 33h 40m  (was: 33.5h)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 33h 40m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233321&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233321
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:41
Start Date: 26/Apr/19 07:41
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486614136
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233321)
Time Spent: 34h 50m  (was: 34h 40m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 34h 50m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233315&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233315
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:41
Start Date: 26/Apr/19 07:41
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486189203
 
 
   Run Python Load Tests GBK Dataflow Batch
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233315)
Time Spent: 33h 50m  (was: 33h 40m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 33h 50m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233313&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233313
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:41
Start Date: 26/Apr/19 07:41
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486181552
 
 
   Run Python Load Tests GBK Dataflow Batch With Reiteration
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233313)
Time Spent: 33.5h  (was: 33h 20m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 33.5h
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233320&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233320
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:41
Start Date: 26/Apr/19 07:41
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486610832
 
 
   Run Python Load Tests GBK Dataflow Batch With Reiteration
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233320)
Time Spent: 34h 40m  (was: 34.5h)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 34h 40m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233322&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233322
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:41
Start Date: 26/Apr/19 07:41
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486614330
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233322)
Time Spent: 35h  (was: 34h 50m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 35h
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233325&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233325
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:42
Start Date: 26/Apr/19 07:42
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486621016
 
 
   Run Python Load Tests GBK Dataflow Batch With Reiteration
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233325)
Time Spent: 35.5h  (was: 35h 20m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 35.5h
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233318&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233318
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:41
Start Date: 26/Apr/19 07:41
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486601138
 
 
   Run Python Load Tests GBK Dataflow Batch
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233318)
Time Spent: 34h 20m  (was: 34h 10m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 34h 20m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233317&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233317
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:41
Start Date: 26/Apr/19 07:41
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-48662
 
 
   Run Python Load Tests GBK Dataflow Batch With Reiteration
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233317)
Time Spent: 34h 10m  (was: 34h)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 34h 10m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233324&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233324
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:41
Start Date: 26/Apr/19 07:41
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486619080
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233324)
Time Spent: 35h 20m  (was: 35h 10m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 35h 20m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=25&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-25
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:42
Start Date: 26/Apr/19 07:42
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486949531
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 25)
Time Spent: 37h  (was: 36h 50m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 37h
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=23&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-23
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:42
Start Date: 26/Apr/19 07:42
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486946186
 
 
   Run Python Load Tests Smoke
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 23)
Time Spent: 36h 40m  (was: 36.5h)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 36h 40m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=24&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-24
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:42
Start Date: 26/Apr/19 07:42
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486949419
 
 
   Run Python Load Tests GBK Dataflow Batch With Reiteration
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 24)
Time Spent: 36h 50m  (was: 36h 40m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 36h 50m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=20&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-20
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:42
Start Date: 26/Apr/19 07:42
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486652321
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 20)
Time Spent: 36h 10m  (was: 36h)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 36h 10m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233326&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233326
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:42
Start Date: 26/Apr/19 07:42
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486633089
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233326)
Time Spent: 35h 40m  (was: 35.5h)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 35h 40m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=22&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-22
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:42
Start Date: 26/Apr/19 07:42
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486944634
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 22)
Time Spent: 36.5h  (was: 36h 20m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 36.5h
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233327&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233327
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:42
Start Date: 26/Apr/19 07:42
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486636642
 
 
   Run Python Load Tests Smoke
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233327)
Time Spent: 35h 50m  (was: 35h 40m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 35h 50m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=26&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-26
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:42
Start Date: 26/Apr/19 07:42
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486953699
 
 
   Run Python Load Tests GBK Dataflow Batch With Reiteration
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 26)
Time Spent: 37h 10m  (was: 37h)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 37h 10m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233329&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233329
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:42
Start Date: 26/Apr/19 07:42
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486640196
 
 
   Run Python Load Tests Smoke
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233329)
Time Spent: 36h  (was: 35h 50m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 36h
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=233328&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233328
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:42
Start Date: 26/Apr/19 07:42
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486638344
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233328)
Time Spent: 36h  (was: 35h 50m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 36h
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=21&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-21
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:42
Start Date: 26/Apr/19 07:42
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486658034
 
 
   Run Python Load Tests Smoke
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 21)
Time Spent: 36h 20m  (was: 36h 10m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 36h 20m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5995) Create Jenkins jobs to run the load tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5995?focusedWorklogId=28&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-28
 ]

ASF GitHub Bot logged work on BEAM-5995:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:46
Start Date: 26/Apr/19 07:46
Worklog Time Spent: 10m 
  Work Description: kkucharc commented on issue #8151: [BEAM-5995] add 
Jenkins job with GBK Python load tests
URL: https://github.com/apache/beam/pull/8151#issuecomment-486961766
 
 
   Run Python Load Tests GBK Dataflow Batch With Reiteration
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 28)
Time Spent: 37h 20m  (was: 37h 10m)

> Create Jenkins jobs to run the load tests
> -
>
> Key: BEAM-5995
> URL: https://issues.apache.org/jira/browse/BEAM-5995
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kasia Kucharczyk
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Labels: triaged
>  Time Spent: 37h 20m
>  Remaining Estimate: 0h
>
> (/) Add SMOKE test 
>  Add GBK load tests.
> Add CoGBK load tests.
> Add Pardo load tests.
> Add SideInput tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6627) Use Metrics API in IO performance tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6627?focusedWorklogId=233343&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233343
 ]

ASF GitHub Bot logged work on BEAM-6627:


Author: ASF GitHub Bot
Created on: 26/Apr/19 07:55
Start Date: 26/Apr/19 07:55
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #8400: [BEAM-6627] Added 
byte and item counting metrics to integration tests
URL: https://github.com/apache/beam/pull/8400#issuecomment-486964357
 
 
   Run Java Precommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233343)
Time Spent: 11h  (was: 10h 50m)

> Use Metrics API in IO performance tests
> ---
>
> Key: BEAM-6627
> URL: https://issues.apache.org/jira/browse/BEAM-6627
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 11h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-7156) Nexmark Query 14 'SESSION_SIDE_INPUT_JOIN' is producing twice the number of results in Spark Runner

[JIRA]

Ismaël Mejía created BEAM-7156:
--

 Summary: Nexmark Query 14 'SESSION_SIDE_INPUT_JOIN' is producing 
twice the number of results in Spark Runner
 Key: BEAM-7156
 URL: https://issues.apache.org/jira/browse/BEAM-7156
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Ismaël Mejía


Running the same query with Direct runner and Flink produces 92000 results for 
the default config but in Spark it is producing twice as much 184000



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7043) Add DynamoDBIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7043?focusedWorklogId=233347&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233347
 ]

ASF GitHub Bot logged work on BEAM-7043:


Author: ASF GitHub Bot
Created on: 26/Apr/19 08:25
Start Date: 26/Apr/19 08:25
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #8390:  [BEAM-7043] Add 
DynamoDBIO
URL: https://github.com/apache/beam/pull/8390#issuecomment-486973154
 
 
   @cmachgodaddy You don't need everyone to review your PR. We commonly do 
reviews by one or two people, in particular people who maintain this area of 
the code, so in this case probably a second better candidate for reviewer would 
be @aromanenko-dev.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233347)
Time Spent: 3.5h  (was: 3h 20m)

> Add DynamoDBIO
> --
>
> Key: BEAM-7043
> URL: https://issues.apache.org/jira/browse/BEAM-7043
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-aws
>Reporter: Cam Mach
>Assignee: Cam Mach
>Priority: Minor
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Currently we don't have any feature to write data to AWS DynamoDB. This 
> feature will enable us to send data to DynamoDB



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233362&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233362
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:00
Start Date: 26/Apr/19 09:00
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-486983635
 
 
   Run Java PortabilityApi PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233362)
Time Spent: 11h 40m  (was: 11.5h)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 11h 40m
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233364&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233364
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:01
Start Date: 26/Apr/19 09:01
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-486983903
 
 
   Run Java PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233364)
Time Spent: 12h  (was: 11h 50m)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 12h
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233363&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233363
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:01
Start Date: 26/Apr/19 09:01
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-486983721
 
 
   Run Python PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233363)
Time Spent: 11h 50m  (was: 11h 40m)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 11h 50m
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-7157) Allow creation of BinaryCombineFn from lambdas.

[Robert Bradshaw (JIRA)]

Robert Bradshaw created BEAM-7157:
-

 Summary: Allow creation of BinaryCombineFn from lambdas.
 Key: BEAM-7157
 URL: https://issues.apache.org/jira/browse/BEAM-7157
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-core
Reporter: Robert Bradshaw






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6619) Add PostCommit suite for integration tests on DataflowRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6619?focusedWorklogId=233365&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233365
 ]

ASF GitHub Bot logged work on BEAM-6619:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:03
Start Date: 26/Apr/19 09:03
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on issue #7872: [BEAM-6619] 
[BEAM-6593] Add more integration tests to postcommit
URL: https://github.com/apache/beam/pull/7872#issuecomment-486984371
 
 
   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 1 week if no further activity occurs. If you think that’s 
incorrect or this pull request requires a review, please simply write any 
comment. If closed, you can revive the PR at any time and @mention a reviewer 
or discuss it on the d...@beam.apache.org list. Thank you for your 
contributions.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233365)
Time Spent: 26h  (was: 25h 50m)

> Add PostCommit suite for integration tests on DataflowRunner
> 
>
> Key: BEAM-6619
> URL: https://issues.apache.org/jira/browse/BEAM-6619
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Labels: triaged
> Fix For: Not applicable
>
>  Time Spent: 26h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7157) Allow creation of BinaryCombineFn from lambdas.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7157?focusedWorklogId=233368&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233368
 ]

ASF GitHub Bot logged work on BEAM-7157:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:04
Start Date: 26/Apr/19 09:04
Worklog Time Spent: 10m 
  Work Description: robertwb commented on pull request #8409: [BEAM-7157] 
Allow creation of BinaryCombineFn from lambdas.
URL: https://github.com/apache/beam/pull/8409
 
 
   This came up in a recent user thread.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)
 | --- | --- | ---
   
   Pre-Commit Tests Status (on master branch)
   

   
   --- |Java | Python | Go | Website
   --- | --- | --- | --- | ---
   Non-portable | [![Build 
Status](https://b

[jira] [Work logged] (BEAM-7157) Allow creation of BinaryCombineFn from lambdas.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7157?focusedWorklogId=233370&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233370
 ]

ASF GitHub Bot logged work on BEAM-7157:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:06
Start Date: 26/Apr/19 09:06
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #8409: [BEAM-7157] Allow 
creation of BinaryCombineFn from lambdas.
URL: https://github.com/apache/beam/pull/8409#issuecomment-486985534
 
 
   R: @youngoli
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233370)
Time Spent: 20m  (was: 10m)

> Allow creation of BinaryCombineFn from lambdas.
> ---
>
> Key: BEAM-7157
> URL: https://issues.apache.org/jira/browse/BEAM-7157
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Robert Bradshaw
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6627) Use Metrics API in IO performance tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6627?focusedWorklogId=233374&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233374
 ]

ASF GitHub Bot logged work on BEAM-6627:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:13
Start Date: 26/Apr/19 09:13
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #8400: [BEAM-6627] Added 
byte and item counting metrics to integration tests
URL: https://github.com/apache/beam/pull/8400#issuecomment-486987503
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233374)
Time Spent: 11.5h  (was: 11h 20m)

> Use Metrics API in IO performance tests
> ---
>
> Key: BEAM-6627
> URL: https://issues.apache.org/jira/browse/BEAM-6627
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 11.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6627) Use Metrics API in IO performance tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6627?focusedWorklogId=233373&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233373
 ]

ASF GitHub Bot logged work on BEAM-6627:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:13
Start Date: 26/Apr/19 09:13
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #8400: [BEAM-6627] Added 
byte and item counting metrics to integration tests
URL: https://github.com/apache/beam/pull/8400#issuecomment-486954187
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233373)
Time Spent: 11h 20m  (was: 11h 10m)

> Use Metrics API in IO performance tests
> ---
>
> Key: BEAM-6627
> URL: https://issues.apache.org/jira/browse/BEAM-6627
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 11h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6627) Use Metrics API in IO performance tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6627?focusedWorklogId=233372&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233372
 ]

ASF GitHub Bot logged work on BEAM-6627:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:13
Start Date: 26/Apr/19 09:13
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #8400: [BEAM-6627] Added 
byte and item counting metrics to integration tests
URL: https://github.com/apache/beam/pull/8400#issuecomment-486964357
 
 
   Run Java Precommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233372)
Time Spent: 11h 10m  (was: 11h)

> Use Metrics API in IO performance tests
> ---
>
> Key: BEAM-6627
> URL: https://issues.apache.org/jira/browse/BEAM-6627
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 11h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7112) State cleanup interferes with user timer callback

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7112?focusedWorklogId=233383&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233383
 ]

ASF GitHub Bot logged work on BEAM-7112:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:19
Start Date: 26/Apr/19 09:19
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #8399: [BEAM-7112] Timer 
race with state cleanup - take two
URL: https://github.com/apache/beam/pull/8399#discussion_r278868610
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 ##
 @@ -592,6 +557,7 @@ public SdkHarnessDoFnRunner(
   }
   this.windowCoder = windowCoder;
   this.outputQueue = new LinkedBlockingQueue<>();
+  this.cleanupWindows = new ArrayDeque<>();
 
 Review comment:
   ```suggestion
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233383)
Time Spent: 6h 50m  (was: 6h 40m)

> State cleanup interferes with user timer callback
> -
>
> Key: BEAM-7112
> URL: https://issues.apache.org/jira/browse/BEAM-7112
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.13.0
>
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> Cleanup timers and user timers are fired at the watermark. Processing of 
> timers in the SDK worker is asynchronous, so it is possible that the state is 
> already removed when the user timer callback executes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7112) State cleanup interferes with user timer callback

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7112?focusedWorklogId=233385&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233385
 ]

ASF GitHub Bot logged work on BEAM-7112:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:19
Start Date: 26/Apr/19 09:19
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #8399: [BEAM-7112] Timer 
race with state cleanup - take two
URL: https://github.com/apache/beam/pull/8399#discussion_r278864096
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 ##
 @@ -812,21 +796,35 @@ public boolean isForWindow(
 
 private final List userStateNames;
 private final Coder windowCoder;
-private final StateInternals stateInternals;
+private final ArrayDeque> cleanupQueue;
+private final Supplier keyedStateBackend;
 
-StateCleaner(List userStateNames, Coder windowCoder, 
StateInternals stateInternals) {
+StateCleaner(
+List userStateNames, Coder windowCoder, Supplier 
keyedStateBackend) {
   this.userStateNames = userStateNames;
   this.windowCoder = windowCoder;
-  this.stateInternals = stateInternals;
+  this.keyedStateBackend = keyedStateBackend;
+  this.cleanupQueue = new ArrayDeque<>();
 }
 
 @Override
 public void clearForWindow(BoundedWindow window) {
-  // Executed in the context of onTimer(..) where the correct key will be 
set
-  for (String userState : userStateNames) {
-StateNamespace namespace = StateNamespaces.window(windowCoder, window);
-BagState state = stateInternals.state(namespace, 
StateTags.bag(userState, null));
-state.clear();
+  cleanupQueue.add(KV.of(keyedStateBackend.get(), window));
 
 Review comment:
   ```suggestion
 // Executed in the context of onTimer(..) where the correct key will 
be set
 ByteBuffer key = keyedStateBackend.get();
 cleanupQueue.add(KV.of(key, window));
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233385)
Time Spent: 7h  (was: 6h 50m)

> State cleanup interferes with user timer callback
> -
>
> Key: BEAM-7112
> URL: https://issues.apache.org/jira/browse/BEAM-7112
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.13.0
>
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> Cleanup timers and user timers are fired at the watermark. Processing of 
> timers in the SDK worker is asynchronous, so it is possible that the state is 
> already removed when the user timer callback executes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7112) State cleanup interferes with user timer callback

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7112?focusedWorklogId=233381&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233381
 ]

ASF GitHub Bot logged work on BEAM-7112:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:19
Start Date: 26/Apr/19 09:19
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #8399: [BEAM-7112] Timer 
race with state cleanup - take two
URL: https://github.com/apache/beam/pull/8399#discussion_r278863048
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 ##
 @@ -812,21 +796,35 @@ public boolean isForWindow(
 
 private final List userStateNames;
 private final Coder windowCoder;
-private final StateInternals stateInternals;
+private final ArrayDeque> cleanupQueue;
 
 Review comment:
   ```suggestion
   private final Deque> cleanupQueue;
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233381)

> State cleanup interferes with user timer callback
> -
>
> Key: BEAM-7112
> URL: https://issues.apache.org/jira/browse/BEAM-7112
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.13.0
>
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> Cleanup timers and user timers are fired at the watermark. Processing of 
> timers in the SDK worker is asynchronous, so it is possible that the state is 
> already removed when the user timer callback executes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7112) State cleanup interferes with user timer callback

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7112?focusedWorklogId=233377&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233377
 ]

ASF GitHub Bot logged work on BEAM-7112:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:19
Start Date: 26/Apr/19 09:19
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #8399: [BEAM-7112] Timer 
race with state cleanup - take two
URL: https://github.com/apache/beam/pull/8399#discussion_r278861104
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 ##
 @@ -812,21 +796,35 @@ public boolean isForWindow(
 
 private final List userStateNames;
 private final Coder windowCoder;
-private final StateInternals stateInternals;
+private final ArrayDeque> cleanupQueue;
+private final Supplier keyedStateBackend;
 
-StateCleaner(List userStateNames, Coder windowCoder, 
StateInternals stateInternals) {
+StateCleaner(
+List userStateNames, Coder windowCoder, Supplier 
keyedStateBackend) {
   this.userStateNames = userStateNames;
   this.windowCoder = windowCoder;
-  this.stateInternals = stateInternals;
+  this.keyedStateBackend = keyedStateBackend;
+  this.cleanupQueue = new ArrayDeque<>();
 }
 
 @Override
 public void clearForWindow(BoundedWindow window) {
-  // Executed in the context of onTimer(..) where the correct key will be 
set
-  for (String userState : userStateNames) {
-StateNamespace namespace = StateNamespaces.window(windowCoder, window);
-BagState state = stateInternals.state(namespace, 
StateTags.bag(userState, null));
-state.clear();
+  cleanupQueue.add(KV.of(keyedStateBackend.get(), window));
+}
+
+@SuppressWarnings("ByteBufferBackingArray")
+void cleanupState(StateInternals stateInternals, Consumer 
keyedStateBackend) {
 
 Review comment:
   ```suggestion
   void cleanupState(StateInternals stateInternals, Consumer 
keyContextConsumer) {
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233377)
Time Spent: 6h  (was: 5h 50m)

> State cleanup interferes with user timer callback
> -
>
> Key: BEAM-7112
> URL: https://issues.apache.org/jira/browse/BEAM-7112
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.13.0
>
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> Cleanup timers and user timers are fired at the watermark. Processing of 
> timers in the SDK worker is asynchronous, so it is possible that the state is 
> already removed when the user timer callback executes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7112) State cleanup interferes with user timer callback

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7112?focusedWorklogId=233378&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233378
 ]

ASF GitHub Bot logged work on BEAM-7112:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:19
Start Date: 26/Apr/19 09:19
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #8399: [BEAM-7112] Timer 
race with state cleanup - take two
URL: https://github.com/apache/beam/pull/8399#discussion_r278861506
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 ##
 @@ -563,6 +527,7 @@ public void processWatermark(Watermark mark) throws 
Exception {
 
 private RemoteBundle remoteBundle;
 private FnDataReceiver> mainInputReceiver;
+private ArrayDeque> cleanupWindows;
 
 Review comment:
   ```suggestion
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233378)
Time Spent: 6h 10m  (was: 6h)

> State cleanup interferes with user timer callback
> -
>
> Key: BEAM-7112
> URL: https://issues.apache.org/jira/browse/BEAM-7112
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.13.0
>
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> Cleanup timers and user timers are fired at the watermark. Processing of 
> timers in the SDK worker is asynchronous, so it is possible that the state is 
> already removed when the user timer callback executes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7112) State cleanup interferes with user timer callback

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7112?focusedWorklogId=233382&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233382
 ]

ASF GitHub Bot logged work on BEAM-7112:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:19
Start Date: 26/Apr/19 09:19
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #8399: [BEAM-7112] Timer 
race with state cleanup - take two
URL: https://github.com/apache/beam/pull/8399#discussion_r278870101
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 ##
 @@ -812,21 +796,35 @@ public boolean isForWindow(
 
 private final List userStateNames;
 private final Coder windowCoder;
-private final StateInternals stateInternals;
+private final ArrayDeque> cleanupQueue;
+private final Supplier keyedStateBackend;
 
-StateCleaner(List userStateNames, Coder windowCoder, 
StateInternals stateInternals) {
+StateCleaner(
+List userStateNames, Coder windowCoder, Supplier 
keyedStateBackend) {
   this.userStateNames = userStateNames;
   this.windowCoder = windowCoder;
-  this.stateInternals = stateInternals;
+  this.keyedStateBackend = keyedStateBackend;
+  this.cleanupQueue = new ArrayDeque<>();
 
 Review comment:
   I think we have a consistency problem here because this is not persisted in 
checkpoints.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233382)
Time Spent: 6h 40m  (was: 6.5h)

> State cleanup interferes with user timer callback
> -
>
> Key: BEAM-7112
> URL: https://issues.apache.org/jira/browse/BEAM-7112
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.13.0
>
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> Cleanup timers and user timers are fired at the watermark. Processing of 
> timers in the SDK worker is asynchronous, so it is possible that the state is 
> already removed when the user timer callback executes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7112) State cleanup interferes with user timer callback

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7112?focusedWorklogId=233379&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233379
 ]

ASF GitHub Bot logged work on BEAM-7112:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:19
Start Date: 26/Apr/19 09:19
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #8399: [BEAM-7112] Timer 
race with state cleanup - take two
URL: https://github.com/apache/beam/pull/8399#discussion_r278862847
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 ##
 @@ -741,10 +707,28 @@ private void emitResults() {
 executableStage.getUserStates().stream()
 .map(UserStateReference::localName)
 .collect(Collectors.toList());
-StateCleaner stateCleaner = new StateCleaner(userStates, windowCoder, 
keyedStateInternals);
 
-return DoFnRunners.defaultStatefulDoFnRunner(
-doFn, sdkHarnessRunner, windowingStrategy, cleanupTimer, stateCleaner);
+KeyedStateBackend stateBackend = getKeyedStateBackend();
+StateCleaner stateCleaner =
+new StateCleaner(userStates, windowCoder, () -> 
stateBackend.getCurrentKey());
+
+return new StatefulDoFnRunner(
+sdkHarnessRunner, windowingStrategy, cleanupTimer, stateCleaner) {
+  @Override
+  public void finishBundle() {
+super.finishBundle();
 
 Review comment:
   ```suggestion
   // Before cleaning up state, first finish bundle for all underlying 
DoFnRunners
   super.finishBundle();
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233379)
Time Spent: 6h 20m  (was: 6h 10m)

> State cleanup interferes with user timer callback
> -
>
> Key: BEAM-7112
> URL: https://issues.apache.org/jira/browse/BEAM-7112
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.13.0
>
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> Cleanup timers and user timers are fired at the watermark. Processing of 
> timers in the SDK worker is asynchronous, so it is possible that the state is 
> already removed when the user timer callback executes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7112) State cleanup interferes with user timer callback

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7112?focusedWorklogId=233384&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233384
 ]

ASF GitHub Bot logged work on BEAM-7112:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:19
Start Date: 26/Apr/19 09:19
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #8399: [BEAM-7112] Timer 
race with state cleanup - take two
URL: https://github.com/apache/beam/pull/8399#discussion_r278866616
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 ##
 @@ -563,6 +527,7 @@ public void processWatermark(Watermark mark) throws 
Exception {
 
 private RemoteBundle remoteBundle;
 private FnDataReceiver> mainInputReceiver;
+private ArrayDeque> cleanupWindows;
 
 Review comment:
   Unused field.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233384)
Time Spent: 7h  (was: 6h 50m)

> State cleanup interferes with user timer callback
> -
>
> Key: BEAM-7112
> URL: https://issues.apache.org/jira/browse/BEAM-7112
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.13.0
>
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> Cleanup timers and user timers are fired at the watermark. Processing of 
> timers in the SDK worker is asynchronous, so it is possible that the state is 
> already removed when the user timer callback executes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7112) State cleanup interferes with user timer callback

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7112?focusedWorklogId=233380&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233380
 ]

ASF GitHub Bot logged work on BEAM-7112:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:19
Start Date: 26/Apr/19 09:19
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #8399: [BEAM-7112] Timer 
race with state cleanup - take two
URL: https://github.com/apache/beam/pull/8399#discussion_r278862970
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 ##
 @@ -741,10 +707,28 @@ private void emitResults() {
 executableStage.getUserStates().stream()
 .map(UserStateReference::localName)
 .collect(Collectors.toList());
-StateCleaner stateCleaner = new StateCleaner(userStates, windowCoder, 
keyedStateInternals);
 
-return DoFnRunners.defaultStatefulDoFnRunner(
-doFn, sdkHarnessRunner, windowingStrategy, cleanupTimer, stateCleaner);
+KeyedStateBackend stateBackend = getKeyedStateBackend();
+StateCleaner stateCleaner =
+new StateCleaner(userStates, windowCoder, () -> 
stateBackend.getCurrentKey());
+
+return new StatefulDoFnRunner(
+sdkHarnessRunner, windowingStrategy, cleanupTimer, stateCleaner) {
+  @Override
+  public void finishBundle() {
+super.finishBundle();
 
 Review comment:
   I like this way of ensure that the bundle has finished.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233380)
Time Spent: 6.5h  (was: 6h 20m)

> State cleanup interferes with user timer callback
> -
>
> Key: BEAM-7112
> URL: https://issues.apache.org/jira/browse/BEAM-7112
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.13.0
>
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> Cleanup timers and user timers are fired at the watermark. Processing of 
> timers in the SDK worker is asynchronous, so it is possible that the state is 
> already removed when the user timer callback executes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-7158) Operate the TensorFlow API with Apache Beam

[zhanghaitao8 (JIRA)]

zhanghaitao8 created BEAM-7158:
--

 Summary: Operate the TensorFlow API with Apache Beam
 Key: BEAM-7158
 URL: https://issues.apache.org/jira/browse/BEAM-7158
 Project: Beam
  Issue Type: Task
  Components: io-ideas
Reporter: zhanghaitao8


Use the Apache Beam to operate the TensorFlow API,
Implementation TensorFlow can be operated for machine learning through the 
Apache Beam program.
Many frameworks machine this module, but Apache Beam does not, such as Spark 
ML, Flink ML appears immediately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7138) keep Java serialized coder in length-prefixed wire coder construction

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7138?focusedWorklogId=233393&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233393
 ]

ASF GitHub Bot logged work on BEAM-7138:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:34
Start Date: 26/Apr/19 09:34
Worklog Time Spent: 10m 
  Work Description: mxm commented on issue #8396: [BEAM-7138] keep Java 
serialized coder in wire coder construction
URL: https://github.com/apache/beam/pull/8396#issuecomment-486994958
 
 
   Thanks for explaining. Good find. Looks like this breaks the Dataflow tests 
now.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233393)
Time Spent: 1h  (was: 50m)

> keep Java serialized coder in length-prefixed wire coder construction
> -
>
> Key: BEAM-7138
> URL: https://issues.apache.org/jira/browse/BEAM-7138
> Project: Beam
>  Issue Type: Improvement
>  Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> don't replace Java serialized coder with byte array coder in length-prefixed 
> wire coder construction.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7012) Support TestStream in FlinkRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7012?focusedWorklogId=233395&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233395
 ]

ASF GitHub Bot logged work on BEAM-7012:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:40
Start Date: 26/Apr/19 09:40
Worklog Time Spent: 10m 
  Work Description: mxm commented on issue #8383: [BEAM-7012] Support 
TestStream in streaming Flink Runner
URL: https://github.com/apache/beam/pull/8383#issuecomment-486996707
 
 
   I just came across this and wondered why we hadn't implemented this. Turns 
out, this was not hard to implement for streaming. Well, minus the processing 
time functionality for which I couldn't find a good solution, so I excluded two 
tests which make use of that feature. I'm planning to also add TestStream for 
the portable Runner.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233395)
Time Spent: 2.5h  (was: 2h 20m)

> Support TestStream in FlinkRunner
> -
>
> Key: BEAM-7012
> URL: https://issues.apache.org/jira/browse/BEAM-7012
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.13.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> TestStream is a primitive transform which is only supported by the 
> DirectRunner. It might be useful to also implement it in the Flink Runner to 
> run similar kind of tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-7159) Add cluster monitoring BI to Apache Beam

[zhanghaitao8 (JIRA)]

zhanghaitao8 created BEAM-7159:
--

 Summary: Add cluster monitoring BI to Apache Beam
 Key: BEAM-7159
 URL: https://issues.apache.org/jira/browse/BEAM-7159
 Project: Beam
  Issue Type: New Feature
  Components: io-ideas
Reporter: zhanghaitao8


Add cluster monitoring BI to Apache Beam for monitoring Beam pipelines, 
monitoring data, and monitoring speed. You can try out various operating 
platforms.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6627) Use Metrics API in IO performance tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6627?focusedWorklogId=233409&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233409
 ]

ASF GitHub Bot logged work on BEAM-6627:


Author: ASF GitHub Bot
Created on: 26/Apr/19 09:59
Start Date: 26/Apr/19 09:59
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #8400: [BEAM-6627] Added 
byte and item counting metrics to integration tests
URL: https://github.com/apache/beam/pull/8400#issuecomment-487002403
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233409)
Time Spent: 11h 40m  (was: 11.5h)

> Use Metrics API in IO performance tests
> ---
>
> Key: BEAM-6627
> URL: https://issues.apache.org/jira/browse/BEAM-6627
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 11h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7029) Support KafkaIO to be configured externally for use with other SDKs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7029?focusedWorklogId=233411&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233411
 ]

ASF GitHub Bot logged work on BEAM-7029:


Author: ASF GitHub Bot
Created on: 26/Apr/19 10:02
Start Date: 26/Apr/19 10:02
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #8322: [BEAM-7029] Add 
KafkaIO.Write as an external transform
URL: https://github.com/apache/beam/pull/8322
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233411)
Time Spent: 13.5h  (was: 13h 20m)

> Support KafkaIO to be configured externally for use with other SDKs
> ---
>
> Key: BEAM-7029
> URL: https://issues.apache.org/jira/browse/BEAM-7029
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka, runner-flink, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 13.5h
>  Remaining Estimate: 0h
>
> As of BEAM-6730, we can externally configure existing transforms from SDKs. 
> We should add more useful transforms then just {{GenerateSequence}}. 
> {{KafkaIO}} is a good candidate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7029) Support KafkaIO to be configured externally for use with other SDKs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7029?focusedWorklogId=233410&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233410
 ]

ASF GitHub Bot logged work on BEAM-7029:


Author: ASF GitHub Bot
Created on: 26/Apr/19 10:02
Start Date: 26/Apr/19 10:02
Worklog Time Spent: 10m 
  Work Description: mxm commented on issue #8322: [BEAM-7029] Add 
KafkaIO.Write as an external transform
URL: https://github.com/apache/beam/pull/8322#issuecomment-487003064
 
 
   Thanks for all the comments!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233410)
Time Spent: 13h 20m  (was: 13h 10m)

> Support KafkaIO to be configured externally for use with other SDKs
> ---
>
> Key: BEAM-7029
> URL: https://issues.apache.org/jira/browse/BEAM-7029
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka, runner-flink, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 13h 20m
>  Remaining Estimate: 0h
>
> As of BEAM-6730, we can externally configure existing transforms from SDKs. 
> We should add more useful transforms then just {{GenerateSequence}}. 
> {{KafkaIO}} is a good candidate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-7029) Support KafkaIO to be configured externally for use with other SDKs

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels closed BEAM-7029.

   Resolution: Fixed
Fix Version/s: 2.13.0

> Support KafkaIO to be configured externally for use with other SDKs
> ---
>
> Key: BEAM-7029
> URL: https://issues.apache.org/jira/browse/BEAM-7029
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-kafka, runner-flink, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.13.0
>
>  Time Spent: 13.5h
>  Remaining Estimate: 0h
>
> As of BEAM-6730, we can externally configure existing transforms from SDKs. 
> We should add more useful transforms then just {{GenerateSequence}}. 
> {{KafkaIO}} is a good candidate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233421&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233421
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 10:18
Start Date: 26/Apr/19 10:18
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-487007734
 
 
   Run Python PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233421)
Time Spent: 12h 10m  (was: 12h)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 12h 10m
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233423&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233423
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 10:20
Start Date: 26/Apr/19 10:20
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-487008089
 
 
   Run SQL PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233423)
Time Spent: 12h 20m  (was: 12h 10m)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 12h 20m
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-7160) Expand ValidatesRunner task to use NeedsRunner test category

[Maximilian Michels (JIRA)]

Maximilian Michels created BEAM-7160:


 Summary: Expand ValidatesRunner task to use NeedsRunner test 
category
 Key: BEAM-7160
 URL: https://issues.apache.org/jira/browse/BEAM-7160
 Project: Beam
  Issue Type: Test
  Components: runner-flink
Reporter: Maximilian Michels


The {{:validatesRunner}} task uses the {{ValidatesRunner}} test category which 
is a subtype of {{NeedsRunner}}. It would be good to expand the scope of the 
tests to {{NeedsRunner}} because there are many additional tests which 
currently excluded.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5775) Make the spark runner not serialize data unless spark is spilling to disk

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5775?focusedWorklogId=233437&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233437
 ]

ASF GitHub Bot logged work on BEAM-5775:


Author: ASF GitHub Bot
Created on: 26/Apr/19 10:47
Start Date: 26/Apr/19 10:47
Worklog Time Spent: 10m 
  Work Description: VaclavPlajt commented on pull request #6714: 
[BEAM-5775] Spark: implement a custom class to lazily encode values for 
persistence.
URL: https://github.com/apache/beam/pull/6714#discussion_r278898397
 
 

 ##
 File path: 
runners/spark/src/main/java/org/apache/beam/runners/spark/translation/ValueAndCoderKryoSerializable.java
 ##
 @@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.spark.translation;
+
+import com.esotericsoftware.kryo.Kryo;
+import com.esotericsoftware.kryo.KryoSerializable;
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import com.google.common.base.Throwables;
+import com.google.common.io.ByteStreams;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.Externalizable;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+import java.io.OutputStream;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.util.VarInt;
+import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
+
+/**
+ * A holder object that lets you serialize an element with a Coder with 
minimal wasted space.
+ * Supports both Kryo and Java serialization.
+ *
+ * There are two different representations: a deserialized representation 
and a serialized
+ * representation.
+ *
+ * The deserialized representation stores a Coder and the value. To 
serialize the value, we write
+ * a length-prefixed encoding of value, but do NOT write the Coder used.
+ *
+ * The serialized representation just reads a byte array - the value is not 
deserialized fully.
+ * In order to get at the deserialized value, the caller must pass the Coder 
used to create this
+ * instance via getOrDecode(Coder). This reverts the representation back to 
the deserialized
+ * representation.
+ *
+ * @param  element type
+ */
+public class ValueAndCoderKryoSerializable implements KryoSerializable, 
Externalizable {
+  private T value;
+  // Re-use a field to save space in-memory. This is either a byte[] or a 
Coder, depending on
+  // which representation we are in.
+  private Object coderOrBytes;
+
+  ValueAndCoderKryoSerializable(T value, Coder currentCoder) {
+this.value = value;
+this.coderOrBytes = currentCoder;
+  }
+
+  @SuppressWarnings("unused") // for serialization
+  public ValueAndCoderKryoSerializable() {}
+
+  public T getOrDecode(Coder coder) throws IOException {
+if (!(coderOrBytes instanceof Coder)) {
+  value =
+  coder.decode(new ByteArrayInputStream((byte[]) this.coderOrBytes), 
Coder.Context.OUTER);
+  this.coderOrBytes = coder;
+}
+
+return value;
+  }
+
+  private static class ByteSizeObserver extends ElementByteSizeObserver {
+private long observedSize = 0;
+
+@Override
+protected void reportElementSize(long elementByteSize) {
+  observedSize += elementByteSize;
+}
+  }
+
+  private void writeCommon(OutputStream out) throws IOException {
+if (!(coderOrBytes instanceof Coder)) {
+  byte[] bytes = (byte[]) coderOrBytes;
+  VarInt.encode(bytes.length, out);
+  out.write(bytes);
+} else {
+  @SuppressWarnings("unchecked")
+  Coder coder = (Coder) coderOrBytes;
+  int bufferSize = 1024;
+
+  if (coder.isRegisterByteSizeObserverCheap(value)) {
+try {
+  ByteSizeObserver observer = new ByteSizeObserver();
+  coder.registerByteSizeObserver(value, observer);
+  bufferSize = (int) observer.observedSize;
+} catch (Exception e) {
+  Throwables.throwIfUnchecked(e);
+  throw new RuntimeException(e);
+}
+  }
+
+  ByteArrayOutputStream bytes = new ByteArrayOutputStream(bufferSize);
+
+ 

[jira] [Work logged] (BEAM-5775) Make the spark runner not serialize data unless spark is spilling to disk

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5775?focusedWorklogId=233440&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233440
 ]

ASF GitHub Bot logged work on BEAM-5775:


Author: ASF GitHub Bot
Created on: 26/Apr/19 10:52
Start Date: 26/Apr/19 10:52
Worklog Time Spent: 10m 
  Work Description: VaclavPlajt commented on pull request #6714: 
[BEAM-5775] Spark: implement a custom class to lazily encode values for 
persistence.
URL: https://github.com/apache/beam/pull/6714#discussion_r278900074
 
 

 ##
 File path: 
runners/spark/src/main/java/org/apache/beam/runners/spark/coders/ValueAndCoderLazySerializable.java
 ##
 @@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.spark.coders;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.OutputStream;
+import java.io.Serializable;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.util.VarInt;
+import org.apache.beam.sdk.util.common.ElementByteSizeObserver;
+import org.apache.beam.vendor.guava.v20_0.com.google.common.base.Throwables;
+import org.apache.beam.vendor.guava.v20_0.com.google.common.io.ByteStreams;
+
+/**
+ * A holder object that lets you serialize an element with a Coder with 
minimal wasted space.
+ * Supports both Kryo and Java serialization.
+ *
+ * There are two different representations: a deserialized representation 
and a serialized
+ * representation.
+ *
+ * The deserialized representation stores a Coder and the value. To 
serialize the value, we write
+ * a length-prefixed encoding of value, but do NOT write the Coder used.
+ *
+ * The serialized representation just reads a byte array - the value is not 
deserialized fully.
+ * In order to get at the deserialized value, the caller must pass the Coder 
used to create this
+ * instance via getOrDecode(Coder). This reverts the representation back to 
the deserialized
+ * representation.
+ *
+ * @param  element type
+ */
+public final class ValueAndCoderLazySerializable implements Serializable {
+  private T value;
+  // Re-use a field to save space in-memory. This is either a byte[] or a 
Coder, depending on
+  // which representation we are in.
+  private Object coderOrBytes;
+
+  private ValueAndCoderLazySerializable(T value, Coder currentCoder) {
+this.value = value;
+this.coderOrBytes = currentCoder;
+  }
+
+  @SuppressWarnings("unused") // for serialization
+  ValueAndCoderLazySerializable() {}
+
+  public static  ValueAndCoderLazySerializable of(T value, Coder 
coder) {
+return new ValueAndCoderLazySerializable<>(value, coder);
+  }
+
+  public T getOrDecode(Coder coder) {
+if (!(coderOrBytes instanceof Coder)) {
+  ByteArrayInputStream bais = new ByteArrayInputStream((byte[]) 
this.coderOrBytes);
+  try {
+value = coder.decode(bais);
+  } catch (IOException e) {
+throw new IllegalStateException("Error decoding bytes for coder: " + 
coder, e);
+  }
+  this.coderOrBytes = coder;
+}
+
+return value;
+  }
+
+  private static class ByteSizeObserver extends ElementByteSizeObserver {
+private long observedSize = 0;
+
+@Override
+protected void reportElementSize(long elementByteSize) {
+  observedSize += elementByteSize;
+}
+  }
+
+  void writeCommon(OutputStream out) throws IOException {
+if (!(coderOrBytes instanceof Coder)) {
+  byte[] bytes = (byte[]) coderOrBytes;
+  VarInt.encode(bytes.length, out);
+  out.write(bytes);
+} else {
+  @SuppressWarnings("unchecked")
+  Coder coder = (Coder) coderOrBytes;
+  int bufferSize = 1024;
+
+  if (coder.isRegisterByteSizeObserverCheap(value)) {
+try {
+  ByteSizeObserver observer = new ByteSizeObserver();
+  coder.registerByteSizeObserver(value, observer);
 
 Review comment:
   :clap: 
 

This is an automated messa

[jira] [Commented] (BEAM-7137) TypeError caused by using str variable as header argument in apache_beam.io.textio.WriteToText

[yoshiki obata (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16826884#comment-16826884
 ] 

yoshiki obata commented on BEAM-7137:
-

[~tvalentyn] I'd like to work on! This issue bothers me.
I also agree header should be string in aspect of smoothly migration to Python3 
from Python2.

And found that beam.io.textio_test.test_write_dataflow_header uses bytes header 
with beam.io.textio.WriteToText, is this affect to problem in postcommit 
integration tests?:
[https://github.com/apache/beam/blob/563fa3772d7a044bf0c28baf28fb77a3b58800d6/sdks/python/apache_beam/io/textio_test.py#L1152|https://github.com/apache/beam/blob/563fa3772d7a044bf0c28baf28fb77a3b58800d6/sdks/python/apache_beam/io/textio_test.py#L1152]

> TypeError caused by using str variable as header argument in 
> apache_beam.io.textio.WriteToText
> --
>
> Key: BEAM-7137
> URL: https://issues.apache.org/jira/browse/BEAM-7137
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Affects Versions: 2.11.0
> Environment: Python 3.5.6
> macOS Mojave 10.14.4
>Reporter: yoshiki obata
>Assignee: yoshiki obata
>Priority: Major
>
> Using str header to apache_beam.io.textio.WriteToText as argument cause 
> TypeError with Python 3.5.6 - or maybe higher - despite docstring says header 
> is str.
>  This error occurred by writing header to file without encoding to bytes at 
> apache_beam.io.textio._TextSink.open.
>  
> {code:java}
> Traceback (most recent call last):
>   File "apache_beam/runners/common.py", line 727, in 
> apache_beam.runners.common.DoFnRunner.process
>   File "apache_beam/runners/common.py", line 555, in 
> apache_beam.runners.common.PerWindowInvoker.invoke_process
>   File "apache_beam/runners/common.py", line 625, in 
> apache_beam.runners.common.PerWindowInvoker._invoke_per_window
>   File 
> "/Users/yob/.local/share/virtualenvs/test/lib/python3.5/site-packages/apache_beam/io/iobase.py",
>  line 1033, in process
>     self.writer = self.sink.open_writer(init_result, str(uuid.uuid4()))
>   File 
> "/Users/yob/.local/share/virtualenvs/test/lib/python3.5/site-packages/apache_beam/options/value_provider.py",
>  line 137, in _f
>     return fnc(self, *args, **kwargs)
>   File 
> "/Users/yob/.local/share/virtualenvs/test/lib/python3.5/site-packages/apache_beam/io/filebasedsink.py",
>  line 185, in open_writer
>     return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
>   File 
> "/Users/yob/.local/share/virtualenvs/test/lib/python3.5/site-packages/apache_beam/io/filebasedsink.py",
>  line 389, in __init__
>     self.temp_handle = self.sink.open(temp_shard_path)
>   File 
> "/Users/yob/.local/share/virtualenvs/test/lib/python3.5/site-packages/apache_beam/io/textio.py",
>  line 393, in open
>     file_handle.write(self._header)
> TypeError: a bytes-like object is required, not 'str'
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233443&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233443
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 11:06
Start Date: 26/Apr/19 11:06
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-487019467
 
 
   Run Python PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233443)
Time Spent: 12h 40m  (was: 12.5h)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 12h 40m
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233442&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233442
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 11:06
Start Date: 26/Apr/19 11:06
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-487019411
 
 
   Run SQL PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233442)
Time Spent: 12.5h  (was: 12h 20m)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 12.5h
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233444&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233444
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 11:06
Start Date: 26/Apr/19 11:06
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-487019515
 
 
   Run Java PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233444)
Time Spent: 12h 50m  (was: 12h 40m)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 12h 50m
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7012) Support TestStream in FlinkRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7012?focusedWorklogId=233447&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233447
 ]

ASF GitHub Bot logged work on BEAM-7012:


Author: ASF GitHub Bot
Created on: 26/Apr/19 11:20
Start Date: 26/Apr/19 11:20
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #8411: [BEAM-7012] 
Support TestStream in portable Flink Runner
URL: https://github.com/apache/beam/pull/8411
 
 
   With #8383 we added support for TestStream in the legacy streaming Flink
   Runner. This ports TestStream also to the streaming portable Flink Runner. In
   the course of this a coder issue was discovered which has been addressed in 
#8396.
   This work is based on that commit.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)
 | --- | --- | ---
   
   Pre-Commit Tests Status (on m

[jira] [Work logged] (BEAM-7012) Support TestStream in FlinkRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7012?focusedWorklogId=233450&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233450
 ]

ASF GitHub Bot logged work on BEAM-7012:


Author: ASF GitHub Bot
Created on: 26/Apr/19 11:21
Start Date: 26/Apr/19 11:21
Worklog Time Spent: 10m 
  Work Description: mxm commented on issue #8411: [BEAM-7012] Support 
TestStream in portable Flink Runner
URL: https://github.com/apache/beam/pull/8411#issuecomment-487022876
 
 
   Run Flink ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233450)
Time Spent: 3h  (was: 2h 50m)

> Support TestStream in FlinkRunner
> -
>
> Key: BEAM-7012
> URL: https://issues.apache.org/jira/browse/BEAM-7012
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.13.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> TestStream is a primitive transform which is only supported by the 
> DirectRunner. It might be useful to also implement it in the Flink Runner to 
> run similar kind of tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7012) Support TestStream in FlinkRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7012?focusedWorklogId=233449&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233449
 ]

ASF GitHub Bot logged work on BEAM-7012:


Author: ASF GitHub Bot
Created on: 26/Apr/19 11:21
Start Date: 26/Apr/19 11:21
Worklog Time Spent: 10m 
  Work Description: mxm commented on issue #8411: [BEAM-7012] Support 
TestStream in portable Flink Runner
URL: https://github.com/apache/beam/pull/8411#issuecomment-487022853
 
 
   Run Java Flink PortableValidatesRunner Streaming
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233449)
Time Spent: 2h 50m  (was: 2h 40m)

> Support TestStream in FlinkRunner
> -
>
> Key: BEAM-7012
> URL: https://issues.apache.org/jira/browse/BEAM-7012
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.13.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> TestStream is a primitive transform which is only supported by the 
> DirectRunner. It might be useful to also implement it in the Flink Runner to 
> run similar kind of tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233452&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233452
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 11:29
Start Date: 26/Apr/19 11:29
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-487024682
 
 
   Run Java PortabilityApi PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233452)
Time Spent: 13h  (was: 12h 50m)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 13h
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7157) Allow creation of BinaryCombineFn from lambdas.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7157?focusedWorklogId=233465&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233465
 ]

ASF GitHub Bot logged work on BEAM-7157:


Author: ASF GitHub Bot
Created on: 26/Apr/19 11:53
Start Date: 26/Apr/19 11:53
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #8409: [BEAM-7157] Allow 
creation of BinaryCombineFn from lambdas.
URL: https://github.com/apache/beam/pull/8409#issuecomment-487030407
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233465)
Time Spent: 0.5h  (was: 20m)

> Allow creation of BinaryCombineFn from lambdas.
> ---
>
> Key: BEAM-7157
> URL: https://issues.apache.org/jira/browse/BEAM-7157
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Robert Bradshaw
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-7161) filter.Distinct should accept a KV type

[Damien Desfontaines (JIRA)]

Damien Desfontaines created BEAM-7161:
-

 Summary: filter.Distinct should accept a KV type
 Key: BEAM-7161
 URL: https://issues.apache.org/jira/browse/BEAM-7161
 Project: Beam
  Issue Type: Improvement
  Components: sdk-go
Reporter: Damien Desfontaines


filter.Distinct only works on PCollection. It would be nice if it worked for 
PCollection> too (returning distinct KV pairs).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2939) Fn API SDF support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2939?focusedWorklogId=233482&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233482
 ]

ASF GitHub Bot logged work on BEAM-2939:


Author: ASF GitHub Bot
Created on: 26/Apr/19 12:48
Start Date: 26/Apr/19 12:48
Worklog Time Spent: 10m 
  Work Description: robertwb commented on pull request #8412: [BEAM-2939] 
Support SDF expansion in the Flink runner.
URL: https://github.com/apache/beam/pull/8412
 
 
   SDFs now expand to Split + Reshard + Read. Informing the SDF about the 
desired number of splits is still TBD. 
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)
 | --- | --- | ---
   
   Pre-Commit Tests Status (on master branch)
   

   
   --- |Java | Python | Go | Website
   --- 

[jira] [Work logged] (BEAM-7012) Support TestStream in FlinkRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7012?focusedWorklogId=233488&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233488
 ]

ASF GitHub Bot logged work on BEAM-7012:


Author: ASF GitHub Bot
Created on: 26/Apr/19 13:03
Start Date: 26/Apr/19 13:03
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #8383: [BEAM-7012] Support 
TestStream in streaming Flink Runner
URL: https://github.com/apache/beam/pull/8383#issuecomment-487048779
 
 
   Very cool.
   
   One of the things that TestStream needs to ensure determinism is the notion
   of quiescence, that is between each event one needs to ensure that the
   downstream elements and watermarks are processed as completely as possible.
   Is this guaranteed by the checkpoint lock? If so, it's worth at least a
   couple of comments.
   
   On Fri, Apr 26, 2019 at 11:40 AM Maximilian Michels <
   notificati...@github.com> wrote:
   
   > I just came across this and wondered why we hadn't implemented this. Turns
   > out, this was not hard to implement for streaming. Well, minus the
   > processing time functionality for which I couldn't find a good solution, so
   > I excluded two tests which make use of that feature. I'm planning to also
   > add TestStream for the portable Runner.
   >
   > —
   > You are receiving this because you are subscribed to this thread.
   > Reply to this email directly, view it on GitHub
   > , or mute
   > the thread
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233488)
Time Spent: 3h 10m  (was: 3h)

> Support TestStream in FlinkRunner
> -
>
> Key: BEAM-7012
> URL: https://issues.apache.org/jira/browse/BEAM-7012
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.13.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> TestStream is a primitive transform which is only supported by the 
> DirectRunner. It might be useful to also implement it in the Flink Runner to 
> run similar kind of tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233497&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233497
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 13:13
Start Date: 26/Apr/19 13:13
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-487051901
 
 
   Run Java Flink PortableValidatesRunner Streaming
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233497)
Time Spent: 13h 10m  (was: 13h)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 13h 10m
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233498&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233498
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 13:13
Start Date: 26/Apr/19 13:13
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-487051980
 
 
   Run Dataflow PortabilityApi ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233498)
Time Spent: 13h 20m  (was: 13h 10m)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 13h 20m
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233500&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233500
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 13:14
Start Date: 26/Apr/19 13:14
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-487052358
 
 
   Run Python Flink ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233500)
Time Spent: 13h 40m  (was: 13.5h)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 13h 40m
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233499&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233499
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 13:14
Start Date: 26/Apr/19 13:14
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-487052108
 
 
   Run Dataflow ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233499)
Time Spent: 13.5h  (was: 13h 20m)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 13.5h
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4046) Decouple gradle project names and maven artifact ids

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4046?focusedWorklogId=233503&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233503
 ]

ASF GitHub Bot logged work on BEAM-4046:


Author: ASF GitHub Bot
Created on: 26/Apr/19 13:18
Start Date: 26/Apr/19 13:18
Worklog Time Spent: 10m 
  Work Description: adude3141 commented on issue #8194: [DO NOT MERGE] 
[BEAM-4046] decouple gradle project names and maven artifact ids
URL: https://github.com/apache/beam/pull/8194#issuecomment-487053347
 
 
   Run Python Dataflow ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233503)
Time Spent: 13h 50m  (was: 13h 40m)

> Decouple gradle project names and maven artifact ids
> 
>
> Key: BEAM-4046
> URL: https://issues.apache.org/jira/browse/BEAM-4046
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
>  Time Spent: 13h 50m
>  Remaining Estimate: 0h
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. 
> It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, 
> like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact 
> ids right.
> These should be able to be decoupled. It seems there are many StackOverflow 
> questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be 
> mandatory then it needs to be documented inline everywhere - the 
> settings.gradle should say why it is so bizarre, and each build.gradle should 
> indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-7162) Cassandra ValueProvider / Write Operations

[Mathieu Blanchard (JIRA)]

Mathieu Blanchard created BEAM-7162:
---

 Summary: Cassandra ValueProvider / Write Operations
 Key: BEAM-7162
 URL: https://issues.apache.org/jira/browse/BEAM-7162
 Project: Beam
  Issue Type: Improvement
  Components: io-java-cassandra
Affects Versions: 2.12.0
Reporter: Mathieu Blanchard
Assignee: Mathieu Blanchard


Currently, the ValueProvider type is only available for the Read.builder() 
attributes and we need it for some write operations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7162) Cassandra ValueProvider / Write Operations

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7162?focusedWorklogId=233501&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233501
 ]

ASF GitHub Bot logged work on BEAM-7162:


Author: ASF GitHub Bot
Created on: 26/Apr/19 13:15
Start Date: 26/Apr/19 13:15
Worklog Time Spent: 10m 
  Work Description: mblmat commented on pull request #8413: [BEAM-7162] 
Cassandra ValueProvider / Write Operations
URL: https://github.com/apache/beam/pull/8413
 
 
   Currently, the ValueProvider type is only available for the Read.builder() 
attributes and we need it for some write operations.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)
 | --- | --- | ---
   
   Pre-Commit Tests Status (on master branch)
   

   
   --- |Java | Python | Go | 

[jira] [Work logged] (BEAM-7162) Cassandra ValueProvider / Write Operations

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7162?focusedWorklogId=233523&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233523
 ]

ASF GitHub Bot logged work on BEAM-7162:


Author: ASF GitHub Bot
Created on: 26/Apr/19 13:41
Start Date: 26/Apr/19 13:41
Worklog Time Spent: 10m 
  Work Description: mblmat commented on issue #8413: [BEAM-7162] Cassandra 
ValueProvider / Write Operations
URL: https://github.com/apache/beam/pull/8413#issuecomment-487060863
 
 
   retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233523)
Time Spent: 0.5h  (was: 20m)

> Cassandra ValueProvider / Write Operations
> --
>
> Key: BEAM-7162
> URL: https://issues.apache.org/jira/browse/BEAM-7162
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-cassandra
>Affects Versions: 2.12.0
>Reporter: Mathieu Blanchard
>Assignee: Mathieu Blanchard
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Currently, the ValueProvider type is only available for the Read.builder() 
> attributes and we need it for some write operations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7162) Cassandra ValueProvider / Write Operations

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7162?focusedWorklogId=233522&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233522
 ]

ASF GitHub Bot logged work on BEAM-7162:


Author: ASF GitHub Bot
Created on: 26/Apr/19 13:41
Start Date: 26/Apr/19 13:41
Worklog Time Spent: 10m 
  Work Description: mblmat commented on issue #8413: [BEAM-7162] Cassandra 
ValueProvider / Write Operations
URL: https://github.com/apache/beam/pull/8413#issuecomment-487060863
 
 
   retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233522)
Time Spent: 20m  (was: 10m)

> Cassandra ValueProvider / Write Operations
> --
>
> Key: BEAM-7162
> URL: https://issues.apache.org/jira/browse/BEAM-7162
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-cassandra
>Affects Versions: 2.12.0
>Reporter: Mathieu Blanchard
>Assignee: Mathieu Blanchard
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently, the ValueProvider type is only available for the Read.builder() 
> attributes and we need it for some write operations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7112) State cleanup interferes with user timer callback

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7112?focusedWorklogId=233530&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233530
 ]

ASF GitHub Bot logged work on BEAM-7112:


Author: ASF GitHub Bot
Created on: 26/Apr/19 13:45
Start Date: 26/Apr/19 13:45
Worklog Time Spent: 10m 
  Work Description: tweise commented on pull request #8399: [BEAM-7112] 
Timer race with state cleanup - take two
URL: https://github.com/apache/beam/pull/8399#discussion_r278954438
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 ##
 @@ -812,21 +796,35 @@ public boolean isForWindow(
 
 private final List userStateNames;
 private final Coder windowCoder;
-private final StateInternals stateInternals;
+private final ArrayDeque> cleanupQueue;
+private final Supplier keyedStateBackend;
 
-StateCleaner(List userStateNames, Coder windowCoder, 
StateInternals stateInternals) {
+StateCleaner(
+List userStateNames, Coder windowCoder, Supplier 
keyedStateBackend) {
   this.userStateNames = userStateNames;
   this.windowCoder = windowCoder;
-  this.stateInternals = stateInternals;
+  this.keyedStateBackend = keyedStateBackend;
+  this.cleanupQueue = new ArrayDeque<>();
 
 Review comment:
   Hmm, that should not be the case when the cleanup is performed as part of 
finishing the bundle. Need to see how that is possible.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233530)
Time Spent: 7h 10m  (was: 7h)

> State cleanup interferes with user timer callback
> -
>
> Key: BEAM-7112
> URL: https://issues.apache.org/jira/browse/BEAM-7112
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.13.0
>
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> Cleanup timers and user timers are fired at the watermark. Processing of 
> timers in the SDK worker is asynchronous, so it is possible that the state is 
> already removed when the user timer callback executes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7162) Cassandra ValueProvider / Write Operations

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7162?focusedWorklogId=233548&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233548
 ]

ASF GitHub Bot logged work on BEAM-7162:


Author: ASF GitHub Bot
Created on: 26/Apr/19 14:10
Start Date: 26/Apr/19 14:10
Worklog Time Spent: 10m 
  Work Description: mblmat commented on issue #8413: [BEAM-7162] Cassandra 
ValueProvider / Write Operations
URL: https://github.com/apache/beam/pull/8413#issuecomment-487070874
 
 
   @echauchot @iemejia 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233548)
Time Spent: 40m  (was: 0.5h)

> Cassandra ValueProvider / Write Operations
> --
>
> Key: BEAM-7162
> URL: https://issues.apache.org/jira/browse/BEAM-7162
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-cassandra
>Affects Versions: 2.12.0
>Reporter: Mathieu Blanchard
>Assignee: Mathieu Blanchard
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, the ValueProvider type is only available for the Read.builder() 
> attributes and we need it for some write operations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7112) State cleanup interferes with user timer callback

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7112?focusedWorklogId=233575&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233575
 ]

ASF GitHub Bot logged work on BEAM-7112:


Author: ASF GitHub Bot
Created on: 26/Apr/19 14:48
Start Date: 26/Apr/19 14:48
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #8399: [BEAM-7112] Timer race 
with state cleanup - take two
URL: https://github.com/apache/beam/pull/8399#issuecomment-487084689
 
 
   We don't need to checkpoint the cleanup queue since it does not cross the 
bundle boundary. But we need to look into another issue (outside of this PR): 
The finish bundle callback starts a new bundle when the watermark advances, 
which can lead to inconsistent state snapshot.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233575)
Time Spent: 7h 20m  (was: 7h 10m)

> State cleanup interferes with user timer callback
> -
>
> Key: BEAM-7112
> URL: https://issues.apache.org/jira/browse/BEAM-7112
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.13.0
>
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> Cleanup timers and user timers are fired at the watermark. Processing of 
> timers in the SDK worker is asynchronous, so it is possible that the state is 
> already removed when the user timer callback executes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-7112) State cleanup interferes with user timer callback

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7112?focusedWorklogId=233576&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233576
 ]

ASF GitHub Bot logged work on BEAM-7112:


Author: ASF GitHub Bot
Created on: 26/Apr/19 14:50
Start Date: 26/Apr/19 14:50
Worklog Time Spent: 10m 
  Work Description: tweise commented on pull request #8399: [BEAM-7112] 
Timer race with state cleanup - take two
URL: https://github.com/apache/beam/pull/8399#discussion_r278982478
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java
 ##
 @@ -812,21 +796,35 @@ public boolean isForWindow(
 
 private final List userStateNames;
 private final Coder windowCoder;
-private final StateInternals stateInternals;
+private final ArrayDeque> cleanupQueue;
 
 Review comment:
   No need to use use an interface here.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233576)
Time Spent: 7.5h  (was: 7h 20m)

> State cleanup interferes with user timer callback
> -
>
> Key: BEAM-7112
> URL: https://issues.apache.org/jira/browse/BEAM-7112
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.12.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.13.0
>
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> Cleanup timers and user timers are fired at the watermark. Processing of 
> timers in the SDK worker is asynchronous, so it is possible that the state is 
> already removed when the user timer callback executes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-7163) Godoc for passert.Sum seems wrong

[Damien Desfontaines (JIRA)]

Damien Desfontaines created BEAM-7163:
-

 Summary: Godoc for passert.Sum seems wrong
 Key: BEAM-7163
 URL: https://issues.apache.org/jira/browse/BEAM-7163
 Project: Beam
  Issue Type: Improvement
  Components: sdk-go
Reporter: Damien Desfontaines


From 
[https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/testing/passert#Sum]:

> Sum validates that the incoming PCollection is a singleton with the 
> given value.

Why is there a size argument if the input is supposed to be a singleton?

>From the implementation, it seems that Sum sums all elements from the input 
>PCollection (which must be int, apparently?), and checks both the number of 
>elements and their sum.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6526) Add ReadFiles transform for AvroIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6526?focusedWorklogId=233597&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233597
 ]

ASF GitHub Bot logged work on BEAM-6526:


Author: ASF GitHub Bot
Created on: 26/Apr/19 15:29
Start Date: 26/Apr/19 15:29
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #8414: [BEAM-6526] 
Add ReadFiles transform for AvroIO
URL: https://github.com/apache/beam/pull/8414
 
 
   I finally fixed the issue with the breaking tests. PTAL again 
   R: @lgajowy 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233597)
Time Spent: 1h 20m  (was: 1h 10m)

> Add ReadFiles transform for AvroIO
> --
>
> Key: BEAM-6526
> URL: https://issues.apache.org/jira/browse/BEAM-6526
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-avro
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>  Labels: triaged
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> AvroIO lacks the `readFiles()` method to make it fully composable with FileIO 
> as other file based IOs do, e.g. TextIO, ParquetIO.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-7160) Expand ValidatesRunner task to use NeedsRunner test category

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16827058#comment-16827058
 ] 

Ismaël Mejía commented on BEAM-7160:


Aren't NeedsRunner tests run as part of Postcommit ? These are two separate 
categories, no? Or probably is better to revise which of the NeedsRunner should 
become ValidatesRunner ones, no?

> Expand ValidatesRunner task to use NeedsRunner test category
> 
>
> Key: BEAM-7160
> URL: https://issues.apache.org/jira/browse/BEAM-7160
> Project: Beam
>  Issue Type: Test
>  Components: runner-flink
>Reporter: Maximilian Michels
>Priority: Major
>
> The {{:validatesRunner}} task uses the {{ValidatesRunner}} test category 
> which is a subtype of {{NeedsRunner}}. It would be good to expand the scope 
> of the tests to {{NeedsRunner}} because there are many additional tests which 
> currently excluded.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2939) Fn API SDF support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2939?focusedWorklogId=233613&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233613
 ]

ASF GitHub Bot logged work on BEAM-2939:


Author: ASF GitHub Bot
Created on: 26/Apr/19 15:47
Start Date: 26/Apr/19 15:47
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #8412: [BEAM-2939] 
Support SDF expansion in the Flink runner.
URL: https://github.com/apache/beam/pull/8412#discussion_r278994487
 
 

 ##
 File path: 
sdks/python/apache_beam/runners/portability/fn_api_runner_transforms.py
 ##
 @@ -788,14 +799,34 @@ def make_stage(base_stage, transform_id, 
extra_must_follow=()):
 inputs=dict(transform.inputs, **{main_input_tag: paired_pcoll_id}),
 outputs={'out': split_pcoll_id})
 
+if common_urns.composites.RESHUFFLE.urn in context.known_runner_urns:
 
 Review comment:
   Is the plan to eventually retrieve the known URNs dynamically?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233613)
Time Spent: 17.5h  (was: 17h 20m)

> Fn API SDF support
> --
>
> Key: BEAM-2939
> URL: https://issues.apache.org/jira/browse/BEAM-2939
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability, triaged
>  Time Spent: 17.5h
>  Remaining Estimate: 0h
>
> The Fn API should support streaming SDF. Detailed design TBD.
> Once design is ready, expand subtasks similarly to BEAM-2822.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2939) Fn API SDF support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2939?focusedWorklogId=233614&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-233614
 ]

ASF GitHub Bot logged work on BEAM-2939:


Author: ASF GitHub Bot
Created on: 26/Apr/19 15:47
Start Date: 26/Apr/19 15:47
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #8412: [BEAM-2939] 
Support SDF expansion in the Flink runner.
URL: https://github.com/apache/beam/pull/8412#discussion_r278995246
 
 

 ##
 File path: 
sdks/python/apache_beam/runners/portability/fn_api_runner_transforms.py
 ##
 @@ -788,14 +799,34 @@ def make_stage(base_stage, transform_id, 
extra_must_follow=()):
 inputs=dict(transform.inputs, **{main_input_tag: paired_pcoll_id}),
 outputs={'out': split_pcoll_id})
 
+if common_urns.composites.RESHUFFLE.urn in context.known_runner_urns:
+  reshuffle_pcoll_id = copy_like(
+  context.components.pcollections,
+  main_input_id,
+  '_reshuffle',
+  coder_id=sized_coder_id)
+  reshuffle_transform_id = copy_like(
+  context.components.transforms,
+  transform,
+  unique_name=transform.unique_name + '/Reshuffle',
+  urn=common_urns.composites.RESHUFFLE.urn,
+  payload=b'',
+  inputs=dict(transform.inputs, **{main_input_tag: 
split_pcoll_id}),
+  outputs={'out': reshuffle_pcoll_id})
+  yield make_stage(stage, reshuffle_transform_id)
+else:
+  reshuffle_pcoll_id = split_pcoll_id
+  reshuffle_transform_id = None
 
 Review comment:
   Isn't Reshuffle necessary to balance split work evenly?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 233614)
Time Spent: 17h 40m  (was: 17.5h)

> Fn API SDF support
> --
>
> Key: BEAM-2939
> URL: https://issues.apache.org/jira/browse/BEAM-2939
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability, triaged
>  Time Spent: 17h 40m
>  Remaining Estimate: 0h
>
> The Fn API should support streaming SDF. Detailed design TBD.
> Once design is ready, expand subtasks similarly to BEAM-2822.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)