[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401631=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401631 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 11/Mar/20 18:05 Start Date: 11/Mar/20 18:05 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401631) Time Spent: 7h 50m (was: 7h 40m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.21.0 > > Time Spent: 7h 50m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401592=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401592
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 11/Mar/20 17:00
Start Date: 11/Mar/20 17:00
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10945: [BEAM-9295] Add
Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r391122018
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -77,6 +67,7 @@ static ExecutionEnvironment createBatchExecutionEnvironment(
// depending on the master, create the right environment.
if ("[local]".equals(flinkMasterHostPort)) {
+ flinkConfiguration.setString("taskmanager.memory.managed.size", "2048m");
Review comment:
Cool, thanks for the changes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 401592)
Time Spent: 7h 40m (was: 7.5h)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401586=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401586 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 11/Mar/20 16:57 Start Date: 11/Mar/20 16:57 Worklog Time Spent: 10m Work Description: mxm commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597749046 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401586) Time Spent: 7.5h (was: 7h 20m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 7.5h > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401432=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401432 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 11/Mar/20 13:26 Start Date: 11/Mar/20 13:26 Worklog Time Spent: 10m Work Description: mxm commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597632969 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401432) Time Spent: 7h 10m (was: 7h) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 7h 10m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401433=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401433 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 11/Mar/20 13:26 Start Date: 11/Mar/20 13:26 Worklog Time Spent: 10m Work Description: mxm commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597633019 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401433) Time Spent: 7h 20m (was: 7h 10m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 7h 20m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401431=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401431 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 11/Mar/20 13:26 Start Date: 11/Mar/20 13:26 Worklog Time Spent: 10m Work Description: mxm commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597632765 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401431) Time Spent: 7h (was: 6h 50m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 7h > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401429=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401429 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 11/Mar/20 13:25 Start Date: 11/Mar/20 13:25 Worklog Time Spent: 10m Work Description: mxm commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597632470 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401429) Time Spent: 6h 40m (was: 6.5h) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h 40m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401430=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401430 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 11/Mar/20 13:25 Start Date: 11/Mar/20 13:25 Worklog Time Spent: 10m Work Description: mxm commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597632518 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401430) Time Spent: 6h 50m (was: 6h 40m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h 50m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401270=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401270 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 11/Mar/20 08:28 Start Date: 11/Mar/20 08:28 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597504813 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401270) Time Spent: 6h 20m (was: 6h 10m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h 20m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401271=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401271 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 11/Mar/20 08:28 Start Date: 11/Mar/20 08:28 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597504859 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401271) Time Spent: 6.5h (was: 6h 20m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6.5h > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401221 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 11/Mar/20 05:44 Start Date: 11/Mar/20 05:44 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597454420 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401221) Time Spent: 6h 10m (was: 6h) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401220=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401220 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 11/Mar/20 05:44 Start Date: 11/Mar/20 05:44 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597454371 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401220) Time Spent: 6h (was: 5h 50m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 6h > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401214=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401214 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 11/Mar/20 05:16 Start Date: 11/Mar/20 05:16 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597448488 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 401214) Time Spent: 5h 50m (was: 5h 40m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=401203=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401203
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 11/Mar/20 05:03
Start Date: 11/Mar/20 05:03
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #10945:
[BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with
Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r390747511
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -77,6 +67,7 @@ static ExecutionEnvironment createBatchExecutionEnvironment(
// depending on the master, create the right environment.
if ("[local]".equals(flinkMasterHostPort)) {
+ flinkConfiguration.setString("taskmanager.memory.managed.size", "2048m");
Review comment:
Thanks @mxm, Sounds good to me ;)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 401203)
Time Spent: 5h 40m (was: 5.5h)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400824=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400824 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 10/Mar/20 16:31 Start Date: 10/Mar/20 16:31 Worklog Time Spent: 10m Work Description: mxm commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597183478 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 400824) Time Spent: 5.5h (was: 5h 20m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 5.5h > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400738=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400738 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 10/Mar/20 14:23 Start Date: 10/Mar/20 14:23 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597113525 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 400738) Time Spent: 5h 20m (was: 5h 10m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 5h 20m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400723=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400723
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 10/Mar/20 13:53
Start Date: 10/Mar/20 13:53
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10945: [BEAM-9295] Add
Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r390328606
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -77,6 +67,7 @@ static ExecutionEnvironment createBatchExecutionEnvironment(
// depending on the master, create the right environment.
if ("[local]".equals(flinkMasterHostPort)) {
+ flinkConfiguration.setString("taskmanager.memory.managed.size", "2048m");
Review comment:
I'm hesitant with this default because it will always pre-allocate 2GB of
memory which won't be used most of the time, except for the one large record
test case you mentioned.
We could set I'd go for something like
https://github.com/apache/flink/blob/42a56f4c75693773e21fa2dea45df640c2d7f9da/flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/TaskExecutorProcessUtils.java#L287
based on the memory available.
Actually, that is what the Flink 1.8 code used to do:
https://github.com/apache/flink/blob/60d9b96456f142f8d18d5882016840a00159403e/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerServices.java#L296
So let's just check the free memory and use a fraction for memory managed
memory by default. What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400723)
Time Spent: 5h 10m (was: 5h)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400706=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400706
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 10/Mar/20 12:47
Start Date: 10/Mar/20 12:47
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #10945:
[BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with
Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r390287605
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -77,6 +67,7 @@ static ExecutionEnvironment createBatchExecutionEnvironment(
// depending on the master, create the right environment.
if ("[local]".equals(flinkMasterHostPort)) {
+ flinkConfiguration.setString("taskmanager.memory.managed.size", "2048m");
Review comment:
It will set the `taskmanager.memory.managed.size` as
[128MB](https://github.com/apache/flink/blob/b90d1a269661f4e5ca132213a03169fc9011aaee/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutorResourceUtils.java#L139)
for MiniCluster if it's not set. I think set
`taskmanager.memory.managed.fraction"` doesn't take effect here. Thoughts? :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400706)
Time Spent: 5h (was: 4h 50m)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400670 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 10/Mar/20 11:11 Start Date: 10/Mar/20 11:11 Worklog Time Spent: 10m Work Description: mxm commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-597030917 Thanks for making the adjustments. A couple more comments but otherwise it looks very good! Great work. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 400670) Time Spent: 4h 50m (was: 4h 40m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 4h 50m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400665
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 10/Mar/20 11:09
Start Date: 10/Mar/20 11:09
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10945: [BEAM-9295] Add
Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r390240614
##
File path:
runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkExecutionEnvironmentsTest.java
##
@@ -438,8 +424,34 @@ public void shouldSetSavepointRestoreForRemoteStreaming()
{
options, Collections.emptyList());
// subject to change with https://issues.apache.org/jira/browse/FLINK-11048
assertThat(sev, instanceOf(RemoteStreamEnvironment.class));
-assertThat(
-Whitebox.getInternalState(sev, "restoreSettings"),
-is(SavepointRestoreSettings.forPath(path)));
+assertThat(getSavepointPath(sev), is(path));
+ }
+
+ private void checkHostAndPort(Object env, String expectedHost, int
expectedPort) {
+try {
+ assertThat(Whitebox.getInternalState(env, "host"), is(expectedHost));
+ assertThat(Whitebox.getInternalState(env, "port"), is(expectedPort));
+} catch (Throwable t) {
Review comment:
We shouldn't be catching throwable here.
```suggestion
} catch (FieldNotFoundException e) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400665)
Time Spent: 4.5h (was: 4h 20m)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400666
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 10/Mar/20 11:09
Start Date: 10/Mar/20 11:09
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10945: [BEAM-9295] Add
Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r390241188
##
File path:
runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironmentTest.java
##
@@ -418,4 +426,20 @@ private FlinkPipelineOptions setPipelineOptions(
})
.collect(Collectors.toList());
}
+
+ private List getJars(Object env) throws Exception {
+try {
+ return (List) Whitebox.getInternalState(env, "jarFiles");
+} catch (Throwable t) {
Review comment:
We shouldn't be catching throwable here.
```suggestion
} catch (FieldNotFoundException e) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400666)
Time Spent: 4h 40m (was: 4.5h)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400664
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 10/Mar/20 11:09
Start Date: 10/Mar/20 11:09
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10945: [BEAM-9295] Add
Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r390240725
##
File path:
runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkExecutionEnvironmentsTest.java
##
@@ -438,8 +424,34 @@ public void shouldSetSavepointRestoreForRemoteStreaming()
{
options, Collections.emptyList());
// subject to change with https://issues.apache.org/jira/browse/FLINK-11048
assertThat(sev, instanceOf(RemoteStreamEnvironment.class));
-assertThat(
-Whitebox.getInternalState(sev, "restoreSettings"),
-is(SavepointRestoreSettings.forPath(path)));
+assertThat(getSavepointPath(sev), is(path));
+ }
+
+ private void checkHostAndPort(Object env, String expectedHost, int
expectedPort) {
+try {
+ assertThat(Whitebox.getInternalState(env, "host"), is(expectedHost));
+ assertThat(Whitebox.getInternalState(env, "port"), is(expectedPort));
+} catch (Throwable t) {
+ // for flink 1.10+
+ String host =
+ ((Configuration) Whitebox.getInternalState(env, "configuration"))
+ .getString(RestOptions.ADDRESS);
+ int port =
+ ((Configuration) Whitebox.getInternalState(env, "configuration"))
+ .getInteger(RestOptions.PORT);
+ assertThat(
+ new InetSocketAddress(host, port), is(new
InetSocketAddress(expectedHost, expectedPort)));
+}
+ }
+
+ private String getSavepointPath(Object env) {
+try {
+ return ((SavepointRestoreSettings) Whitebox.getInternalState(env,
"restoreSettings"))
+ .getRestorePath();
+} catch (Throwable t) {
Review comment:
We shouldn't be catching throwable here.
```suggestion
} catch (FieldNotFoundException e) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400664)
Time Spent: 4.5h (was: 4h 20m)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400663
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 10/Mar/20 11:09
Start Date: 10/Mar/20 11:09
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10945: [BEAM-9295] Add
Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r390239907
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -77,6 +67,7 @@ static ExecutionEnvironment createBatchExecutionEnvironment(
// depending on the master, create the right environment.
if ("[local]".equals(flinkMasterHostPort)) {
+ flinkConfiguration.setString("taskmanager.memory.managed.size", "2048m");
Review comment:
I have the feeling this won't be reliable enough. Why not instead
`taskmanager.memory.managed.fraction`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400663)
Time Spent: 4.5h (was: 4h 20m)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400608=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400608 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 10/Mar/20 08:44 Start Date: 10/Mar/20 08:44 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-596962643 Thanks for the review @mxm ! And the suggestion about test case is great :) , I have update the PR accordingly :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 400608) Time Spent: 4h 20m (was: 4h 10m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400601=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400601 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 10/Mar/20 08:28 Start Date: 10/Mar/20 08:28 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-596962643 Thanks for the review @mxm ! I have update the PR accordingly :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 400601) Time Spent: 4h 10m (was: 4h) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400310=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400310 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 09/Mar/20 18:46 Start Date: 09/Mar/20 18:46 Worklog Time Spent: 10m Work Description: mxm commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-596716342 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 400310) Time Spent: 4h (was: 3h 50m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400179=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400179
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 09/Mar/20 15:26
Start Date: 09/Mar/20 15:26
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10945: [BEAM-9295] Add
Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r389761717
##
File path:
runners/flink/1.10/src/test/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironmentTest.java
##
@@ -0,0 +1,441 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import static org.apache.beam.sdk.testing.RegexMatcher.matches;
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.hamcrest.CoreMatchers.instanceOf;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.CoreMatchers.startsWith;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.core.Every.everyItem;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.fail;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.PrintStream;
+import java.io.Serializable;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.beam.runners.core.construction.PTransformMatchers;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.resources.PipelineResources;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.io.GenerateSequence;
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.runners.PTransformOverride;
+import org.apache.beam.sdk.runners.PTransformOverrideFactory;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api.java.RemoteEnvironment;
+import org.apache.flink.client.cli.ExecutionConfigAccessor;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.environment.RemoteStreamEnvironment;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.hamcrest.BaseMatcher;
+import org.hamcrest.Description;
+import org.hamcrest.Matchers;
+import org.joda.time.Duration;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mockito;
+import org.powermock.reflect.Whitebox;
+
+/**
+ * Tests for {@link FlinkPipelineExecutionEnvironment}.
+ *
+ * This test is copied to 1.10 is becauses the field jarFiles has been
removed from
+ * RemoteEnvironment in Flink 1.10, please refer to
+ *
https://github.com/apache/flink/commit/057c036784242c674ea6091549cdbc98688827a6
for more details.
+ */
+@RunWith(JUnit4.class)
+public class FlinkPipelineExecutionEnvironmentTest implements Serializable {
+
+ @Rule public transient TemporaryFolder tmpFolder = new TemporaryFolder();
+
+ @Test
+ public void shouldRecognizeAndTranslateStreamingPipeline() {
+FlinkPipelineOptions options =
PipelineOptionsFactory.as(FlinkPipelineOptions.class);
+options.setRunner(TestFlinkRunner.class);
+options.setFlinkMaster("[auto]");
+
+
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400180=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400180
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 09/Mar/20 15:26
Start Date: 09/Mar/20 15:26
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10945: [BEAM-9295] Add
Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r389761038
##
File path:
runners/flink/1.10/src/test/java/org/apache/beam/runners/flink/FlinkExecutionEnvironmentsTest.java
##
@@ -0,0 +1,533 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.instanceOf;
+import static org.hamcrest.core.Is.is;
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.util.Collections;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api.java.LocalEnvironment;
+import org.apache.flink.api.java.RemoteEnvironment;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.runtime.jobgraph.SavepointConfigOptions;
+import org.apache.flink.streaming.api.environment.LocalStreamEnvironment;
+import org.apache.flink.streaming.api.environment.RemoteStreamEnvironment;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
+import org.powermock.reflect.Whitebox;
+
+/**
+ * Tests for {@link FlinkExecutionEnvironments}.
+ *
+ * This test is copied to 1.10 is becauses the field host, port, etc have
been removed from
+ * RemoteEnvironment in Flink 1.10, please refer to
+ *
https://github.com/apache/flink/commit/057c036784242c674ea6091549cdbc98688827a6
for more details.
+ */
+public class FlinkExecutionEnvironmentsTest {
+
+ @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder();
+ @Rule public ExpectedException expectedException = ExpectedException.none();
+
+ @Test
+ public void shouldSetParallelismBatch() {
+FlinkPipelineOptions options =
PipelineOptionsFactory.as(FlinkPipelineOptions.class);
+options.setRunner(TestFlinkRunner.class);
+options.setParallelism(42);
+
+ExecutionEnvironment bev =
+FlinkExecutionEnvironments.createBatchExecutionEnvironment(
+options, Collections.emptyList());
+
+assertThat(options.getParallelism(), is(42));
+assertThat(bev.getParallelism(), is(42));
+ }
+
+ @Test
+ public void shouldSetParallelismStreaming() {
+FlinkPipelineOptions options =
PipelineOptionsFactory.as(FlinkPipelineOptions.class);
+options.setRunner(TestFlinkRunner.class);
+options.setParallelism(42);
+
+StreamExecutionEnvironment sev =
+FlinkExecutionEnvironments.createStreamExecutionEnvironment(
+options, Collections.emptyList());
+
+assertThat(options.getParallelism(), is(42));
+assertThat(sev.getParallelism(), is(42));
+ }
+
+ @Test
+ public void shouldSetMaxParallelismStreaming() {
+FlinkPipelineOptions options =
PipelineOptionsFactory.as(FlinkPipelineOptions.class);
+options.setRunner(TestFlinkRunner.class);
+options.setMaxParallelism(42);
+
+StreamExecutionEnvironment sev =
+FlinkExecutionEnvironments.createStreamExecutionEnvironment(
+options, Collections.emptyList());
+
+assertThat(options.getMaxParallelism(), is(42));
+assertThat(sev.getMaxParallelism(), is(42));
+ }
+
+ @Test
+ public void shouldInferParallelismFromEnvironmentBatch() throws IOException {
+String flinkConfDir = extractFlinkConfig();
+
+FlinkPipelineOptions options =
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=400178=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400178
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 09/Mar/20 15:26
Start Date: 09/Mar/20 15:26
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10945: [BEAM-9295] Add
Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r389763087
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -311,78 +302,4 @@ private static void applyLatencyTrackingInterval(
long latencyTrackingInterval = options.getLatencyTrackingInterval();
config.setLatencyTrackingInterval(latencyTrackingInterval);
}
-
- /**
- * Remote stream environment that supports job execution with restore from
savepoint.
- *
- * This class can be removed once Flink provides this functionality.
- *
- * TODO: https://issues.apache.org/jira/browse/BEAM-5396
- */
- private static class BeamFlinkRemoteStreamEnvironment extends
RemoteStreamEnvironment {
Review comment:
I didn't see that. In this case
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 400178)
Time Spent: 3h 40m (was: 3.5h)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=395401=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395401 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 29/Feb/20 05:26 Start Date: 29/Feb/20 05:26 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-592885120 Run CommunityMetrics PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 395401) Time Spent: 3.5h (was: 3h 20m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=395400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395400 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 29/Feb/20 05:26 Start Date: 29/Feb/20 05:26 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-592885034 Run Spotless PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 395400) Time Spent: 3h 20m (was: 3h 10m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=395398=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395398
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 29/Feb/20 05:20
Start Date: 29/Feb/20 05:20
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #10945:
[BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with
Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r386002840
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -311,78 +302,4 @@ private static void applyLatencyTrackingInterval(
long latencyTrackingInterval = options.getLatencyTrackingInterval();
config.setLatencyTrackingInterval(latencyTrackingInterval);
}
-
- /**
- * Remote stream environment that supports job execution with restore from
savepoint.
- *
- * This class can be removed once Flink provides this functionality.
- *
- * TODO: https://issues.apache.org/jira/browse/BEAM-5396
- */
- private static class BeamFlinkRemoteStreamEnvironment extends
RemoteStreamEnvironment {
Review comment:
BeamFlinkRemoteStreamEnvironment has not been removed. Actually it has been
moved to runner 1.7. The reason is that in 1.10 we don't need it any more. What
do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395398)
Time Spent: 3h (was: 2h 50m)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=395399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395399 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 29/Feb/20 05:20 Start Date: 29/Feb/20 05:20 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-592882962 @mxm Thanks for the review. I have updated the PR according to your comments. Currently it only copied 3 tests and I think even we drop 1.7 support, we still need to copy these tests. What's your thought? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 395399) Time Spent: 3h 10m (was: 3h) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=394752=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394752 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 28/Feb/20 09:07 Start Date: 28/Feb/20 09:07 Worklog Time Spent: 10m Work Description: mxm commented on pull request #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#discussion_r385578594 ## File path: runners/flink/1.10/src/test/java/org/apache/beam/runners/flink/FlinkSubmissionTest.java ## @@ -0,0 +1,256 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.runners.flink; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.is; + +import java.io.File; +import java.lang.reflect.Field; +import java.lang.reflect.Modifier; +import java.nio.file.Files; +import java.security.Permission; +import java.util.Collection; +import java.util.Map; +import java.util.concurrent.TimeUnit; +import org.apache.beam.runners.core.construction.resources.PipelineResources; +import org.apache.beam.sdk.Pipeline; +import org.apache.beam.sdk.io.GenerateSequence; +import org.apache.beam.sdk.options.PipelineOptions; +import org.apache.beam.sdk.options.PipelineOptionsFactory; +import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Charsets; +import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList; +import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap; +import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterables; +import org.apache.flink.api.common.JobStatus; +import org.apache.flink.client.cli.CliFrontend; +import org.apache.flink.configuration.ConfigConstants; +import org.apache.flink.configuration.Configuration; +import org.apache.flink.configuration.JobManagerOptions; +import org.apache.flink.configuration.RestOptions; +import org.apache.flink.runtime.client.JobStatusMessage; +import org.apache.flink.runtime.minicluster.MiniClusterConfiguration; +import org.apache.flink.runtime.minicluster.RpcServiceSharing; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.ClassRule; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; +import org.junit.rules.Timeout; + +/** + * End-to-end submission test of Beam jobs on a Flink cluster. + * + * This test is copied to 1.10 is becauses the package name of JobStatus has changed in Flink + * 1.10, please refer to Review comment: No need to copy everything because of one change package name. We can load JobStatus dynamically. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394752) Time Spent: 2h 50m (was: 2h 40m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=394751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394751
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 28/Feb/20 09:07
Start Date: 28/Feb/20 09:07
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #10945: [BEAM-9295] Add
Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r385577426
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -311,78 +302,4 @@ private static void applyLatencyTrackingInterval(
long latencyTrackingInterval = options.getLatencyTrackingInterval();
config.setLatencyTrackingInterval(latencyTrackingInterval);
}
-
- /**
- * Remote stream environment that supports job execution with restore from
savepoint.
- *
- * This class can be removed once Flink provides this functionality.
- *
- * TODO: https://issues.apache.org/jira/browse/BEAM-5396
- */
- private static class BeamFlinkRemoteStreamEnvironment extends
RemoteStreamEnvironment {
Review comment:
This cannot be removed yet. The feature is only present in 1.8.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 394751)
Time Spent: 2h 40m (was: 2.5h)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=394555=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394555 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 28/Feb/20 00:18 Start Date: 28/Feb/20 00:18 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-592244965 Test failure is not caused by the current PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394555) Time Spent: 2.5h (was: 2h 20m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=394006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394006 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 27/Feb/20 07:48 Start Date: 27/Feb/20 07:48 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-591809987 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394006) Time Spent: 2h 20m (was: 2h 10m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=394005=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394005 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 27/Feb/20 07:47 Start Date: 27/Feb/20 07:47 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-591830495 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394005) Time Spent: 2h 10m (was: 2h) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=393988=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-393988 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 27/Feb/20 06:44 Start Date: 27/Feb/20 06:44 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-591809987 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 393988) Time Spent: 1h 50m (was: 1h 40m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=393989=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-393989 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 27/Feb/20 06:44 Start Date: 27/Feb/20 06:44 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-591810030 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 393989) Time Spent: 2h (was: 1h 50m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=393191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-393191
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 26/Feb/20 05:41
Start Date: 26/Feb/20 05:41
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on pull request #10945:
[BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with
Flink 1.10
URL: https://github.com/apache/beam/pull/10945#discussion_r384284237
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -67,6 +69,7 @@ static ExecutionEnvironment createBatchExecutionEnvironment(
// depending on the master, create the right environment.
if ("[local]".equals(flinkMasterHostPort)) {
+ flinkConfiguration.set(TaskManagerOptions.MANAGED_MEMORY_SIZE,
MemorySize.parse("2048m"));
Review comment:
It has sets the default managed memory size to 128MB for MiniCluster in
https://issues.apache.org/jira/browse/FLINK-15763. Have set it to a large value
when the master host is [local]. Appreciate for any suggestions on a better way
to address this issue.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 393191)
Time Spent: 1h 40m (was: 1.5h)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=392602=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392602
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 25/Feb/20 14:51
Start Date: 25/Feb/20 14:51
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295]
Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#issuecomment-590905056
Hi @mxm I found that the test case `test_large_elements` failed with
execption:
```
java.io.IOException: Cannot write record to fresh sort buffer. Record too
large.
at
org.apache.flink.runtime.operators.chaining.SynchronousChainedCombineDriver.collect(SynchronousChainedCombineDriver.java:176)
at
org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
at
org.apache.flink.runtime.operators.MapDriver.run(MapDriver.java:103)
at
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:504)
at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:369)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
```
I guess it's related to the feature "Unified Memory Configuration for
TaskExecutors"(https://issues.apache.org/jira/browse/FLINK-13980) which is
introduced in Flink 1.10. Before 1.10, the memory managed by the Flink's
MemoryManager is calculated dynamically if not configured and I have checked
that it will be about 2500 MB in my local machine. Since 1.10, it will be 128
MB if not configured (taskmanager.memory.managed.size).
I have performed a simple test and the failed test `test_large_elements`
could pass after adding the following code at
https://github.com/apache/beam/blob/7b3a3fa6c9291692b56dbc358dfc075724b993b6/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java#L77
:
```
flinkConfiguration.set(TaskManagerOptions.MANAGED_MEMORY_SIZE,
MemorySize.parse("2048m"));
```
I'm still investigating the best way to address this issue at Beam side. And
appreciate if you have any suggestion on this :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 392602)
Time Spent: 1.5h (was: 1h 20m)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=392597=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392597 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 25/Feb/20 14:49 Start Date: 25/Feb/20 14:49 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-590903651 > There a large amount of code duplication. Is that really necessary? AFAIK the changes between 1.9 and 1.10 should be minimal. Could you please revise this or explain why it is necessary? There are a lot of changes related to the job client API(https://issues.apache.org/jira/browse/FLINK-14392, https://issues.apache.org/jira/browse/FLINK-14376) in 1.10. I have added comments at the header of each test case copied to 1.10 for the reason making a copy of them. Could you help to take a look at if the reason makes sense for you? :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 392597) Time Spent: 1h 20m (was: 1h 10m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=392585=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392585
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 25/Feb/20 14:38
Start Date: 25/Feb/20 14:38
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295]
Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#issuecomment-590897942
The test case `test_large_elements` failed with execption:
```
java.io.IOException: Cannot write record to fresh sort buffer. Record too
large.
at
org.apache.flink.runtime.operators.chaining.SynchronousChainedCombineDriver.collect(SynchronousChainedCombineDriver.java:176)
at
org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
at
org.apache.flink.runtime.operators.MapDriver.run(MapDriver.java:103)
at
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:504)
at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:369)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
```
I guess it's related to the feature "Unified Memory Configuration for
TaskExecutors"(https://issues.apache.org/jira/browse/FLINK-13980) which is
introduced in Flink 1.10. Before 1.10, the memory managed by the Flink's
MemoryManager is calculated dynamically if not configured and I have checked
that it will be about 2500 MB in my local machine. Since 1.10, it will be 128
MB if not configured (taskmanager.memory.managed.size).
I have performed a simple test and the failed test `test_large_elements`
could pass after adding the following code at
https://github.com/apache/beam/blob/7b3a3fa6c9291692b56dbc358dfc075724b993b6/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java#L77
:
```
flinkConfiguration.set(TaskManagerOptions.MANAGED_MEMORY_SIZE,
MemorySize.parse("2048m"));
```
I'm still investigating the best way to address this issue at Beam side.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 392585)
Time Spent: 1h 10m (was: 1h)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[
https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=392584=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392584
]
ASF GitHub Bot logged work on BEAM-9295:
Author: ASF GitHub Bot
Created on: 25/Feb/20 14:37
Start Date: 25/Feb/20 14:37
Worklog Time Spent: 10m
Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295]
Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
URL: https://github.com/apache/beam/pull/10945#issuecomment-590897942
The test case `test_large_elements` failed with execption:
```
java.io.IOException: Cannot write record to fresh sort buffer. Record too
large.
at
org.apache.flink.runtime.operators.chaining.SynchronousChainedCombineDriver.collect(SynchronousChainedCombineDriver.java:176)
at
org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
at
org.apache.flink.runtime.operators.MapDriver.run(MapDriver.java:103)
at
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:504)
at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:369)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
```
I guess it's related to the feature "Unified Memory Configuration for
TaskExecutors"(https://issues.apache.org/jira/browse/FLINK-13980) which is
introduced in Flink 1.10. Before 1.10, the memory managed by the Flink's
MemoryManager is calculated dynamically if not configured and I have checked
that it will be about 2500 MB in my local machine. Since 1.10, it will be 128
MB if not configured (taskmanager.memory.managed.size).
I have performed a simple test and the failed test `test_large_elements`
could pass after adding the following code at
https://github.com/apache/beam/blob/7b3a3fa6c9291692b56dbc358dfc075724b993b6/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java#L77
:
```
flinkConfiguration.set(TaskManagerOptions.MANAGED_MEMORY_SIZE,
MemorySize.parse("2048m"));
```
I'm still investigating the best way to address this issue at Beam side.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 392584)
Time Spent: 1h (was: 50m)
> Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
> ---
>
> Key: BEAM-9295
> URL: https://issues.apache.org/jira/browse/BEAM-9295
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Apache Flink 1.10 has completed the final release vote, see [1]. So, I would
> like to add Flink 1.10 build target and make Flink Runner compatible with
> Flink 1.10.
> And I appreciate it if you can leave your suggestions or comments!
> [1]
> https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=391548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391548 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 24/Feb/20 07:18 Start Date: 24/Feb/20 07:18 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-590186860 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391548) Time Spent: 50m (was: 40m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=391545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391545 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 24/Feb/20 07:06 Start Date: 24/Feb/20 07:06 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-590190781 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391545) Time Spent: 40m (was: 0.5h) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=391540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391540 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 24/Feb/20 06:46 Start Date: 24/Feb/20 06:46 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-590186806 Run CommunityMetrics PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391540) Time Spent: 20m (was: 10m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=391541=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391541 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 24/Feb/20 06:46 Start Date: 24/Feb/20 06:46 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945#issuecomment-590186860 Run Python2_PVR_Flink PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 391541) Time Spent: 0.5h (was: 20m) > Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 > --- > > Key: BEAM-9295 > URL: https://issues.apache.org/jira/browse/BEAM-9295 > Project: Beam > Issue Type: New Feature > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Apache Flink 1.10 has completed the final release vote, see [1]. So, I would > like to add Flink 1.10 build target and make Flink Runner compatible with > Flink 1.10. > And I appreciate it if you can leave your suggestions or comments! > [1] > https://lists.apache.org/thread.html/r97672d4d1e47372cebf23e6643a6cc30a06bfbdf3f277b0be3695b15%40%3Cdev.flink.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9295) Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10
[ https://issues.apache.org/jira/browse/BEAM-9295?focusedWorklogId=391491=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-391491 ] ASF GitHub Bot logged work on BEAM-9295: Author: ASF GitHub Bot Created on: 24/Feb/20 03:17 Start Date: 24/Feb/20 03:17 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #10945: [BEAM-9295] Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 URL: https://github.com/apache/beam/pull/10945 Add Flink 1.10 build target and Make FlinkRunner compatible with Flink 1.10 Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[