[jira] [Resolved] (FLINK-7462) Add very obvious warning about outdated docs

[Timo Walther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther resolved FLINK-7462.
-
   Resolution: Fixed
Fix Version/s: 1.4.0

Fixed in 1.0: 5028373881accfdf39988e8b1c085e9b8c048367
Fixed in 1.1: bc1bfbc0bb6b85a38401cd993fe9ca84ef539dc8
Fixed in 1.2: 4b218692513dae415a2e90971c63e4a9c002fba6
Fixed in 1.3: 93e9dba36c80cf957be4ef370a15a47d18a7a51d
Fixed in 1.4: 6c6d90084c9be27eb8c43f0f642c76e4dec9a4f6

> Add very obvious warning about outdated docs
> 
>
> Key: FLINK-7462
> URL: https://issues.apache.org/jira/browse/FLINK-7462
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
> Fix For: 1.4.0
>
>
> The current warning for outdated docs is not very obvious in the footer of 
> the page. I would like to increase the visibility of this by adjusting this 
> footer and adding a warning to actual content.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4553: [FLINK-7642] [docs] Add very obvious warning about outdat...

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/4553
  
I merge this to all release branches >= 1.0. Feel free to check if I didn't 
make a mistake.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4553: [FLINK-7642] [docs] Add very obvious warning about...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/4553


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3089) State API Should Support Data Expiration (State TTL)

[Yonatan Most (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129939#comment-16129939
 ] 

Yonatan Most commented on FLINK-3089:
-

[~sihuazhou]
That's very interesting!
Could you possibly share or point me to a code sample of how you got Flink to 
use TtlDB?

> State API Should Support Data Expiration (State TTL)
> 
>
> Key: FLINK-3089
> URL: https://issues.apache.org/jira/browse/FLINK-3089
> Project: Flink
>  Issue Type: New Feature
>  Components: DataStream API, State Backends, Checkpointing
>Reporter: Niels Basjes
>
> In some usecases (webanalytics) there is a need to have a state per visitor 
> on a website (i.e. keyBy(sessionid) ).
> At some point the visitor simply leaves and no longer creates new events (so 
> a special 'end of session' event will not occur).
> The only way to determine that a visitor has left is by choosing a timeout, 
> like "After 30 minutes no events we consider the visitor 'gone'".
> Only after this (chosen) timeout has expired should we discard this state.
> In the Trigger part of Windows we can set a timer and close/discard this kind 
> of information. But that introduces the buffering effect of the window (which 
> in some scenarios is unwanted).
> What I would like is to be able to set a timeout on a specific state which I 
> can update afterwards.
> This makes it possible to create a map function that assigns the right value 
> and that discards the state automatically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4553: [FLINK-7642] [docs] Add very obvious warning about outdat...

[twalthr]

Github user twalthr commented on the issue:

https://github.com/apache/flink/pull/4553
  
+1 Thank you @uce. I will merge this...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3089) State API Should Support Data Expiration (State TTL)

[Sihua Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129896#comment-16129896
 ] 

Sihua Zhou commented on FLINK-3089:
---

We use TtlDB(provided by rocksdb that extends Rocksdbs) to support TTL in flink 
and it works fine. Is it also acceptable for community?

> State API Should Support Data Expiration (State TTL)
> 
>
> Key: FLINK-3089
> URL: https://issues.apache.org/jira/browse/FLINK-3089
> Project: Flink
>  Issue Type: New Feature
>  Components: DataStream API, State Backends, Checkpointing
>Reporter: Niels Basjes
>
> In some usecases (webanalytics) there is a need to have a state per visitor 
> on a website (i.e. keyBy(sessionid) ).
> At some point the visitor simply leaves and no longer creates new events (so 
> a special 'end of session' event will not occur).
> The only way to determine that a visitor has left is by choosing a timeout, 
> like "After 30 minutes no events we consider the visitor 'gone'".
> Only after this (chosen) timeout has expired should we discard this state.
> In the Trigger part of Windows we can set a timer and close/discard this kind 
> of information. But that introduces the buffering effect of the window (which 
> in some scenarios is unwanted).
> What I would like is to be able to set a timeout on a specific state which I 
> can update afterwards.
> This makes it possible to create a map function that assigns the right value 
> and that discards the state automatically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6465) support FIRST_VALUE on Table API & SQL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129822#comment-16129822
 ] 

ASF GitHub Bot commented on FLINK-6465:
---

GitHub user sunjincheng121 opened a pull request:

https://github.com/apache/flink/pull/4556

[FLINK-6465][table]support FIRST_VALUE on Table API & SQL

*Thank you very much for contributing to Apache Flink - we are happy that 
you want to help us improve Flink. To help the community review your 
contribution in the best possible way, please go through the checklist below, 
which will get the contribution into a shape in which it can be best reviewed.*

*Please understand that we do not do this to make contributions to Flink a 
hassle. In order to uphold a high standard of quality for code contributions, 
while at the same time managing a large number of contributions, we need 
contributors to prepare the contributions well, and give reviewers enough 
contextual information for the review. Please also understand that 
contributions that do not follow this guide will take longer to review and thus 
typically be picked up with lower priority by the community.*

## Contribution Checklist

  - Make sure that the pull request corresponds to a [JIRA 
issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are 
made for typos in JavaDoc or documentation files, which need no JIRA issue.
  
  - Name the pull request in the form "[FLINK-] [component] Title of 
the pull request", where *FLINK-* should be replaced by the actual issue 
number. Skip *component* if you are unsure about which is the best component.
  Typo fixes that have no associated JIRA issue should be named following 
this pattern: `[hotfix] [docs] Fix typo in event time introduction` or 
`[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.

  - Fill out the template below to describe the changes contributed by the 
pull request. That will give reviewers the context they need to do the review.
  
  - Make sure that the change passes the automated tests, i.e., `mvn clean 
verify` passes. You can set up Travis CI to do that following [this 
guide](http://flink.apache.org/contribute-code.html#best-practices).

  - Each pull request should address only one issue, not mix up code from 
multiple issues.
  
  - Each commit in the pull request has a meaningful commit message 
(including the JIRA id)

  - Once all items of the checklist are addressed, remove the above text 
and this checklist, leaving only the filled out template below.


**(The sections below can be removed for hotfixes of typos)**

## What is the purpose of the change
This PR. try to add `FIRST_VALUE` aggregate function OVER window on table 
API

## Brief change log
- *Add `FIRST_VALUE` aggregate function.
- *Add `FIRST_VALUE` test in OVER window.
- *Add `FIRST_VALUE` test case.(only for aggregate function)

## Verifying this change
This change added tests and can be verified as follows:
  - *Added integration tests for over window(tableAPI)
  - *Added test that validates that  `FIRST_VALUE` can deal with 
BYTE/SHORT/INT/LONG/DOUBLE/STRING/BIGDECIMAL
  
## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: ( no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (yes)
  - If yes, how is the feature documented? (docs)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sunjincheng121/flink FLINK-6465-PR

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4556.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4556


commit b6278aa3a318ba647dfa6aa96790dd18131a7670
Author: sunjincheng121 
Date:   2017-08-17T02:56:50Z

[FLINK-6465][table]support FIRST_VALUE on Table API & SQL




> support FIRST_VALUE on Table API & SQL
> --
>
> Key: FLINK-6465
> URL: https://issues.apache.org/jira/browse/FLINK-6465
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Hequn Cheng
>Assignee: sunjincheng
>
> {{FIRST_VALUE}} is a OVER 

[GitHub] flink pull request #4556: [FLINK-6465][table]support FIRST_VALUE on Table AP...

[sunjincheng121]

GitHub user sunjincheng121 opened a pull request:

https://github.com/apache/flink/pull/4556

[FLINK-6465][table]support FIRST_VALUE on Table API & SQL

*Thank you very much for contributing to Apache Flink - we are happy that 
you want to help us improve Flink. To help the community review your 
contribution in the best possible way, please go through the checklist below, 
which will get the contribution into a shape in which it can be best reviewed.*

*Please understand that we do not do this to make contributions to Flink a 
hassle. In order to uphold a high standard of quality for code contributions, 
while at the same time managing a large number of contributions, we need 
contributors to prepare the contributions well, and give reviewers enough 
contextual information for the review. Please also understand that 
contributions that do not follow this guide will take longer to review and thus 
typically be picked up with lower priority by the community.*

## Contribution Checklist

  - Make sure that the pull request corresponds to a [JIRA 
issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are 
made for typos in JavaDoc or documentation files, which need no JIRA issue.
  
  - Name the pull request in the form "[FLINK-] [component] Title of 
the pull request", where *FLINK-* should be replaced by the actual issue 
number. Skip *component* if you are unsure about which is the best component.
  Typo fixes that have no associated JIRA issue should be named following 
this pattern: `[hotfix] [docs] Fix typo in event time introduction` or 
`[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.

  - Fill out the template below to describe the changes contributed by the 
pull request. That will give reviewers the context they need to do the review.
  
  - Make sure that the change passes the automated tests, i.e., `mvn clean 
verify` passes. You can set up Travis CI to do that following [this 
guide](http://flink.apache.org/contribute-code.html#best-practices).

  - Each pull request should address only one issue, not mix up code from 
multiple issues.
  
  - Each commit in the pull request has a meaningful commit message 
(including the JIRA id)

  - Once all items of the checklist are addressed, remove the above text 
and this checklist, leaving only the filled out template below.


**(The sections below can be removed for hotfixes of typos)**

## What is the purpose of the change
This PR. try to add `FIRST_VALUE` aggregate function OVER window on table 
API

## Brief change log
- *Add `FIRST_VALUE` aggregate function.
- *Add `FIRST_VALUE` test in OVER window.
- *Add `FIRST_VALUE` test case.(only for aggregate function)

## Verifying this change
This change added tests and can be verified as follows:
  - *Added integration tests for over window(tableAPI)
  - *Added test that validates that  `FIRST_VALUE` can deal with 
BYTE/SHORT/INT/LONG/DOUBLE/STRING/BIGDECIMAL
  
## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: ( no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (yes)
  - If yes, how is the feature documented? (docs)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sunjincheng121/flink FLINK-6465-PR

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4556.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4556


commit b6278aa3a318ba647dfa6aa96790dd18131a7670
Author: sunjincheng121 
Date:   2017-08-17T02:56:50Z

[FLINK-6465][table]support FIRST_VALUE on Table API & SQL




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-7463) Release nascent workers on slot request timeout

[Eron Wright (JIRA)]

Eron Wright  created FLINK-7463:
---

 Summary: Release nascent workers on slot request timeout 
 Key: FLINK-7463
 URL: https://issues.apache.org/jira/browse/FLINK-7463
 Project: Flink
  Issue Type: Sub-task
Reporter: Eron Wright 


A slot request causes a new worker to be allocated.   If the slot request times 
out or is cancelled before the worker is launched, cancel the worker request if 
possible.

This is an optimization because an idle slot is eventually released anyway.   
However, I observe that a lot of worker requests pile up in the launch 
coordinator, as the JM keeps making request after request.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7362) CheckpointProperties are not correctly set when restoring savepoint with HA enabled

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129564#comment-16129564
 ] 

Ted Yu commented on FLINK-7362:
---

{code}
+   CheckpointProperties ephemeral = new 
CheckpointProperties(false, false, false, true, true, true, true, true);
{code}
There may be more parameter(s) added to CheckpointProperties.

Would it make sense to adopt builder pattern to CheckpointProperties ?

> CheckpointProperties are not correctly set when restoring savepoint with HA 
> enabled
> ---
>
> Key: FLINK-7362
> URL: https://issues.apache.org/jira/browse/FLINK-7362
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Aljoscha Krettek
>Assignee: Stefan Richter
> Fix For: 1.4.0
>
>
> When restoring a savepoint on a HA setup (with ZooKeeper) the web frontend 
> incorrectly says "Type: Checkpoint" in the information box about the latest 
> restore event.
> The information that this uses is set here: 
> https://github.com/apache/flink/blob/09caa9ffdc8168610c7d0260360c034ea87f904c/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1101
> It seems that the {{CheckpointProperties}} of a restored savepoint somehow 
> get lost, maybe because of the recover step that the 
> {{ZookeeperCompletedCheckpointStore}} is going through.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6630) Implement FLIP-6 MesosAppMasterRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129532#comment-16129532
 ] 

ASF GitHub Bot commented on FLINK-6630:
---

GitHub user EronWright opened a pull request:

https://github.com/apache/flink/pull/4555

[FLINK-6630] [Mesos] Implement FLIP-6 MesosAppMasterRunner

## What is the purpose of the change
Implement Mesos runners for FLIP-6.

[FLINK-6630] [Mesos] Implement FLIP-6 MesosAppMasterRunner
[FLINK-6631] [Mesos] Implement FLIP-6 MesosTaskExecutorRunner

## Brief change log

- bin: new entrypoints scripts for flip-6
- ClusterEntrypoint: Refactor the shutdown method
- ClusterEntrypoint: Install default FileSystem (for parity with legacy 
entrypoints)
- ClusterEntrypoint: new MesosJobClusterEntrypoint, 
MesosSessionClusterEntrypoint, MesosEntrypointUtils, MesosTaskExecutorRunner
- MesosServices: enhanced with artifactServer, localActorSystem
- MesosResourceManager: Fallback to old TM params when UNKNOWN resource 
profile is provided
- MesosResourceManager: config setting for taskmanager startup script 
(mesos.resourcemanager.tasks.taskmanager-cmd)
- test: added a 'noop' job graph for testing purposes

## Testing
This change involves manual testing and is verified as follows:
1. Configure Flink to use the FLIP-6 TM runner.  
```
mesos.resourcemanager.tasks.taskmanager-cmd: 
$FLINK_HOME/bin/mesos-taskmanager-flip6.sh
```
2. Configure other Mesos options as normal.
3. Launch a job-specific cluster using `mesos-appmaster-flip6-job.sh`:
```
$ bin/mesos-appmaster-flip6-job.sh 
-Dflink.jobgraph.path=/flink-tests/src/test/resources/jobgraphs/streaming-noop-3.graph
```


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/EronWright/flink FLINK-6630

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4555.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4555


commit 7d257db84f4b9bf2a02d1375a04ff64516266186
Author: Wright, Eron 
Date:   2017-08-16T21:30:24Z

[FLINK-6630] Implement FLIP-6 MesosAppMasterRunner
[FLINK-6631] Implement FLIP-6 MesosTaskExecutorRunner

- bin: new entrypoints scripts for flip-6
- ClusterEntrypoint: Refactor the shutdown method
- ClusterEntrypoint: Install default FileSystem (for parity with legacy 
entrypoints)
- ClusterEntrypoint: new MesosJobClusterEntrypoint, 
MesosSessionClusterEntrypoint, MesosEntrypointUtils, MesosTaskExecutorRunner
- MesosServices: enhanced with artifactServer, localActorSystem
- MesosResourceManager: Fallback to old TM params when UNKNOWN resource 
profile is provided
- MesosResourceManager: config setting for taskmanager startup script 
(mesos.resourcemanager.tasks.taskmanager-cmd)
- test: added a 'noop' job graph for testing purposes




> Implement FLIP-6 MesosAppMasterRunner
> -
>
> Key: FLINK-6630
> URL: https://issues.apache.org/jira/browse/FLINK-6630
> Project: Flink
>  Issue Type: Sub-task
>  Components: Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
>
> A new runner must be developed for the FLIP-6 RM.  Target the "single job" 
> scenario.
> Take some time to consider a general solution or a base implementation that 
> is shared with the old implementation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4555: [FLINK-6630] [Mesos] Implement FLIP-6 MesosAppMast...

[EronWright]

GitHub user EronWright opened a pull request:

https://github.com/apache/flink/pull/4555

[FLINK-6630] [Mesos] Implement FLIP-6 MesosAppMasterRunner

## What is the purpose of the change
Implement Mesos runners for FLIP-6.

[FLINK-6630] [Mesos] Implement FLIP-6 MesosAppMasterRunner
[FLINK-6631] [Mesos] Implement FLIP-6 MesosTaskExecutorRunner

## Brief change log

- bin: new entrypoints scripts for flip-6
- ClusterEntrypoint: Refactor the shutdown method
- ClusterEntrypoint: Install default FileSystem (for parity with legacy 
entrypoints)
- ClusterEntrypoint: new MesosJobClusterEntrypoint, 
MesosSessionClusterEntrypoint, MesosEntrypointUtils, MesosTaskExecutorRunner
- MesosServices: enhanced with artifactServer, localActorSystem
- MesosResourceManager: Fallback to old TM params when UNKNOWN resource 
profile is provided
- MesosResourceManager: config setting for taskmanager startup script 
(mesos.resourcemanager.tasks.taskmanager-cmd)
- test: added a 'noop' job graph for testing purposes

## Testing
This change involves manual testing and is verified as follows:
1. Configure Flink to use the FLIP-6 TM runner.  
```
mesos.resourcemanager.tasks.taskmanager-cmd: 
$FLINK_HOME/bin/mesos-taskmanager-flip6.sh
```
2. Configure other Mesos options as normal.
3. Launch a job-specific cluster using `mesos-appmaster-flip6-job.sh`:
```
$ bin/mesos-appmaster-flip6-job.sh 
-Dflink.jobgraph.path=/flink-tests/src/test/resources/jobgraphs/streaming-noop-3.graph
```


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/EronWright/flink FLINK-6630

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4555.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4555


commit 7d257db84f4b9bf2a02d1375a04ff64516266186
Author: Wright, Eron 
Date:   2017-08-16T21:30:24Z

[FLINK-6630] Implement FLIP-6 MesosAppMasterRunner
[FLINK-6631] Implement FLIP-6 MesosTaskExecutorRunner

- bin: new entrypoints scripts for flip-6
- ClusterEntrypoint: Refactor the shutdown method
- ClusterEntrypoint: Install default FileSystem (for parity with legacy 
entrypoints)
- ClusterEntrypoint: new MesosJobClusterEntrypoint, 
MesosSessionClusterEntrypoint, MesosEntrypointUtils, MesosTaskExecutorRunner
- MesosServices: enhanced with artifactServer, localActorSystem
- MesosResourceManager: Fallback to old TM params when UNKNOWN resource 
profile is provided
- MesosResourceManager: config setting for taskmanager startup script 
(mesos.resourcemanager.tasks.taskmanager-cmd)
- test: added a 'noop' job graph for testing purposes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #4553: [FLINK-7642] [docs] Add very obvious warning about outdat...

[alpinegizmo]

Github user alpinegizmo commented on the issue:

https://github.com/apache/flink/pull/4553
  
+1 I like these improvements.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7442) Add option for using a child-first classloader for loading user code

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129159#comment-16129159
 ] 

ASF GitHub Bot commented on FLINK-7442:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/4554
  
@StephanEwen Could you please have a look since this `ClassLoader` business 
is quite important to get right.


> Add option for using a child-first classloader for loading user code
> 
>
> Key: FLINK-7442
> URL: https://issues.apache.org/jira/browse/FLINK-7442
> Project: Flink
>  Issue Type: Improvement
>  Components: Local Runtime
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4554: [FLINK-7442] Add option for using child-first classloader...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/4554
  
@StephanEwen Could you please have a look since this `ClassLoader` business 
is quite important to get right.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7442) Add option for using a child-first classloader for loading user code

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129158#comment-16129158
 ] 

ASF GitHub Bot commented on FLINK-7442:
---

GitHub user aljoscha opened a pull request:

https://github.com/apache/flink/pull/4554

[FLINK-7442] Add option for using child-first classloader for loading user 
code

## What is the purpose of the change

This PR introduces a new core option (`classloader.resolve-order: 
child-first`) that allows using a child-first class loader for user code. The 
default is still to use a parent-first class loader.

This also does a minor refactoring in the way the blob manager retrieves 
the cleanup interval. It's now also read from the `Configuration`, since we 
already have the `Configuration` for the class loader settings.

## Brief change log

 - Introduce new option
 - Pass `Configuration` thought to all places where we previously created a 
user class loader
 - Instantiate correct class loader based on config

## Verifying this change

This PR introduces new end-to-end tests that verify the new feature in a 
complete Flink workflow, including starting the program using `bin/flink run`.

## Does this pull request potentially affect one of the following parts:


This affects class loader, which is quite important to get right.

## Documentation

 - the new flag is documented in the config documentation


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/flink 
jira-7441-child-first-classloader

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4554.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4554


commit 23405759ec0578060a8a91beb4fbab12e5578607
Author: Aljoscha Krettek 
Date:   2017-08-14T12:53:14Z

[FLINK-7442] Add option for using a child-first classloader for loading 
user code

commit 61a1482baf3c36b939fa5befd08b96edf82a0d95
Author: Aljoscha Krettek 
Date:   2017-08-16T11:28:32Z

Add end-to-end tests




> Add option for using a child-first classloader for loading user code
> 
>
> Key: FLINK-7442
> URL: https://issues.apache.org/jira/browse/FLINK-7442
> Project: Flink
>  Issue Type: Improvement
>  Components: Local Runtime
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4554: [FLINK-7442] Add option for using child-first clas...

[aljoscha]

GitHub user aljoscha opened a pull request:

https://github.com/apache/flink/pull/4554

[FLINK-7442] Add option for using child-first classloader for loading user 
code

## What is the purpose of the change

This PR introduces a new core option (`classloader.resolve-order: 
child-first`) that allows using a child-first class loader for user code. The 
default is still to use a parent-first class loader.

This also does a minor refactoring in the way the blob manager retrieves 
the cleanup interval. It's now also read from the `Configuration`, since we 
already have the `Configuration` for the class loader settings.

## Brief change log

 - Introduce new option
 - Pass `Configuration` thought to all places where we previously created a 
user class loader
 - Instantiate correct class loader based on config

## Verifying this change

This PR introduces new end-to-end tests that verify the new feature in a 
complete Flink workflow, including starting the program using `bin/flink run`.

## Does this pull request potentially affect one of the following parts:


This affects class loader, which is quite important to get right.

## Documentation

 - the new flag is documented in the config documentation


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/flink 
jira-7441-child-first-classloader

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4554.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4554


commit 23405759ec0578060a8a91beb4fbab12e5578607
Author: Aljoscha Krettek 
Date:   2017-08-14T12:53:14Z

[FLINK-7442] Add option for using a child-first classloader for loading 
user code

commit 61a1482baf3c36b939fa5befd08b96edf82a0d95
Author: Aljoscha Krettek 
Date:   2017-08-16T11:28:32Z

Add end-to-end tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7461) Remove Backwards compatibility for Flink 1.1 from Flink 1.4

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129115#comment-16129115
 ] 

ASF GitHub Bot commented on FLINK-7461:
---

Github user dawidwys commented on the issue:

https://github.com/apache/flink/pull/4550
  
Just wanted to add to what @kl0u said that if we remove the compatibility 
with <= 1.2 of CEP library  there will be lot more code that will be no longer 
used than just that in this PR. E.g.

- `NFA#migrateNFA`
- `NFACompiler#migrateGraph`
- `SharedBuffer#migrateSharedBuffer`
- `SharedBuffer#readObject`

just to name a few. I think it would be really important to create a 
following JIRA to introduce those changes in CEP, if we proceed with this PR.


> Remove Backwards compatibility for Flink 1.1 from Flink 1.4
> ---
>
> Key: FLINK-7461
> URL: https://issues.apache.org/jira/browse/FLINK-7461
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> This issue tracks the removal of Flink 1.1 backwards compatibility from Flink 
> 1.4. This step is helpful for further developments because it will remove 
> many old code paths and special cases. In particular, we can drop all 
> handling for non-partitionable state, i.e. state that was created with the 
> old {{Checkpointed}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4550: [FLINK-7461] Remove Backwards compatibility with <= Flink...

[dawidwys]

Github user dawidwys commented on the issue:

https://github.com/apache/flink/pull/4550
  
Just wanted to add to what @kl0u said that if we remove the compatibility 
with <= 1.2 of CEP library  there will be lot more code that will be no longer 
used than just that in this PR. E.g.

- `NFA#migrateNFA`
- `NFACompiler#migrateGraph`
- `SharedBuffer#migrateSharedBuffer`
- `SharedBuffer#readObject`

just to name a few. I think it would be really important to create a 
following JIRA to introduce those changes in CEP, if we proceed with this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6233) Support rowtime inner equi-join between two streams in the SQL API

[Xingcan Cui (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129080#comment-16129080
 ] 

Xingcan Cui commented on FLINK-6233:


Hi [~fhueske], I find my approach for dealing with the time-window was wrong.

After thinking it over and over, finally I draw a conclusion that "*there is no 
absolute window size, and the removing time for a record should be decided by 
the watermarks of the other stream*". I also present a group of 
[figures|https://goo.gl/VW5Gpd] to give an illustration and I think it can 
clearly explain why we should separate the watermarks.

The code will be updated as soon as possible. Sorry for that.

> Support rowtime inner equi-join between two streams in the SQL API
> --
>
> Key: FLINK-6233
> URL: https://issues.apache.org/jira/browse/FLINK-6233
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: hongyuhong
>Assignee: Xingcan Cui
>
> The goal of this issue is to add support for inner equi-join on proc time 
> streams to the SQL interface.
> Queries similar to the following should be supported:
> {code}
> SELECT o.rowtime , o.productId, o.orderId, s.rowtime AS shipTime 
> FROM Orders AS o 
> JOIN Shipments AS s 
> ON o.orderId = s.orderId 
> AND o.rowtime BETWEEN s.rowtime AND s.rowtime + INTERVAL '1' HOUR;
> {code}
> The following restrictions should initially apply:
> * The join hint only support inner join
> * The ON clause should include equi-join condition
> * The time-condition {{o.rowtime BETWEEN s.rowtime AND s.rowtime + INTERVAL 
> '1' HOUR}} only can use rowtime that is a system attribute, the time 
> condition only support bounded time range like {{o.rowtime BETWEEN s.rowtime 
> - INTERVAL '1' HOUR AND s.rowtime + INTERVAL '1' HOUR}}, not support 
> unbounded like {{o.rowtime  s.rowtime}} ,  and  should include both two 
> stream's rowtime attribute, {{o.rowtime between rowtime () and rowtime () + 
> 1}} should also not be supported.
> An row-time streams join will not be able to handle late data, because this 
> would mean in insert a row into a sorted order shift all other computations. 
> This would be too expensive to maintain. Therefore, we will throw an error if 
> a user tries to use an row-time stream join with late data handling.
> This issue includes:
> * Design of the DataStream operator to deal with stream join
> * Translation from Calcite's RelNode representation (LogicalJoin). 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4553: [FLINK-7642] [docs] Add very obvious warning about...

[uce]

GitHub user uce opened a pull request:

https://github.com/apache/flink/pull/4553

[FLINK-7642] [docs] Add very obvious warning about outdated docs

## What is the purpose of the change

This pull requests make the warning about outdated docs more obvious.

![screen shot 2017-08-16 at 18 34 
03](https://user-images.githubusercontent.com/1756620/29374684-93ac1abe-82b2-11e7-9b9b-5008ac33a8e5.png)

Please compare the screenshot to our current state: 
https://ci.apache.org/projects/flink/flink-docs-release-1.1/

If you like this, I would back/forward port this to all versions >= 1.0 and 
update the releasing Wiki page to add a note about updating the configuration 
of the docs.


## Brief change log
- Change the color of the outdated warning footer to red
- Rename the config key from `is_latest` to `show_outdated_warning`
- Add an outdated warning to every page before the actual content

## Verifying this change
- We don't have any tests for the docs
- You can manually check out my branch and build the docs 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/uce/flink 7462-outdated_docs_warning

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4553.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4553


commit e140634687a56fb5fdd0e32a1dce3d5ac41fd123
Author: Ufuk Celebi 
Date:   2017-08-16T16:21:49Z

[FLINK-7462] [docs] Make outdated docs footer background light red

commit 15fe71e0402d2e2d931485497338973e12cce9db
Author: Ufuk Celebi 
Date:   2017-08-16T16:34:51Z

[FLINK-7462] [docs] Rename is_latest flag to show_outdated_warning

commit 9bdeeae0cb88b59f0dfaee547fbf9b54d125645e
Author: Ufuk Celebi 
Date:   2017-08-16T16:35:03Z

[FLINK-7462] [docs] Add outdated warning to content




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7057) move BLOB ref-counting from LibraryCacheManager to BlobCache

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129057#comment-16129057
 ] 

ASF GitHub Bot commented on FLINK-7057:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4238#discussion_r133503488
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java 
---
@@ -148,7 +149,10 @@
/** Service to contend for and retrieve the leadership of JM and RM */
private final HighAvailabilityServices highAvailabilityServices;
 
-   /** Blob cache manager used across jobs */
+   /** Blob server used across jobs */
+   private final BlobServer blobServer;
--- End diff --

I think the `JobMaster` should not depend on the `BlobServer` because the 
`BlobServer` might run on a different node (along side to the dispatcher, for 
example). Can we pass in the `BlobService` which can either be a `BlobServer` 
or a `BlobCache`? I think we have to move the `register/releaseJob` methods to 
the `BlobService` for that. Additionally, the `BlobServer` should simply do 
nothing when these methods are called. That way, we can also pass in a 
`BlobService` to the `Task` and the `TaskExecutor/TaskManager`


> move BLOB ref-counting from LibraryCacheManager to BlobCache
> 
>
> Key: FLINK-7057
> URL: https://issues.apache.org/jira/browse/FLINK-7057
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination, Network
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> Currently, the {{LibraryCacheManager}} is doing some ref-counting for JAR 
> files managed by it. Instead, we want the {{BlobCache}} to do that itself for 
> all job-related BLOBs. Also, we do not want to operate on a per-{{BlobKey}} 
> level but rather per job. Therefore, the cleanup process should be adapted, 
> too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7057) move BLOB ref-counting from LibraryCacheManager to BlobCache

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129056#comment-16129056
 ] 

ASF GitHub Bot commented on FLINK-7057:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4238#discussion_r133494977
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/BlobServerOptions.java 
---
@@ -73,4 +73,9 @@
public static final ConfigOption SSL_ENABLED =
key("blob.service.ssl.enabled")
.defaultValue(true);
+
+   public static final ConfigOption CLEANUP_INTERVAL =
+   key("blob.service.cleanup.interval")
--- End diff --

We should document this configuration parameter in the `config.md`


> move BLOB ref-counting from LibraryCacheManager to BlobCache
> 
>
> Key: FLINK-7057
> URL: https://issues.apache.org/jira/browse/FLINK-7057
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination, Network
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> Currently, the {{LibraryCacheManager}} is doing some ref-counting for JAR 
> files managed by it. Instead, we want the {{BlobCache}} to do that itself for 
> all job-related BLOBs. Also, we do not want to operate on a per-{{BlobKey}} 
> level but rather per job. Therefore, the cleanup process should be adapted, 
> too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7057) move BLOB ref-counting from LibraryCacheManager to BlobCache

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129055#comment-16129055
 ] 

ASF GitHub Bot commented on FLINK-7057:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4238#discussion_r133497146
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCache.java ---
@@ -341,8 +424,39 @@ public int getPort() {
return serverAddress.getPort();
}
 
+   /**
+* Cleans up BLOBs which are not referenced anymore.
+*/
+   @Override
+   public void run() {
+   synchronized (jobRefCounters) {
+   Iterator> entryIter = 
jobRefCounters.entrySet().iterator();
+
+   while (entryIter.hasNext()) {
+   Map.Entry entry = 
entryIter.next();
+   RefCount ref = entry.getValue();
+
+   if (ref.references <= 0 && ref.keepUntil > 0 && 
System.currentTimeMillis() >= ref.keepUntil) {
--- End diff --

We could store the current time so that we don't retrieve it for every 
entry.


> move BLOB ref-counting from LibraryCacheManager to BlobCache
> 
>
> Key: FLINK-7057
> URL: https://issues.apache.org/jira/browse/FLINK-7057
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination, Network
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> Currently, the {{LibraryCacheManager}} is doing some ref-counting for JAR 
> files managed by it. Instead, we want the {{BlobCache}} to do that itself for 
> all job-related BLOBs. Also, we do not want to operate on a per-{{BlobKey}} 
> level but rather per job. Therefore, the cleanup process should be adapted, 
> too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7057) move BLOB ref-counting from LibraryCacheManager to BlobCache

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129054#comment-16129054
 ] 

ASF GitHub Bot commented on FLINK-7057:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4238#discussion_r133500735
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/JobManagerConnection.java
 ---
@@ -63,21 +67,22 @@
private final PartitionProducerStateChecker partitionStateChecker;
 
public JobManagerConnection(
-   JobID jobID,
-   ResourceID resourceID,
-   JobMasterGateway jobMasterGateway,
-   UUID leaderId,
-   TaskManagerActions taskManagerActions,
-   CheckpointResponder checkpointResponder,
-   LibraryCacheManager libraryCacheManager,
-   ResultPartitionConsumableNotifier 
resultPartitionConsumableNotifier,
-   PartitionProducerStateChecker partitionStateChecker) {
+   JobID jobID,
+   ResourceID resourceID,
+   JobMasterGateway jobMasterGateway,
+   UUID leaderId,
+   TaskManagerActions taskManagerActions,
+   CheckpointResponder checkpointResponder,
+   BlobCache blobCache, LibraryCacheManager 
libraryCacheManager,
--- End diff --

Formatting


> move BLOB ref-counting from LibraryCacheManager to BlobCache
> 
>
> Key: FLINK-7057
> URL: https://issues.apache.org/jira/browse/FLINK-7057
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination, Network
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> Currently, the {{LibraryCacheManager}} is doing some ref-counting for JAR 
> files managed by it. Instead, we want the {{BlobCache}} to do that itself for 
> all job-related BLOBs. Also, we do not want to operate on a per-{{BlobKey}} 
> level but rather per job. Therefore, the cleanup process should be adapted, 
> too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4238: [FLINK-7057][blob] move BLOB ref-counting from Lib...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4238#discussion_r133494977
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/BlobServerOptions.java 
---
@@ -73,4 +73,9 @@
public static final ConfigOption SSL_ENABLED =
key("blob.service.ssl.enabled")
.defaultValue(true);
+
+   public static final ConfigOption CLEANUP_INTERVAL =
+   key("blob.service.cleanup.interval")
--- End diff --

We should document this configuration parameter in the `config.md`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4238: [FLINK-7057][blob] move BLOB ref-counting from Lib...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4238#discussion_r133500735
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/JobManagerConnection.java
 ---
@@ -63,21 +67,22 @@
private final PartitionProducerStateChecker partitionStateChecker;
 
public JobManagerConnection(
-   JobID jobID,
-   ResourceID resourceID,
-   JobMasterGateway jobMasterGateway,
-   UUID leaderId,
-   TaskManagerActions taskManagerActions,
-   CheckpointResponder checkpointResponder,
-   LibraryCacheManager libraryCacheManager,
-   ResultPartitionConsumableNotifier 
resultPartitionConsumableNotifier,
-   PartitionProducerStateChecker partitionStateChecker) {
+   JobID jobID,
+   ResourceID resourceID,
+   JobMasterGateway jobMasterGateway,
+   UUID leaderId,
+   TaskManagerActions taskManagerActions,
+   CheckpointResponder checkpointResponder,
+   BlobCache blobCache, LibraryCacheManager 
libraryCacheManager,
--- End diff --

Formatting


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4238: [FLINK-7057][blob] move BLOB ref-counting from Lib...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4238#discussion_r133497146
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobCache.java ---
@@ -341,8 +424,39 @@ public int getPort() {
return serverAddress.getPort();
}
 
+   /**
+* Cleans up BLOBs which are not referenced anymore.
+*/
+   @Override
+   public void run() {
+   synchronized (jobRefCounters) {
+   Iterator> entryIter = 
jobRefCounters.entrySet().iterator();
+
+   while (entryIter.hasNext()) {
+   Map.Entry entry = 
entryIter.next();
+   RefCount ref = entry.getValue();
+
+   if (ref.references <= 0 && ref.keepUntil > 0 && 
System.currentTimeMillis() >= ref.keepUntil) {
--- End diff --

We could store the current time so that we don't retrieve it for every 
entry.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4238: [FLINK-7057][blob] move BLOB ref-counting from Lib...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/4238#discussion_r133503488
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java 
---
@@ -148,7 +149,10 @@
/** Service to contend for and retrieve the leadership of JM and RM */
private final HighAvailabilityServices highAvailabilityServices;
 
-   /** Blob cache manager used across jobs */
+   /** Blob server used across jobs */
+   private final BlobServer blobServer;
--- End diff --

I think the `JobMaster` should not depend on the `BlobServer` because the 
`BlobServer` might run on a different node (along side to the dispatcher, for 
example). Can we pass in the `BlobService` which can either be a `BlobServer` 
or a `BlobCache`? I think we have to move the `register/releaseJob` methods to 
the `BlobService` for that. Additionally, the `BlobServer` should simply do 
nothing when these methods are called. That way, we can also pass in a 
`BlobService` to the `Task` and the `TaskExecutor/TaskManager`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7456) Implement Netty sender incoming pipeline for credit-based

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129035#comment-16129035
 ] 

ASF GitHub Bot commented on FLINK-7456:
---

GitHub user zhijiangW opened a pull request:

https://github.com/apache/flink/pull/4552

[FLINK-7456][network]Implement Netty sender incoming pipeline for 
credit-based

## What is the purpose of the change

This PR is based on #4533 whose commits are also included for passing 
travis. Review the last commit for this PR change.

On sender side, it maintains credit from receiver's `PartitionRequest` and 
`AddCredit` messages, then sends buffer based on credit and network capacity. 
This PR is mainly involved in incoming pipeline logic for credit-based.

## Brief change log

  - *Each subpartition view maintains current credit and a boolean field to 
mark whether it is already registered available for transfer*
  - *Update current credit in processing `PartitionRequest` and `AddCredit` 
messages*
  - *The mechanism of enqueue the subpartition view and update the 
registered status field*

## Verifying this change

This change added tests and can be verified as follows:

  - *Added test to verify that current credit is updated correctly and 
subpartition view is enqueued when received `AddCredit` message*

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhijiangW/flink FLINK-7456

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4552.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4552


commit e35c1ff8066bf44344495d132a1092b9db3ef182
Author: Zhijiang 
Date:   2017-08-07T09:31:17Z

[FLINK-7378][core]Create a fix size (non rebalancing) buffer pool type for 
the floating buffers

commit 969c24d3bf80c1ff89ada11e81b9bf4fea14066f
Author: Zhijiang 
Date:   2017-08-14T06:30:47Z

[FLINK-7394][core]Implement basic InputChannel with free buffers,credit and 
backlog

commit 15fa828449d73f53042c57e9c5494d75ddee575f
Author: Zhijiang 
Date:   2017-08-10T05:29:13Z

[FLINK-7406][network]Implement Netty receiver incoming pipeline for 
credit-based

commit d0674244f15701863a5dd3f68b7274b3bd49c64d
Author: Zhijiang 
Date:   2017-08-12T14:13:25Z

[FLINK-7416][network] Implement Netty receiver outgoing pipeline for 
credit-based

commit 6eaff7877ad43eab674e184153365b50ec8e1559
Author: Zhijiang 
Date:   2017-08-16T13:24:53Z

[FLINK-7456][network]Implement Netty sender incoming pipeline for 
credit-based




> Implement Netty sender incoming pipeline for credit-based
> -
>
> Key: FLINK-7456
> URL: https://issues.apache.org/jira/browse/FLINK-7456
> Project: Flink
>  Issue Type: Sub-task
>  Components: Network
>Reporter: zhijiang
>Assignee: zhijiang
> Fix For: 1.4.0
>
>
> This is a part of work for credit-based network flow control.
> On sender side, each subpartition view maintains an atomic integer 
> {{currentCredit}} from receiver. Once receiving the messages of 
> {{PartitionRequest}} and {{AddCredit}}, the {{currentCredit}} is added by 
> deltas.
> Each view also maintains an atomic boolean field to mark it as registered 
> available for transfer to make sure it is enqueued in handler only once. If 
> the {{currentCredit}} increases from zero and there are available buffers in 
> the subpartition, the corresponding view will be enqueued for transferring 
> data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4552: [FLINK-7456][network]Implement Netty sender incomi...

[zhijiangW]

GitHub user zhijiangW opened a pull request:

https://github.com/apache/flink/pull/4552

[FLINK-7456][network]Implement Netty sender incoming pipeline for 
credit-based

## What is the purpose of the change

This PR is based on #4533 whose commits are also included for passing 
travis. Review the last commit for this PR change.

On sender side, it maintains credit from receiver's `PartitionRequest` and 
`AddCredit` messages, then sends buffer based on credit and network capacity. 
This PR is mainly involved in incoming pipeline logic for credit-based.

## Brief change log

  - *Each subpartition view maintains current credit and a boolean field to 
mark whether it is already registered available for transfer*
  - *Update current credit in processing `PartitionRequest` and `AddCredit` 
messages*
  - *The mechanism of enqueue the subpartition view and update the 
registered status field*

## Verifying this change

This change added tests and can be verified as follows:

  - *Added test to verify that current credit is updated correctly and 
subpartition view is enqueued when received `AddCredit` message*

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhijiangW/flink FLINK-7456

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4552.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4552


commit e35c1ff8066bf44344495d132a1092b9db3ef182
Author: Zhijiang 
Date:   2017-08-07T09:31:17Z

[FLINK-7378][core]Create a fix size (non rebalancing) buffer pool type for 
the floating buffers

commit 969c24d3bf80c1ff89ada11e81b9bf4fea14066f
Author: Zhijiang 
Date:   2017-08-14T06:30:47Z

[FLINK-7394][core]Implement basic InputChannel with free buffers,credit and 
backlog

commit 15fa828449d73f53042c57e9c5494d75ddee575f
Author: Zhijiang 
Date:   2017-08-10T05:29:13Z

[FLINK-7406][network]Implement Netty receiver incoming pipeline for 
credit-based

commit d0674244f15701863a5dd3f68b7274b3bd49c64d
Author: Zhijiang 
Date:   2017-08-12T14:13:25Z

[FLINK-7416][network] Implement Netty receiver outgoing pipeline for 
credit-based

commit 6eaff7877ad43eab674e184153365b50ec8e1559
Author: Zhijiang 
Date:   2017-08-16T13:24:53Z

[FLINK-7456][network]Implement Netty sender incoming pipeline for 
credit-based




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7367) Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, MaxConnections, RequestTimeout, etc)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129016#comment-16129016
 ] 

ASF GitHub Bot commented on FLINK-7367:
---

Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/4473
  
@tzulitai done!


> Parameterize more configs for FlinkKinesisProducer (RecordMaxBufferedTime, 
> MaxConnections, RequestTimeout, etc)
> ---
>
> Key: FLINK-7367
> URL: https://issues.apache.org/jira/browse/FLINK-7367
> Project: Flink
>  Issue Type: Bug
>  Components: Kinesis Connector
>Affects Versions: 1.3.0
>Reporter: Bowen Li
>Assignee: Bowen Li
> Fix For: 1.4.0, 1.3.3
>
>
> Right now, FlinkKinesisProducer only expose two configs for the underlying 
> KinesisProducer:
> - AGGREGATION_MAX_COUNT
> - COLLECTION_MAX_COUNT
> Well, according to [AWS 
> doc|http://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-config.html] 
> and [their sample on 
> github|https://github.com/awslabs/amazon-kinesis-producer/blob/master/java/amazon-kinesis-producer-sample/default_config.properties],
>  developers can set more to make the max use of KinesisProducer, and make it 
> fault-tolerant (e.g. by increasing timeout).
> I select a few more configs that we need when using Flink with Kinesis:
> - MAX_CONNECTIONS
> - RATE_LIMIT
> - RECORD_MAX_BUFFERED_TIME
> - RECORD_TIME_TO_LIVE
> - REQUEST_TIMEOUT
> Flink is using KPL's default values. They make Flink writing too fast to 
> Kinesis, which fail Flink job too frequently. We need to parameterize 
> FlinkKinesisProducer to pass in the above params, in order to slowing down 
> Flink's write rate to Kinesis.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4473: [FLINK-7367][kinesis connector] Parameterize more configs...

[bowenli86]

Github user bowenli86 commented on the issue:

https://github.com/apache/flink/pull/4473
  
@tzulitai done!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7461) Remove Backwards compatibility for Flink 1.1 from Flink 1.4

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129012#comment-16129012
 ] 

ASF GitHub Bot commented on FLINK-7461:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4550#discussion_r133490898
  
--- Diff: 
flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/testutils/ExactlyOnceValidatingConsumerThread.java
 ---
@@ -95,7 +94,7 @@ public void run() {
return new Thread(exactlyOnceValidationConsumer);
}
 
-   private static class ExactlyOnceValidatingMapper implements 
FlatMapFunction, Checkpointed {
+   private static class ExactlyOnceValidatingMapper implements 
FlatMapFunction {
--- End diff --

This _should_ cause the test to fail as far as i can tell.


> Remove Backwards compatibility for Flink 1.1 from Flink 1.4
> ---
>
> Key: FLINK-7461
> URL: https://issues.apache.org/jira/browse/FLINK-7461
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> This issue tracks the removal of Flink 1.1 backwards compatibility from Flink 
> 1.4. This step is helpful for further developments because it will remove 
> many old code paths and special cases. In particular, we can drop all 
> handling for non-partitionable state, i.e. state that was created with the 
> old {{Checkpointed}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4550: [FLINK-7461] Remove Backwards compatibility with <...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4550#discussion_r133493395
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/StateUtil.java ---
@@ -49,27 +49,8 @@ public static long getStateSize(StateObject handle) {
 * @throws Exception exception that is a collection of all suppressed 
exceptions that were caught during iteration
 */
public static void bestEffortDiscardAllStateObjects(
-   Iterable handlesToDiscard) 
throws Exception {
-
-   if (handlesToDiscard != null) {
-   Exception exception = null;
-
-   for (StateObject state : handlesToDiscard) {
-
-   if (state != null) {
-   try {
-   state.discardState();
-   }
-   catch (Exception ex) {
-   exception = 
ExceptionUtils.firstOrSuppressed(ex, exception);
-   }
-   }
-   }
-
-   if (exception != null) {
-   throw exception;
-   }
-   }
+   Iterable handlesToDiscard) throws 
Exception {
+   
LambdaUtil.applyToAllWhileSuppressingExceptions(handlesToDiscard, 
StateObject::discardState);
--- End diff --

Is this change really necessary?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4550: [FLINK-7461] Remove Backwards compatibility with <...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4550#discussion_r133490230
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
 ---
@@ -43,17 +40,17 @@
  * This class encapsulates the state for one parallel instance of an 
operator. The complete state of a (logical)
  * operator (e.g. a flatmap operator) consists of the union of all {@link 
OperatorSubtaskState}s from all
  * parallel tasks that physically execute parallelized, physical instances 
of the operator.
- *
+ * 
--- End diff --

this should be an empty line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7461) Remove Backwards compatibility for Flink 1.1 from Flink 1.4

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129008#comment-16129008
 ] 

ASF GitHub Bot commented on FLINK-7461:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4550#discussion_r133490130
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
 ---
@@ -228,12 +195,11 @@ public StreamStateHandle getLegacyOperatorState() {
public void discardState() {
try {
List toDispose =
-   new ArrayList<>(1 +
+   new ArrayList<>(
managedOperatorState.size() +
-   rawOperatorState.size() +
-   managedKeyedState.size() +
-   rawKeyedState.size());
-   toDispose.add(legacyOperatorState);
+   rawOperatorState.size() +
--- End diff --

odd indentation


> Remove Backwards compatibility for Flink 1.1 from Flink 1.4
> ---
>
> Key: FLINK-7461
> URL: https://issues.apache.org/jira/browse/FLINK-7461
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> This issue tracks the removal of Flink 1.1 backwards compatibility from Flink 
> 1.4. This step is helpful for further developments because it will remove 
> many old code paths and special cases. In particular, we can drop all 
> handling for non-partitionable state, i.e. state that was created with the 
> old {{Checkpointed}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7461) Remove Backwards compatibility for Flink 1.1 from Flink 1.4

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129010#comment-16129010
 ] 

ASF GitHub Bot commented on FLINK-7461:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4550#discussion_r133492522
  
--- Diff: 
flink-contrib/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackendTest.java
 ---
@@ -122,6 +124,22 @@ protected RocksDBStateBackend getStateBackend() throws 
IOException {
return backend;
}
 
+   // small safety net for instance cleanups, so that no native objects 
are left
--- End diff --

are these changes necessary for this PR?


> Remove Backwards compatibility for Flink 1.1 from Flink 1.4
> ---
>
> Key: FLINK-7461
> URL: https://issues.apache.org/jira/browse/FLINK-7461
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> This issue tracks the removal of Flink 1.1 backwards compatibility from Flink 
> 1.4. This step is helpful for further developments because it will remove 
> many old code paths and special cases. In particular, we can drop all 
> handling for non-partitionable state, i.e. state that was created with the 
> old {{Checkpointed}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7461) Remove Backwards compatibility for Flink 1.1 from Flink 1.4

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129009#comment-16129009
 ] 

ASF GitHub Bot commented on FLINK-7461:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4550#discussion_r133490230
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
 ---
@@ -43,17 +40,17 @@
  * This class encapsulates the state for one parallel instance of an 
operator. The complete state of a (logical)
  * operator (e.g. a flatmap operator) consists of the union of all {@link 
OperatorSubtaskState}s from all
  * parallel tasks that physically execute parallelized, physical instances 
of the operator.
- *
+ * 
--- End diff --

this should be an empty line


> Remove Backwards compatibility for Flink 1.1 from Flink 1.4
> ---
>
> Key: FLINK-7461
> URL: https://issues.apache.org/jira/browse/FLINK-7461
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> This issue tracks the removal of Flink 1.1 backwards compatibility from Flink 
> 1.4. This step is helpful for further developments because it will remove 
> many old code paths and special cases. In particular, we can drop all 
> handling for non-partitionable state, i.e. state that was created with the 
> old {{Checkpointed}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4550: [FLINK-7461] Remove Backwards compatibility with <...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4550#discussion_r133490898
  
--- Diff: 
flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/testutils/ExactlyOnceValidatingConsumerThread.java
 ---
@@ -95,7 +94,7 @@ public void run() {
return new Thread(exactlyOnceValidationConsumer);
}
 
-   private static class ExactlyOnceValidatingMapper implements 
FlatMapFunction, Checkpointed {
+   private static class ExactlyOnceValidatingMapper implements 
FlatMapFunction {
--- End diff --

This _should_ cause the test to fail as far as i can tell.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7461) Remove Backwards compatibility for Flink 1.1 from Flink 1.4

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129011#comment-16129011
 ] 

ASF GitHub Bot commented on FLINK-7461:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4550#discussion_r133493395
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/StateUtil.java ---
@@ -49,27 +49,8 @@ public static long getStateSize(StateObject handle) {
 * @throws Exception exception that is a collection of all suppressed 
exceptions that were caught during iteration
 */
public static void bestEffortDiscardAllStateObjects(
-   Iterable handlesToDiscard) 
throws Exception {
-
-   if (handlesToDiscard != null) {
-   Exception exception = null;
-
-   for (StateObject state : handlesToDiscard) {
-
-   if (state != null) {
-   try {
-   state.discardState();
-   }
-   catch (Exception ex) {
-   exception = 
ExceptionUtils.firstOrSuppressed(ex, exception);
-   }
-   }
-   }
-
-   if (exception != null) {
-   throw exception;
-   }
-   }
+   Iterable handlesToDiscard) throws 
Exception {
+   
LambdaUtil.applyToAllWhileSuppressingExceptions(handlesToDiscard, 
StateObject::discardState);
--- End diff --

Is this change really necessary?


> Remove Backwards compatibility for Flink 1.1 from Flink 1.4
> ---
>
> Key: FLINK-7461
> URL: https://issues.apache.org/jira/browse/FLINK-7461
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> This issue tracks the removal of Flink 1.1 backwards compatibility from Flink 
> 1.4. This step is helpful for further developments because it will remove 
> many old code paths and special cases. In particular, we can drop all 
> handling for non-partitionable state, i.e. state that was created with the 
> old {{Checkpointed}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4550: [FLINK-7461] Remove Backwards compatibility with <...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4550#discussion_r133490130
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
 ---
@@ -228,12 +195,11 @@ public StreamStateHandle getLegacyOperatorState() {
public void discardState() {
try {
List toDispose =
-   new ArrayList<>(1 +
+   new ArrayList<>(
managedOperatorState.size() +
-   rawOperatorState.size() +
-   managedKeyedState.size() +
-   rawKeyedState.size());
-   toDispose.add(legacyOperatorState);
+   rawOperatorState.size() +
--- End diff --

odd indentation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4550: [FLINK-7461] Remove Backwards compatibility with <...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4550#discussion_r133492522
  
--- Diff: 
flink-contrib/flink-statebackend-rocksdb/src/test/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackendTest.java
 ---
@@ -122,6 +124,22 @@ protected RocksDBStateBackend getStateBackend() throws 
IOException {
return backend;
}
 
+   // small safety net for instance cleanups, so that no native objects 
are left
--- End diff --

are these changes necessary for this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7057) move BLOB ref-counting from LibraryCacheManager to BlobCache

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129002#comment-16129002
 ] 

ASF GitHub Bot commented on FLINK-7057:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4238
  
Would be good to rebase again @NicoK because otherwise I don't see what 
Travis thinks about the PR.


> move BLOB ref-counting from LibraryCacheManager to BlobCache
> 
>
> Key: FLINK-7057
> URL: https://issues.apache.org/jira/browse/FLINK-7057
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination, Network
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> Currently, the {{LibraryCacheManager}} is doing some ref-counting for JAR 
> files managed by it. Instead, we want the {{BlobCache}} to do that itself for 
> all job-related BLOBs. Also, we do not want to operate on a per-{{BlobKey}} 
> level but rather per job. Therefore, the cleanup process should be adapted, 
> too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4238: [FLINK-7057][blob] move BLOB ref-counting from LibraryCac...

[tillrohrmann]

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/4238
  
Would be good to rebase again @NicoK because otherwise I don't see what 
Travis thinks about the PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-7462) Add very obvious warning about outdated docs

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-7462:
--

 Summary: Add very obvious warning about outdated docs
 Key: FLINK-7462
 URL: https://issues.apache.org/jira/browse/FLINK-7462
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Ufuk Celebi
Assignee: Ufuk Celebi


The current warning for outdated docs is not very obvious in the footer of the 
page. I would like to increase the visibility of this by adjusting this footer 
and adding a warning to actual content.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7459) Introduce RedirectHandler for the WebRuntimeMonitor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128984#comment-16128984
 ] 

ASF GitHub Bot commented on FLINK-7459:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4551

[FLINK-7459] Generalize Flink's redirection logic

## What is the purpose of the change

Introduce RedirectHandler which can be extended to add redirection 
functionality to all
SimpleInboundChannelHandlers. This allows to share the same functionality 
across the
StaticFileServerHandler and the RuntimeMonitorHandlerBase which could now 
be removed.
In the future, the AbstractRestHandler will also extend the RedirectHandler.

This PR is based on #4549.

## Brief change log

- Introduce `RedirectHandler` which encapsulates the redirection logic
- Let `StaticFileServerHandler` and `RuntimeMonitorHandler` extend the 
`RedirectHandler`

## Verifying this change

This change added tests and can be verified as follows:

- Added test `RedirectHandlerTest` which tests the redirection logic

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink generalizeWebRedirect

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4551.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4551


commit 2c0c36244f75de000f884618330bc2f885c91d76
Author: Till Rohrmann 
Date:   2017-08-10T08:56:12Z

[FLINK-7409] [web] Make WebRuntimeMonitor reactive

This commit changes the behaviour of the WebRuntimeMonitor to not longer 
block serving
threads by waiting on the result of futures. Instead the RequestHandler now 
returns a
CompletableFuture which is written out to the Netty 
channel upon
completion. This will improve the performance of our WebRuntimeMonitor.

commit 01c877a24d65b8d75c49696bc8ab9e254d99
Author: Till Rohrmann 
Date:   2017-08-15T10:00:58Z

[FLINK-7458] Generalize GatewayRetriever for WebRuntimeMonitor

Introduce a generalized GatewayRetriever replacing the JobManagerRetriever. 
The
GatewayRetriever fulfills the same purpose as the JobManagerRetriever with 
the
ability to retrieve the gateway for an arbitrary endpoint type.

commit b44bc2eb0085e7e7f3b4d13e1942ceb28a0e00c8
Author: Till Rohrmann 
Date:   2017-08-15T11:55:47Z

[FLINK-7459] Generalize Flink's redirection logic

Introduce RedirectHandler which can be extended to add redirection 
functionality to all
SimpleInboundChannelHandlers. This allows to share the same functionality 
across the
StaticFileServerHandler and the RuntimeMonitorHandlerBase which could now 
be removed.
In the future, the AbstractRestHandler will also extend the RedirectHandler.




> Introduce RedirectHandler for the WebRuntimeMonitor
> ---
>
> Key: FLINK-7459
> URL: https://issues.apache.org/jira/browse/FLINK-7459
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination, Webfrontend
>Affects Versions: 1.4.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>  Labels: flip-6
>
> Currently, the {{StaticFileServerHandler}} and the 
> {{RuntimeMonitorHandlerBase}} both have functionality to redirect the 
> requests to the current leader. In the future, there will be an 
> {{AbstractRestHandler}} for Flip-6 which also needs this functionality. 
> I therefore propose to create a general purpose {{RedirectHandler}} which can 
> be extended to automatically add support for redirection.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4551: [FLINK-7459] Generalize Flink's redirection logic

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4551

[FLINK-7459] Generalize Flink's redirection logic

## What is the purpose of the change

Introduce RedirectHandler which can be extended to add redirection 
functionality to all
SimpleInboundChannelHandlers. This allows to share the same functionality 
across the
StaticFileServerHandler and the RuntimeMonitorHandlerBase which could now 
be removed.
In the future, the AbstractRestHandler will also extend the RedirectHandler.

This PR is based on #4549.

## Brief change log

- Introduce `RedirectHandler` which encapsulates the redirection logic
- Let `StaticFileServerHandler` and `RuntimeMonitorHandler` extend the 
`RedirectHandler`

## Verifying this change

This change added tests and can be verified as follows:

- Added test `RedirectHandlerTest` which tests the redirection logic

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink generalizeWebRedirect

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4551.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4551


commit 2c0c36244f75de000f884618330bc2f885c91d76
Author: Till Rohrmann 
Date:   2017-08-10T08:56:12Z

[FLINK-7409] [web] Make WebRuntimeMonitor reactive

This commit changes the behaviour of the WebRuntimeMonitor to not longer 
block serving
threads by waiting on the result of futures. Instead the RequestHandler now 
returns a
CompletableFuture which is written out to the Netty 
channel upon
completion. This will improve the performance of our WebRuntimeMonitor.

commit 01c877a24d65b8d75c49696bc8ab9e254d99
Author: Till Rohrmann 
Date:   2017-08-15T10:00:58Z

[FLINK-7458] Generalize GatewayRetriever for WebRuntimeMonitor

Introduce a generalized GatewayRetriever replacing the JobManagerRetriever. 
The
GatewayRetriever fulfills the same purpose as the JobManagerRetriever with 
the
ability to retrieve the gateway for an arbitrary endpoint type.

commit b44bc2eb0085e7e7f3b4d13e1942ceb28a0e00c8
Author: Till Rohrmann 
Date:   2017-08-15T11:55:47Z

[FLINK-7459] Generalize Flink's redirection logic

Introduce RedirectHandler which can be extended to add redirection 
functionality to all
SimpleInboundChannelHandlers. This allows to share the same functionality 
across the
StaticFileServerHandler and the RuntimeMonitorHandlerBase which could now 
be removed.
In the future, the AbstractRestHandler will also extend the RedirectHandler.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7461) Remove Backwards compatibility for Flink 1.1 from Flink 1.4

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128966#comment-16128966
 ] 

ASF GitHub Bot commented on FLINK-7461:
---

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/4550
  
I agree that this is a change that we agreed upon. My only point is that we 
have to communicate it also in the ML. Probably with a thread that points to 
this JIRA and the PR and mentions this issue.


> Remove Backwards compatibility for Flink 1.1 from Flink 1.4
> ---
>
> Key: FLINK-7461
> URL: https://issues.apache.org/jira/browse/FLINK-7461
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> This issue tracks the removal of Flink 1.1 backwards compatibility from Flink 
> 1.4. This step is helpful for further developments because it will remove 
> many old code paths and special cases. In particular, we can drop all 
> handling for non-partitionable state, i.e. state that was created with the 
> old {{Checkpointed}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4550: [FLINK-7461] Remove Backwards compatibility with <= Flink...

[kl0u]

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/4550
  
I agree that this is a change that we agreed upon. My only point is that we 
have to communicate it also in the ML. Probably with a thread that points to 
this JIRA and the PR and mentions this issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #4550: [FLINK-7461] Remove Backwards compatibility with <= Flink...

[StefanRRichter]

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/4550
  
Yes, that is true, but I think it should because otherwise there is still 
no way to remove the `Checkpointed` / legacy state related code paths.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7461) Remove Backwards compatibility for Flink 1.1 from Flink 1.4

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128962#comment-16128962
 ] 

ASF GitHub Bot commented on FLINK-7461:
---

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/4550
  
Yes, that is true, but I think it should because otherwise there is still 
no way to remove the `Checkpointed` / legacy state related code paths.


> Remove Backwards compatibility for Flink 1.1 from Flink 1.4
> ---
>
> Key: FLINK-7461
> URL: https://issues.apache.org/jira/browse/FLINK-7461
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> This issue tracks the removal of Flink 1.1 backwards compatibility from Flink 
> 1.4. This step is helpful for further developments because it will remove 
> many old code paths and special cases. In particular, we can drop all 
> handling for non-partitionable state, i.e. state that was created with the 
> old {{Checkpointed}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7461) Remove Backwards compatibility for Flink 1.1 from Flink 1.4

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128955#comment-16128955
 ] 

ASF GitHub Bot commented on FLINK-7461:
---

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/4550
  
From a brief check, I see that this also removed backwards compatibility 
also for 1.2 for the CEP library. The reason is that the CEP library in Flink 
1.2 was the same as in Flink 1.1 (no upgrade) and so it was using the old state 
abstractions.

Given that in the ML we discussed about dropping compatibility with 1.1 and 
agreed on that, a library should not affect that development, but there must be 
an explicit discussion in the ML for this specific matter before merging it. 
Even if this just means just saying that you have to go through 1.3 for CEP if 
you want to migrate from 1.2 to 1.4.



> Remove Backwards compatibility for Flink 1.1 from Flink 1.4
> ---
>
> Key: FLINK-7461
> URL: https://issues.apache.org/jira/browse/FLINK-7461
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> This issue tracks the removal of Flink 1.1 backwards compatibility from Flink 
> 1.4. This step is helpful for further developments because it will remove 
> many old code paths and special cases. In particular, we can drop all 
> handling for non-partitionable state, i.e. state that was created with the 
> old {{Checkpointed}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4550: [FLINK-7461] Remove Backwards compatibility with <= Flink...

[kl0u]

Github user kl0u commented on the issue:

https://github.com/apache/flink/pull/4550
  
From a brief check, I see that this also removed backwards compatibility 
also for 1.2 for the CEP library. The reason is that the CEP library in Flink 
1.2 was the same as in Flink 1.1 (no upgrade) and so it was using the old state 
abstractions.

Given that in the ML we discussed about dropping compatibility with 1.1 and 
agreed on that, a library should not affect that development, but there must be 
an explicit discussion in the ML for this specific matter before merging it. 
Even if this just means just saying that you have to go through 1.3 for CEP if 
you want to migrate from 1.2 to 1.4.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7461) Remove Backwards compatibility for Flink 1.1 from Flink 1.4

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128950#comment-16128950
 ] 

ASF GitHub Bot commented on FLINK-7461:
---

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/4550
  
CC @aljoscha @StephanEwen ; the relevant commit is only 
95e44099784c9deaf2ca422b8dfc11c3d67d7f82 .


> Remove Backwards compatibility for Flink 1.1 from Flink 1.4
> ---
>
> Key: FLINK-7461
> URL: https://issues.apache.org/jira/browse/FLINK-7461
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> This issue tracks the removal of Flink 1.1 backwards compatibility from Flink 
> 1.4. This step is helpful for further developments because it will remove 
> many old code paths and special cases. In particular, we can drop all 
> handling for non-partitionable state, i.e. state that was created with the 
> old {{Checkpointed}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4550: [FLINK-7461] Remove Backwards compatibility with <= Flink...

[StefanRRichter]

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/4550
  
CC @aljoscha @StephanEwen ; the relevant commit is only 
95e44099784c9deaf2ca422b8dfc11c3d67d7f82 .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7461) Remove Backwards compatibility for Flink 1.1 from Flink 1.4

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128948#comment-16128948
 ] 

ASF GitHub Bot commented on FLINK-7461:
---

GitHub user StefanRRichter opened a pull request:

https://github.com/apache/flink/pull/4550

[FLINK-7461] Remove Backwards compatibility with <= Flink 1.1

## Brief change log

This PR removes backwards compatibility with Flink <= 1.1 from the code. In 
particular, we can remove many special cases, such as the non-partitionable 
state that was created by the `Checkpointed` interface (replaced by 
`CheckpointedFunction` in 1.2).

We also drop aligned windows, which have already been a "hidden feature" 
since 1.2, because they still rely on the outdated `Checkpointed` interface.

The `Checkpointed` and `CheckpointedRestoring` interfaces are also removed 
and all related test cases have either been deleted or adapted.


## Verifying this change

This change is already covered by existing tests, such as *(please describe 
tests)*.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes)
  - The serializers: (yes, SavepointSerializer)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)

## Documentation

  - Does this pull request introduce a new feature? (no)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StefanRRichter/flink drop-1.1-compatibility

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4550


commit 50bbc938f3484e8a0d81511db719ebe296b8df94
Author: Stefan Richter 
Date:   2017-08-14T12:01:03Z

[FLINK-7460] [state backends] Close all ColumnFamilyHandles when restoring 
from rescaled incremental checkpoints

commit 95e44099784c9deaf2ca422b8dfc11c3d67d7f82
Author: Stefan Richter 
Date:   2017-08-14T12:02:37Z

[FLINK-7461] Remove Backwards compatibility with <= Flink 1.1




> Remove Backwards compatibility for Flink 1.1 from Flink 1.4
> ---
>
> Key: FLINK-7461
> URL: https://issues.apache.org/jira/browse/FLINK-7461
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>
> This issue tracks the removal of Flink 1.1 backwards compatibility from Flink 
> 1.4. This step is helpful for further developments because it will remove 
> many old code paths and special cases. In particular, we can drop all 
> handling for non-partitionable state, i.e. state that was created with the 
> old {{Checkpointed}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4550: [FLINK-7461] Remove Backwards compatibility with <...

[StefanRRichter]

GitHub user StefanRRichter opened a pull request:

https://github.com/apache/flink/pull/4550

[FLINK-7461] Remove Backwards compatibility with <= Flink 1.1

## Brief change log

This PR removes backwards compatibility with Flink <= 1.1 from the code. In 
particular, we can remove many special cases, such as the non-partitionable 
state that was created by the `Checkpointed` interface (replaced by 
`CheckpointedFunction` in 1.2).

We also drop aligned windows, which have already been a "hidden feature" 
since 1.2, because they still rely on the outdated `Checkpointed` interface.

The `Checkpointed` and `CheckpointedRestoring` interfaces are also removed 
and all related test cases have either been deleted or adapted.


## Verifying this change

This change is already covered by existing tests, such as *(please describe 
tests)*.

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes)
  - The serializers: (yes, SavepointSerializer)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)

## Documentation

  - Does this pull request introduce a new feature? (no)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StefanRRichter/flink drop-1.1-compatibility

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4550


commit 50bbc938f3484e8a0d81511db719ebe296b8df94
Author: Stefan Richter 
Date:   2017-08-14T12:01:03Z

[FLINK-7460] [state backends] Close all ColumnFamilyHandles when restoring 
from rescaled incremental checkpoints

commit 95e44099784c9deaf2ca422b8dfc11c3d67d7f82
Author: Stefan Richter 
Date:   2017-08-14T12:02:37Z

[FLINK-7461] Remove Backwards compatibility with <= Flink 1.1




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4331: [FLINK-7169][CEP] Support AFTER MATCH SKIP functio...

[yestinchen]

Github user yestinchen commented on a diff in the pull request:

https://github.com/apache/flink/pull/4331#discussion_r133478539
  
--- Diff: 
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/NFA.java ---
@@ -340,6 +362,65 @@ public void resetNFAChanged() {
return Tuple2.of(result, timeoutResult);
}
 
+   private void 
discardComputationStatesAccordingToStrategy(Queue 
computationStates,
--- End diff --

You are absolutely right. Thanks for the tip.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7169) Support AFTER MATCH SKIP function in CEP library API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128921#comment-16128921
 ] 

ASF GitHub Bot commented on FLINK-7169:
---

Github user yestinchen commented on a diff in the pull request:

https://github.com/apache/flink/pull/4331#discussion_r133478539
  
--- Diff: 
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/NFA.java ---
@@ -340,6 +362,65 @@ public void resetNFAChanged() {
return Tuple2.of(result, timeoutResult);
}
 
+   private void 
discardComputationStatesAccordingToStrategy(Queue 
computationStates,
--- End diff --

You are absolutely right. Thanks for the tip.


> Support AFTER MATCH SKIP function in CEP library API
> 
>
> Key: FLINK-7169
> URL: https://issues.apache.org/jira/browse/FLINK-7169
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Reporter: Yueting Chen
>Assignee: Yueting Chen
> Fix For: 1.4.0
>
>
> In order to support Oracle's MATCH_RECOGNIZE on top of the CEP library, we 
> need to support AFTER MATCH SKIP function in CEP API.
> There're four options in AFTER MATCH SKIP, listed as follows:
> 1. AFTER MATCH SKIP TO NEXT ROW: resume pattern matching at the row after the 
> first row of the current match.
> 2. AFTER MATCH SKIP PAST LAST ROW: resume pattern matching at the next row 
> after the last row of the current match.
> 3. AFTER MATCH SKIP TO FIST *RPV*: resume pattern matching at the first row 
> that is mapped to the row pattern variable RPV.
> 4. AFTER MATCH SKIP TO LAST *RPV*: resume pattern matching at the last row 
> that is mapped to the row pattern variable RPV.
> I think we can introduce a new function to `CEP` class, which takes a new 
> parameter as AfterMatchSKipStrategy.
> The new API may looks like this
> {code}
> public static  PatternStream pattern(DataStream input, Pattern 
> pattern, AfterMatchSkipStrategy afterMatchSkipStrategy) 
> {code}
> We can also make `SKIP TO NEXT ROW` as the default option, because that's 
> what CEP library behaves currently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7169) Support AFTER MATCH SKIP function in CEP library API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128920#comment-16128920
 ] 

ASF GitHub Bot commented on FLINK-7169:
---

Github user yestinchen commented on a diff in the pull request:

https://github.com/apache/flink/pull/4331#discussion_r133478273
  
--- Diff: 
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/compiler/NFACompiler.java
 ---
@@ -150,6 +160,29 @@ long getWindowTime() {
}
 
/**
+* Check pattern after match skip strategy.
+*/
--- End diff --

We only need to check the skip strategy before compile the `Pattern` to 
`NFA`, I think it's more reasonable to place it here. Also, we need to check 
whether the `patternName` field in the `AfterMatchSkipStrategy` is a valid 
reference, which can not be done easily in `Pattern` class.


> Support AFTER MATCH SKIP function in CEP library API
> 
>
> Key: FLINK-7169
> URL: https://issues.apache.org/jira/browse/FLINK-7169
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Reporter: Yueting Chen
>Assignee: Yueting Chen
> Fix For: 1.4.0
>
>
> In order to support Oracle's MATCH_RECOGNIZE on top of the CEP library, we 
> need to support AFTER MATCH SKIP function in CEP API.
> There're four options in AFTER MATCH SKIP, listed as follows:
> 1. AFTER MATCH SKIP TO NEXT ROW: resume pattern matching at the row after the 
> first row of the current match.
> 2. AFTER MATCH SKIP PAST LAST ROW: resume pattern matching at the next row 
> after the last row of the current match.
> 3. AFTER MATCH SKIP TO FIST *RPV*: resume pattern matching at the first row 
> that is mapped to the row pattern variable RPV.
> 4. AFTER MATCH SKIP TO LAST *RPV*: resume pattern matching at the last row 
> that is mapped to the row pattern variable RPV.
> I think we can introduce a new function to `CEP` class, which takes a new 
> parameter as AfterMatchSKipStrategy.
> The new API may looks like this
> {code}
> public static  PatternStream pattern(DataStream input, Pattern 
> pattern, AfterMatchSkipStrategy afterMatchSkipStrategy) 
> {code}
> We can also make `SKIP TO NEXT ROW` as the default option, because that's 
> what CEP library behaves currently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4331: [FLINK-7169][CEP] Support AFTER MATCH SKIP functio...

[yestinchen]

Github user yestinchen commented on a diff in the pull request:

https://github.com/apache/flink/pull/4331#discussion_r133478273
  
--- Diff: 
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/compiler/NFACompiler.java
 ---
@@ -150,6 +160,29 @@ long getWindowTime() {
}
 
/**
+* Check pattern after match skip strategy.
+*/
--- End diff --

We only need to check the skip strategy before compile the `Pattern` to 
`NFA`, I think it's more reasonable to place it here. Also, we need to check 
whether the `patternName` field in the `AfterMatchSkipStrategy` is a valid 
reference, which can not be done easily in `Pattern` class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-7460) Close all ColumnFamilyHandles when restoring rescaled incremental checkpoints

[Stefan Richter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Richter updated FLINK-7460:
--
Affects Version/s: 1.4.0
   1.3.2

> Close all ColumnFamilyHandles when restoring rescaled incremental checkpoints
> -
>
> Key: FLINK-7460
> URL: https://issues.apache.org/jira/browse/FLINK-7460
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.4.0, 1.3.2
>Reporter: Stefan Richter
>Assignee: Stefan Richter
>Priority: Minor
>
> In the restore method of the `RocksDBKeyedStateBackend` exists a case for 
> rescaling incremental checkpoints. This code creates temporary RocksDB 
> instances, but does not close the ColumnFamilyHandles created for those 
> temporary instances. This has shown in some assertion errors from RocksDB in 
> certain tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7461) Remove Backwards compatibility for Flink 1.1 from Flink 1.4

[Stefan Richter (JIRA)]

Stefan Richter created FLINK-7461:
-

 Summary: Remove Backwards compatibility for Flink 1.1 from Flink 
1.4
 Key: FLINK-7461
 URL: https://issues.apache.org/jira/browse/FLINK-7461
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.4.0
Reporter: Stefan Richter
Assignee: Stefan Richter


This issue tracks the removal of Flink 1.1 backwards compatibility from Flink 
1.4. This step is helpful for further developments because it will remove many 
old code paths and special cases. In particular, we can drop all handling for 
non-partitionable state, i.e. state that was created with the old 
{{Checkpointed}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7460) Close all ColumnFamilyHandles when restoring rescaled incremental checkpoints

[Stefan Richter (JIRA)]

Stefan Richter created FLINK-7460:
-

 Summary: Close all ColumnFamilyHandles when restoring rescaled 
incremental checkpoints
 Key: FLINK-7460
 URL: https://issues.apache.org/jira/browse/FLINK-7460
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing
Reporter: Stefan Richter
Assignee: Stefan Richter
Priority: Minor


In the restore method of the `RocksDBKeyedStateBackend` exists a case for 
rescaling incremental checkpoints. This code creates temporary RocksDB 
instances, but does not close the ColumnFamilyHandles created for those 
temporary instances. This has shown in some assertion errors from RocksDB in 
certain tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7459) Introduce RedirectHandler for the WebRuntimeMonitor

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-7459:


 Summary: Introduce RedirectHandler for the WebRuntimeMonitor
 Key: FLINK-7459
 URL: https://issues.apache.org/jira/browse/FLINK-7459
 Project: Flink
  Issue Type: Improvement
  Components: Distributed Coordination, Webfrontend
Affects Versions: 1.4.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann


Currently, the {{StaticFileServerHandler}} and the 
{{RuntimeMonitorHandlerBase}} both have functionality to redirect the requests 
to the current leader. In the future, there will be an {{AbstractRestHandler}} 
for Flip-6 which also needs this functionality. 

I therefore propose to create a general purpose {{RedirectHandler}} which can 
be extended to automatically add support for redirection.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7458) Generalize GatewayRetriever for WebRuntimeMonitor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128887#comment-16128887
 ] 

ASF GitHub Bot commented on FLINK-7458:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4549

[FLINK-7458] Generalize GatewayRetriever for WebRuntimeMonitor

## What is the purpose of the change

Introduce a generalized GatewayRetriever replacing the JobManagerRetriever. 
The
GatewayRetriever fulfills the same purpose as the JobManagerRetriever with 
the
ability to retrieve the gateway for an arbitrary endpoint type.

This PR is based on #4527.

## Brief change log

- Added a `LeaderRetriever` which retrieves and stores the current leader 
information
- Introduce the generic `GatewayRetriever` to retrieve arbitrary 
`RpcGateways`
- `LeaderGatewayRetriever` implements `GatewayRetriever` and extends 
`LeaderRetriever` to resolve the leader address into the specified gateway type
- `RpcGatewayRetriever` extends `LeaderGatewayRetriever` for the flip-6 
code, `AkkaJobManagerRetriever` does the same for the old code which only needs 
to retrieve the `JobManagerGateway`

## Verifying this change

This change added tests and can be verified as follows:

- Added `AkkaJobManagerRetrieverTest` and `RpcGatewayRetrieverTest`

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink 
generalizeJobManagerRetriever

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4549.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4549


commit 2c0c36244f75de000f884618330bc2f885c91d76
Author: Till Rohrmann 
Date:   2017-08-10T08:56:12Z

[FLINK-7409] [web] Make WebRuntimeMonitor reactive

This commit changes the behaviour of the WebRuntimeMonitor to not longer 
block serving
threads by waiting on the result of futures. Instead the RequestHandler now 
returns a
CompletableFuture which is written out to the Netty 
channel upon
completion. This will improve the performance of our WebRuntimeMonitor.

commit 01c877a24d65b8d75c49696bc8ab9e254d99
Author: Till Rohrmann 
Date:   2017-08-15T10:00:58Z

[FLINK-7458] Generalize GatewayRetriever for WebRuntimeMonitor

Introduce a generalized GatewayRetriever replacing the JobManagerRetriever. 
The
GatewayRetriever fulfills the same purpose as the JobManagerRetriever with 
the
ability to retrieve the gateway for an arbitrary endpoint type.




> Generalize GatewayRetriever for WebRuntimeMonitor
> -
>
> Key: FLINK-7458
> URL: https://issues.apache.org/jira/browse/FLINK-7458
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination, Webfrontend
>Affects Versions: 1.4.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>  Labels: flip-6
>
> Currently, all web frontend information can be retrieved from the 
> {{JobManager}}. For that, we have the {{JobManagerRetriever}} which listens 
> to leader changes and always connects to the leading {{JobManager}}. This 
> gateway can then be used by the REST handlers. In the future (with Flip-6) 
> the information will be distributed across different components and multiple 
> components will have a REST endpoint. Due to that the REST handlers no longer 
> can rely on the {{JobManagerGateway}}. Instead they might connect to 
> different entities which need to be retrieved. In order to solve this 
> problem, I propose to introduce a generalized gateway retriever which allows 
> to retrieve the {{RpcGateway}} for different components.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4549: [FLINK-7458] Generalize GatewayRetriever for WebRu...

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4549

[FLINK-7458] Generalize GatewayRetriever for WebRuntimeMonitor

## What is the purpose of the change

Introduce a generalized GatewayRetriever replacing the JobManagerRetriever. 
The
GatewayRetriever fulfills the same purpose as the JobManagerRetriever with 
the
ability to retrieve the gateway for an arbitrary endpoint type.

This PR is based on #4527.

## Brief change log

- Added a `LeaderRetriever` which retrieves and stores the current leader 
information
- Introduce the generic `GatewayRetriever` to retrieve arbitrary 
`RpcGateways`
- `LeaderGatewayRetriever` implements `GatewayRetriever` and extends 
`LeaderRetriever` to resolve the leader address into the specified gateway type
- `RpcGatewayRetriever` extends `LeaderGatewayRetriever` for the flip-6 
code, `AkkaJobManagerRetriever` does the same for the old code which only needs 
to retrieve the `JobManagerGateway`

## Verifying this change

This change added tests and can be verified as follows:

- Added `AkkaJobManagerRetrieverTest` and `RpcGatewayRetrieverTest`

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink 
generalizeJobManagerRetriever

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4549.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4549


commit 2c0c36244f75de000f884618330bc2f885c91d76
Author: Till Rohrmann 
Date:   2017-08-10T08:56:12Z

[FLINK-7409] [web] Make WebRuntimeMonitor reactive

This commit changes the behaviour of the WebRuntimeMonitor to not longer 
block serving
threads by waiting on the result of futures. Instead the RequestHandler now 
returns a
CompletableFuture which is written out to the Netty 
channel upon
completion. This will improve the performance of our WebRuntimeMonitor.

commit 01c877a24d65b8d75c49696bc8ab9e254d99
Author: Till Rohrmann 
Date:   2017-08-15T10:00:58Z

[FLINK-7458] Generalize GatewayRetriever for WebRuntimeMonitor

Introduce a generalized GatewayRetriever replacing the JobManagerRetriever. 
The
GatewayRetriever fulfills the same purpose as the JobManagerRetriever with 
the
ability to retrieve the gateway for an arbitrary endpoint type.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7444) Make external calls non-blocking

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128873#comment-16128873
 ] 

Stephan Ewen commented on FLINK-7444:
-

The case you mentioned to me sounds like it should be the responsibility of the 
MiniCluster to do the shutdown asynchronously.

> Make external calls non-blocking
> 
>
> Key: FLINK-7444
> URL: https://issues.apache.org/jira/browse/FLINK-7444
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination
>Affects Versions: 1.4.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>  Labels: flip-6
>
> All external calls from a {{RpcEndpoint}} can be potentially blocking, e.g. 
> calls to the {{FatalErrorHandler}}. Therefore, I propose to make all these 
> calls coming from the {{RpcEndpoint's}} main thread non-blocking by running 
> them in an {{Executor}}. That way the main thread will never be blocked.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7458) Generalize GatewayRetriever for WebRuntimeMonitor

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-7458:


 Summary: Generalize GatewayRetriever for WebRuntimeMonitor
 Key: FLINK-7458
 URL: https://issues.apache.org/jira/browse/FLINK-7458
 Project: Flink
  Issue Type: Improvement
  Components: Distributed Coordination, Webfrontend
Affects Versions: 1.4.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann


Currently, all web frontend information can be retrieved from the 
{{JobManager}}. For that, we have the {{JobManagerRetriever}} which listens to 
leader changes and always connects to the leading {{JobManager}}. This gateway 
can then be used by the REST handlers. In the future (with Flip-6) the 
information will be distributed across different components and multiple 
components will have a REST endpoint. Due to that the REST handlers no longer 
can rely on the {{JobManagerGateway}}. Instead they might connect to different 
entities which need to be retrieved. In order to solve this problem, I propose 
to introduce a generalized gateway retriever which allows to retrieve the 
{{RpcGateway}} for different components.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4548: [FLINK-7457] Make Dispatcher highly available

[tillrohrmann]

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4548

[FLINK-7457] Make Dispatcher highly available

## What is the purpose of the change

This commit introduces a dispatcher leader election and retrieval service 
to the
HighAvailabilityServices. Moreover it adds code such that the Dispatcher 
now takes
part in the leader election process using the afore-mentioned services.

Let Dispatcher participate in leader election

Add test for Dispatcher leader election

## Brief change log

- Introduce `LeaderElectionService` and `LeaderElectionRetrievalService` 
for the `Dispatcher` to the `HighAvailabililtyServices`
- Let the `Dispatcher participate in the leader election process
- Guard `Dispatcher#submitJob` with leader session id

## Verifying this change

This change added tests and can be verified as follows:

- Added `DispatcherTest#testLeaderElection` which simulates a leader 
election process

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink addDispatcherHA

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4548.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4548


commit 9be93a9814ac0727015710f6764ce5e6a4166ea7
Author: Till Rohrmann 
Date:   2017-08-16T12:36:13Z

[FLINK-7457] Make Dispatcher highly available

This commit introduces a dispatcher leader election and retrieval service 
to the
HighAvailabilityServices. Moreover it adds code such that the Dispatcher 
now takes
part in the leader election process using the afore-mentioned services.

Let Dispatcher participate in leader election

Add test for Dispatcher leader election




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7457) Make dispatcher highly available

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128824#comment-16128824
 ] 

ASF GitHub Bot commented on FLINK-7457:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/4548

[FLINK-7457] Make Dispatcher highly available

## What is the purpose of the change

This commit introduces a dispatcher leader election and retrieval service 
to the
HighAvailabilityServices. Moreover it adds code such that the Dispatcher 
now takes
part in the leader election process using the afore-mentioned services.

Let Dispatcher participate in leader election

Add test for Dispatcher leader election

## Brief change log

- Introduce `LeaderElectionService` and `LeaderElectionRetrievalService` 
for the `Dispatcher` to the `HighAvailabililtyServices`
- Let the `Dispatcher participate in the leader election process
- Guard `Dispatcher#submitJob` with leader session id

## Verifying this change

This change added tests and can be verified as follows:

- Added `DispatcherTest#testLeaderElection` which simulates a leader 
election process

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink addDispatcherHA

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/4548.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4548


commit 9be93a9814ac0727015710f6764ce5e6a4166ea7
Author: Till Rohrmann 
Date:   2017-08-16T12:36:13Z

[FLINK-7457] Make Dispatcher highly available

This commit introduces a dispatcher leader election and retrieval service 
to the
HighAvailabilityServices. Moreover it adds code such that the Dispatcher 
now takes
part in the leader election process using the afore-mentioned services.

Let Dispatcher participate in leader election

Add test for Dispatcher leader election




> Make dispatcher highly available
> 
>
> Key: FLINK-7457
> URL: https://issues.apache.org/jira/browse/FLINK-7457
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Affects Versions: 1.4.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>  Labels: flip-6
>
> The dispatcher component should be made highly available similar to the 
> {{ResourceManager}} and the {{JobMasters}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7413) Release Hadoop 2.8.x convenience binaries for 1.3.x

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128794#comment-16128794
 ] 

Robert Metzger commented on FLINK-7413:
---

I would suggest to wait for now.
If 1.4 is hugely delayed, and more users run into it, we can do it (an 
additional binary would not affect the dependencies of the other hadoop 
version).

> Release Hadoop 2.8.x convenience binaries for 1.3.x 
> 
>
> Key: FLINK-7413
> URL: https://issues.apache.org/jira/browse/FLINK-7413
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.3.3
>
>
> At least one user on the mailing lists had an issue because Hadoop 2.8.x 
> binaries are not available: 
> https://lists.apache.org/thread.html/c8badc66778144d9d6c3ee5cb23dd732a66cb6690c6867f47f4bd456@%3Cuser.flink.apache.org%3E
> It should be as easy as adding Hadoop 2.8.x to the list of created binaries 
> in the release files.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-5886) Python API for streaming applications

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128767#comment-16128767
 ] 

ASF GitHub Bot commented on FLINK-5886:
---

Github user zohar-mizrahi commented on the issue:

https://github.com/apache/flink/pull/3838
  
Regarding the exception - 
```java.io.IOException: java.io.IOException: The given HDFS file URI ...```

In general, using the python interface requires a valid configuration of 
shared file system (.e.g HDFS), which designed to distribute the python files. 
Someone can bypass this issue by set the second argument to 'True' when 
calling to ```env.execute(...)``` in the python script.


> Python API for streaming applications
> -
>
> Key: FLINK-5886
> URL: https://issues.apache.org/jira/browse/FLINK-5886
> Project: Flink
>  Issue Type: New Feature
>  Components: Python API
>Reporter: Zohar Mizrahi
>Assignee: Zohar Mizrahi
>
> A work in progress to provide python interface for Flink streaming APIs. The 
> core technology is based on jython and thus imposes two limitations: a. user 
> defined functions cannot use python extensions. b. the python version is 2.x
> The branch is based on Flink release 1.2.0, as can be found here:
> https://github.com/zohar-pm/flink/tree/python-streaming
> In order to test it, someone can use IntelliJ IDE. Assuming IntelliJ was 
> setup properly (see: 
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/ide_setup.html),
>  one can run/debug {{org.apache.flink.python.api.PythonStreamBinderTest}}, 
> which in return will execute all the tests under 
> {{/Users/zohar/dev/pm-flink/flink-libraries/flink-python/src/test/python/org/apache/flink/python/api/streaming}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #3838: [FLINK-5886] Python API for streaming applications

[zohar-mizrahi]

Github user zohar-mizrahi commented on the issue:

https://github.com/apache/flink/pull/3838
  
Regarding the exception - 
```java.io.IOException: java.io.IOException: The given HDFS file URI ...```

In general, using the python interface requires a valid configuration of 
shared file system (.e.g HDFS), which designed to distribute the python files. 
Someone can bypass this issue by set the second argument to 'True' when 
calling to ```env.execute(...)``` in the python script.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6995) Add a warning to outdated documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128698#comment-16128698
 ] 

ASF GitHub Bot commented on FLINK-6995:
---

Github user zhangminglei commented on the issue:

https://github.com/apache/flink/pull/4480
  
Thanks @tzulitai for review~


> Add a warning to outdated documentation
> ---
>
> Key: FLINK-6995
> URL: https://issues.apache.org/jira/browse/FLINK-6995
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Timo Walther
>Assignee: mingleizhang
>
> When I search for "flink yarn" by Google, the first result is a outdated 0.8 
> release documentation page. We should add a warning to outdated documentation 
> pages.
> There are other problems as well:
> The main page only links to 1.3 and 1.4 but the flink-docs-master 
> documentation links to 1.3, 1.2, 1.1, and 1.0. But each of those packages 
> only links to older releases so if a user arrives on a 1.2 page they won't 
> see 1.3.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6995) Add a warning to outdated documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128697#comment-16128697
 ] 

ASF GitHub Bot commented on FLINK-6995:
---

Github user zhangminglei closed the pull request at:

https://github.com/apache/flink/pull/4480


> Add a warning to outdated documentation
> ---
>
> Key: FLINK-6995
> URL: https://issues.apache.org/jira/browse/FLINK-6995
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Timo Walther
>Assignee: mingleizhang
>
> When I search for "flink yarn" by Google, the first result is a outdated 0.8 
> release documentation page. We should add a warning to outdated documentation 
> pages.
> There are other problems as well:
> The main page only links to 1.3 and 1.4 but the flink-docs-master 
> documentation links to 1.3, 1.2, 1.1, and 1.0. But each of those packages 
> only links to older releases so if a user arrives on a 1.2 page they won't 
> see 1.3.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4480: [FLINK-6995] [docs] Enable is_latest attribute to false

[zhangminglei]

Github user zhangminglei commented on the issue:

https://github.com/apache/flink/pull/4480
  
Thanks @tzulitai for review~


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4480: [FLINK-6995] [docs] Enable is_latest attribute to ...

[zhangminglei]

Github user zhangminglei closed the pull request at:

https://github.com/apache/flink/pull/4480


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-7457) Make dispatcher highly available

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann updated FLINK-7457:
-
Summary: Make dispatcher highly available  (was: Make dispatcher HA)

> Make dispatcher highly available
> 
>
> Key: FLINK-7457
> URL: https://issues.apache.org/jira/browse/FLINK-7457
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Coordination
>Affects Versions: 1.4.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>  Labels: flip-6
>
> The dispatcher component should be made highly available similar to the 
> {{ResourceManager}} and the {{JobMasters}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7457) Make dispatcher HA

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-7457:


 Summary: Make dispatcher HA
 Key: FLINK-7457
 URL: https://issues.apache.org/jira/browse/FLINK-7457
 Project: Flink
  Issue Type: Improvement
  Components: Distributed Coordination
Affects Versions: 1.4.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann


The dispatcher component should be made highly available similar to the 
{{ResourceManager}} and the {{JobMasters}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-3089) State API Should Support Data Expiration (State TTL)

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated FLINK-3089:

Description: 
In some usecases (webanalytics) there is a need to have a state per visitor on 
a website (i.e. keyBy(sessionid) ).
At some point the visitor simply leaves and no longer creates new events (so a 
special 'end of session' event will not occur).

The only way to determine that a visitor has left is by choosing a timeout, 
like "After 30 minutes no events we consider the visitor 'gone'".

Only after this (chosen) timeout has expired should we discard this state.

In the Trigger part of Windows we can set a timer and close/discard this kind 
of information. But that introduces the buffering effect of the window (which 
in some scenarios is unwanted).

What I would like is to be able to set a timeout on a specific state which I 
can update afterwards.

This makes it possible to create a map function that assigns the right value 
and that discards the state automatically.


  was:
In some usecases (webanalytics) there is a need to have a state per visitor on 
a website (i.e. keyBy(sessionid) ).
At some point the visitor simply leaves and no longer creates new events (so a 
special 'end of session' event will not occur).

The only way to determine that a visitor has left is by choosing a timeout, 
like "After 30 minutes no events we consider the visitor 'gone'".

Only after this (chosen) timeout has expired should we discard this state.

In the Trigger part of Windows we can set a timer and close/discard this kind 
of information. But that introduces the buffering effect of the window (which 
in some scenarios is unwanted).

What I would like is to be able to set a timeout on a specific OperatorState 
value which I can update afterwards.

This makes it possible to create a map function that assigns the right value 
and that discards the state automatically.



> State API Should Support Data Expiration (State TTL)
> 
>
> Key: FLINK-3089
> URL: https://issues.apache.org/jira/browse/FLINK-3089
> Project: Flink
>  Issue Type: New Feature
>  Components: DataStream API, State Backends, Checkpointing
>Reporter: Niels Basjes
>
> In some usecases (webanalytics) there is a need to have a state per visitor 
> on a website (i.e. keyBy(sessionid) ).
> At some point the visitor simply leaves and no longer creates new events (so 
> a special 'end of session' event will not occur).
> The only way to determine that a visitor has left is by choosing a timeout, 
> like "After 30 minutes no events we consider the visitor 'gone'".
> Only after this (chosen) timeout has expired should we discard this state.
> In the Trigger part of Windows we can set a timer and close/discard this kind 
> of information. But that introduces the buffering effect of the window (which 
> in some scenarios is unwanted).
> What I would like is to be able to set a timeout on a specific state which I 
> can update afterwards.
> This makes it possible to create a map function that assigns the right value 
> and that discards the state automatically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (FLINK-7455) RocksDB TTL support

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-7455.
---
Resolution: Duplicate

This is a (more specific) duplicate of FLINK-3089.

> RocksDB TTL support
> ---
>
> Key: FLINK-7455
> URL: https://issues.apache.org/jira/browse/FLINK-7455
> Project: Flink
>  Issue Type: New Feature
>  Components: State Backends, Checkpointing
>Reporter: Andrey Konyaev
>
> We do not have the ability to set ttl for record in rocksdb.
> If key no longer comes, the information about it will always remain in memory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-3089) State API Should Support Data Expiration (State TTL)

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated FLINK-3089:

Summary: State API Should Support Data Expiration (State TTL)  (was: State 
API Should Support Data Expiration)

> State API Should Support Data Expiration (State TTL)
> 
>
> Key: FLINK-3089
> URL: https://issues.apache.org/jira/browse/FLINK-3089
> Project: Flink
>  Issue Type: New Feature
>  Components: DataStream API, State Backends, Checkpointing
>Reporter: Niels Basjes
>
> In some usecases (webanalytics) there is a need to have a state per visitor 
> on a website (i.e. keyBy(sessionid) ).
> At some point the visitor simply leaves and no longer creates new events (so 
> a special 'end of session' event will not occur).
> The only way to determine that a visitor has left is by choosing a timeout, 
> like "After 30 minutes no events we consider the visitor 'gone'".
> Only after this (chosen) timeout has expired should we discard this state.
> In the Trigger part of Windows we can set a timer and close/discard this kind 
> of information. But that introduces the buffering effect of the window (which 
> in some scenarios is unwanted).
> What I would like is to be able to set a timeout on a specific OperatorState 
> value which I can update afterwards.
> This makes it possible to create a map function that assigns the right value 
> and that discards the state automatically.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (FLINK-7456) Implement Netty sender incoming pipeline for credit-based

[zhijiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhijiang reassigned FLINK-7456:
---

Assignee: zhijiang

> Implement Netty sender incoming pipeline for credit-based
> -
>
> Key: FLINK-7456
> URL: https://issues.apache.org/jira/browse/FLINK-7456
> Project: Flink
>  Issue Type: Sub-task
>  Components: Network
>Reporter: zhijiang
>Assignee: zhijiang
> Fix For: 1.4.0
>
>
> This is a part of work for credit-based network flow control.
> On sender side, each subpartition view maintains an atomic integer 
> {{currentCredit}} from receiver. Once receiving the messages of 
> {{PartitionRequest}} and {{AddCredit}}, the {{currentCredit}} is added by 
> deltas.
> Each view also maintains an atomic boolean field to mark it as registered 
> available for transfer to make sure it is enqueued in handler only once. If 
> the {{currentCredit}} increases from zero and there are available buffers in 
> the subpartition, the corresponding view will be enqueued for transferring 
> data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLINK-7456) Implement Netty sender incoming pipeline for credit-based

[zhijiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-7456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhijiang updated FLINK-7456:

Description: 
This is a part of work for credit-based network flow control.

On sender side, each subpartition view maintains an atomic integer 
{{currentCredit}} from receiver. Once receiving the messages of 
{{PartitionRequest}} and {{AddCredit}}, the {{currentCredit}} is added by 
deltas.

Each view also maintains an atomic boolean field to mark it as registered 
available for transfer to make sure it is enqueued in handler only once. If the 
{{currentCredit}} increases from zero and there are available buffers in the 
subpartition, the corresponding view will be enqueued for transferring data.

  was:
This is a part of work for credit-based network flow control.

On sender side, each subpartition view maintains an atomic integer 
`currentCredit` from receiver. Once receiving the messages of 
`PartitionRequest` and `AddCredit`, the `currentCredit` is added by deltas.

Each view also maintains an atomic boolean field to mark it as registered 
available for transfer to make sure it is enqueued in handler only once. If the 
`currentCredit` increases from zero and there are available buffers in the 
subpartition, the corresponding view will be enqueued for transferring data.


> Implement Netty sender incoming pipeline for credit-based
> -
>
> Key: FLINK-7456
> URL: https://issues.apache.org/jira/browse/FLINK-7456
> Project: Flink
>  Issue Type: Sub-task
>  Components: Network
>Reporter: zhijiang
> Fix For: 1.4.0
>
>
> This is a part of work for credit-based network flow control.
> On sender side, each subpartition view maintains an atomic integer 
> {{currentCredit}} from receiver. Once receiving the messages of 
> {{PartitionRequest}} and {{AddCredit}}, the {{currentCredit}} is added by 
> deltas.
> Each view also maintains an atomic boolean field to mark it as registered 
> available for transfer to make sure it is enqueued in handler only once. If 
> the {{currentCredit}} increases from zero and there are available buffers in 
> the subpartition, the corresponding view will be enqueued for transferring 
> data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-7456) Implement Netty sender incoming pipeline for credit-based

[zhijiang (JIRA)]

zhijiang created FLINK-7456:
---

 Summary: Implement Netty sender incoming pipeline for credit-based
 Key: FLINK-7456
 URL: https://issues.apache.org/jira/browse/FLINK-7456
 Project: Flink
  Issue Type: Sub-task
  Components: Network
Reporter: zhijiang
 Fix For: 1.4.0


This is a part of work for credit-based network flow control.

On sender side, each subpartition view maintains an atomic integer 
`currentCredit` from receiver. Once receiving the messages of 
`PartitionRequest` and `AddCredit`, the `currentCredit` is added by deltas.

Each view also maintains an atomic boolean field to mark it as registered 
available for transfer to make sure it is enqueued in handler only once. If the 
`currentCredit` increases from zero and there are available buffers in the 
subpartition, the corresponding view will be enqueued for transferring data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-6805) Flink Cassandra connector dependency on Netty disagrees with Flink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128581#comment-16128581
 ] 

ASF GitHub Bot commented on FLINK-6805:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4545#discussion_r133407236
  
--- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
@@ -83,6 +85,10 @@ under the License.

com.google.inject.**



+   
+   
io.netty
+   
org.apache.flink.shaded.netty4.io.netty
--- End diff --

1) We haven't written them down anywhere; but there are examples on how the 
pattern should look like in many modules. (Granted, there are some relocations 
that don't follow the points I made; standardizing them is a work-in-progress) 
I would generally refer to existing relocations instead of some document that 
is bound to be out-dated at some point.
2) Currently we shade dependencies that already caused conflicts.
3) Yes, but there is no better solution. While it is possible to re-use 
some dependencies from Flink this tends to be a maintaining nightmare, as an 
upgrade to a dependency can suddenly cause issues in another part. The netty 
shading is a perfect example of this: we replaced the netty dependency and now 
the connector is broken.


> Flink Cassandra connector dependency on Netty disagrees with Flink
> --
>
> Key: FLINK-6805
> URL: https://issues.apache.org/jira/browse/FLINK-6805
> Project: Flink
>  Issue Type: Bug
>  Components: Cassandra Connector
>Affects Versions: 1.3.0, 1.2.1
>Reporter: Shannon Carey
>Assignee: Michael Fong
> Fix For: 1.4.0
>
>
> The Flink Cassandra connector has a dependency on Netty libraries (via 
> promotion of transitive dependencies by the Maven shade plugin) at version 
> 4.0.33.Final, which disagrees with the version included in Flink of 
> 4.0.27.Final which is included & managed by the parent POM via dependency on 
> netty-all.
> Due to use of netty-all, the dependency management doesn't take effect on the 
> individual libraries such as netty-handler, netty-codec, etc.
> I suggest that dependency management of Netty should be added for all Netty 
> libraries individually (netty-handler, etc.) so that all Flink modules use 
> the same version, and similarly I suggest that exclusions be added to the 
> quickstart example POM for the individual Netty libraries so that fat JARs 
> don't include conflicting versions of Netty.
> It seems like this problem started when FLINK-6084 was implemented: 
> transitive dependencies of the flink-connector-cassandra were previously 
> omitted, and now that they are included we must make sure that they agree 
> with the Flink distribution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4545: [FLINK-6805] [Cassandra-Connector] Shade indirect ...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/4545#discussion_r133407236
  
--- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
@@ -83,6 +85,10 @@ under the License.

com.google.inject.**



+   
+   
io.netty
+   
org.apache.flink.shaded.netty4.io.netty
--- End diff --

1) We haven't written them down anywhere; but there are examples on how the 
pattern should look like in many modules. (Granted, there are some relocations 
that don't follow the points I made; standardizing them is a work-in-progress) 
I would generally refer to existing relocations instead of some document that 
is bound to be out-dated at some point.
2) Currently we shade dependencies that already caused conflicts.
3) Yes, but there is no better solution. While it is possible to re-use 
some dependencies from Flink this tends to be a maintaining nightmare, as an 
upgrade to a dependency can suddenly cause issues in another part. The netty 
shading is a perfect example of this: we replaced the netty dependency and now 
the connector is broken.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6938) IterativeCondition should support RichFunction interface

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128575#comment-16128575
 ] 

ASF GitHub Bot commented on FLINK-6938:
---

Github user dianfu commented on the issue:

https://github.com/apache/flink/pull/4513
  
@dawidwys @kl0u In case you missed this PR, could you help to take a look 
at? Very appreciated.


> IterativeCondition should support RichFunction interface
> 
>
> Key: FLINK-6938
> URL: https://issues.apache.org/jira/browse/FLINK-6938
> Project: Flink
>  Issue Type: Sub-task
>  Components: CEP
>Reporter: Jark Wu
>Assignee: Jark Wu
> Fix For: 1.4.0
>
>
> In FLIP-20, we need IterativeCondition to support an {{open()}} method to 
> compile the generated code once. We do not want to insert a if condition  in 
> the {{filter()}} method. So I suggest make IterativeCondition support 
> {{RichFunction}} interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4513: [FLINK-6938][FLINK-6939] [cep] Not store IterativeConditi...

[dianfu]

Github user dianfu commented on the issue:

https://github.com/apache/flink/pull/4513
  
@dawidwys @kl0u In case you missed this PR, could you help to take a look 
at? Very appreciated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7123) Support timesOrMore in CEP

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128573#comment-16128573
 ] 

ASF GitHub Bot commented on FLINK-7123:
---

Github user dianfu commented on the issue:

https://github.com/apache/flink/pull/4523
  
@dawidwys @kl0u Could you help to take a look at this PR? Thanks a lot.


> Support timesOrMore in CEP
> --
>
> Key: FLINK-7123
> URL: https://issues.apache.org/jira/browse/FLINK-7123
> Project: Flink
>  Issue Type: Bug
>  Components: CEP
>Reporter: Dian Fu
>Assignee: Dian Fu
>
> The CEP API should provide API such as timesOrMore(#ofTimes) for quantifier 
> {n,}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4523: [FLINK-7123] [cep] Support timesOrMore in CEP

[dianfu]

Github user dianfu commented on the issue:

https://github.com/apache/flink/pull/4523
  
@dawidwys @kl0u Could you help to take a look at this PR? Thanks a lot.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6805) Flink Cassandra connector dependency on Netty disagrees with Flink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128565#comment-16128565
 ] 

ASF GitHub Bot commented on FLINK-6805:
---

Github user mcfongtw commented on a diff in the pull request:

https://github.com/apache/flink/pull/4545#discussion_r133404126
  
--- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
@@ -83,6 +85,10 @@ under the License.

com.google.inject.**



+   
+   
io.netty
+   
org.apache.flink.shaded.netty4.io.netty
--- End diff --

Thanks for your review and reply! I see your points now, and I still have a 
few questions to the shading approach though:
1. There seems to be some policies for making shadedPattern for different 
scenarios. Would there be any external documents that possibly address these 
rules?
2. It is obvious to shade a *direct dependency, since we need it to be free 
of class clashes. How would we decide which *indirect dependencies to be shaded?
3. The benefit of shading is clear, but wouldn't it increase the jar file 
size if there are more dependencies that need to be managed later on - also the 
effort of managing the including/relocation rules.

I will update another version of PR later. Thanks!


> Flink Cassandra connector dependency on Netty disagrees with Flink
> --
>
> Key: FLINK-6805
> URL: https://issues.apache.org/jira/browse/FLINK-6805
> Project: Flink
>  Issue Type: Bug
>  Components: Cassandra Connector
>Affects Versions: 1.3.0, 1.2.1
>Reporter: Shannon Carey
>Assignee: Michael Fong
> Fix For: 1.4.0
>
>
> The Flink Cassandra connector has a dependency on Netty libraries (via 
> promotion of transitive dependencies by the Maven shade plugin) at version 
> 4.0.33.Final, which disagrees with the version included in Flink of 
> 4.0.27.Final which is included & managed by the parent POM via dependency on 
> netty-all.
> Due to use of netty-all, the dependency management doesn't take effect on the 
> individual libraries such as netty-handler, netty-codec, etc.
> I suggest that dependency management of Netty should be added for all Netty 
> libraries individually (netty-handler, etc.) so that all Flink modules use 
> the same version, and similarly I suggest that exclusions be added to the 
> quickstart example POM for the individual Netty libraries so that fat JARs 
> don't include conflicting versions of Netty.
> It seems like this problem started when FLINK-6084 was implemented: 
> transitive dependencies of the flink-connector-cassandra were previously 
> omitted, and now that they are included we must make sure that they agree 
> with the Flink distribution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink pull request #4545: [FLINK-6805] [Cassandra-Connector] Shade indirect ...

[mcfongtw]

Github user mcfongtw commented on a diff in the pull request:

https://github.com/apache/flink/pull/4545#discussion_r133404126
  
--- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
@@ -83,6 +85,10 @@ under the License.

com.google.inject.**



+   
+   
io.netty
+   
org.apache.flink.shaded.netty4.io.netty
--- End diff --

Thanks for your review and reply! I see your points now, and I still have a 
few questions to the shading approach though:
1. There seems to be some policies for making shadedPattern for different 
scenarios. Would there be any external documents that possibly address these 
rules?
2. It is obvious to shade a *direct dependency, since we need it to be free 
of class clashes. How would we decide which *indirect dependencies to be shaded?
3. The benefit of shading is clear, but wouldn't it increase the jar file 
size if there are more dependencies that need to be managed later on - also the 
effort of managing the including/relocation rules.

I will update another version of PR later. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #4545: [FLINK-6805] [Cassandra-Connector] Shade indirect ...

[mcfongtw]

Github user mcfongtw commented on a diff in the pull request:

https://github.com/apache/flink/pull/4545#discussion_r13339
  
--- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
@@ -39,6 +39,7 @@ under the License.

2.2.5
3.0.0
+   4.0.27.Final
--- End diff --

Good call! It's not used currently. I will remove it .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-6805) Flink Cassandra connector dependency on Netty disagrees with Flink

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128540#comment-16128540
 ] 

ASF GitHub Bot commented on FLINK-6805:
---

Github user mcfongtw commented on a diff in the pull request:

https://github.com/apache/flink/pull/4545#discussion_r13339
  
--- Diff: flink-connectors/flink-connector-cassandra/pom.xml ---
@@ -39,6 +39,7 @@ under the License.

2.2.5
3.0.0
+   4.0.27.Final
--- End diff --

Good call! It's not used currently. I will remove it .


> Flink Cassandra connector dependency on Netty disagrees with Flink
> --
>
> Key: FLINK-6805
> URL: https://issues.apache.org/jira/browse/FLINK-6805
> Project: Flink
>  Issue Type: Bug
>  Components: Cassandra Connector
>Affects Versions: 1.3.0, 1.2.1
>Reporter: Shannon Carey
>Assignee: Michael Fong
> Fix For: 1.4.0
>
>
> The Flink Cassandra connector has a dependency on Netty libraries (via 
> promotion of transitive dependencies by the Maven shade plugin) at version 
> 4.0.33.Final, which disagrees with the version included in Flink of 
> 4.0.27.Final which is included & managed by the parent POM via dependency on 
> netty-all.
> Due to use of netty-all, the dependency management doesn't take effect on the 
> individual libraries such as netty-handler, netty-codec, etc.
> I suggest that dependency management of Netty should be added for all Netty 
> libraries individually (netty-handler, etc.) so that all Flink modules use 
> the same version, and similarly I suggest that exclusions be added to the 
> quickstart example POM for the individual Netty libraries so that fat JARs 
> don't include conflicting versions of Netty.
> It seems like this problem started when FLINK-6084 was implemented: 
> transitive dependencies of the flink-connector-cassandra were previously 
> omitted, and now that they are included we must make sure that they agree 
> with the Flink distribution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flink issue #4238: [FLINK-7057][blob] move BLOB ref-counting from LibraryCac...

[NicoK]

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/4238
  
rebased, squashed, and fixed the new issue found by the spotbugs plugin

since Flink is moving fast, though, there's a conflict again - should I 
rebase again?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-7057) move BLOB ref-counting from LibraryCacheManager to BlobCache

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128538#comment-16128538
 ] 

ASF GitHub Bot commented on FLINK-7057:
---

Github user NicoK commented on the issue:

https://github.com/apache/flink/pull/4238
  
rebased, squashed, and fixed the new issue found by the spotbugs plugin

since Flink is moving fast, though, there's a conflict again - should I 
rebase again?


> move BLOB ref-counting from LibraryCacheManager to BlobCache
> 
>
> Key: FLINK-7057
> URL: https://issues.apache.org/jira/browse/FLINK-7057
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination, Network
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>
> Currently, the {{LibraryCacheManager}} is doing some ref-counting for JAR 
> files managed by it. Instead, we want the {{BlobCache}} to do that itself for 
> all job-related BLOBs. Also, we do not want to operate on a per-{{BlobKey}} 
> level but rather per job. Therefore, the cleanup process should be adapted, 
> too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-7447) Hope add more committer information to "Community & Project Info" page.

[Hai Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-7447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128502#comment-16128502
 ] 

Hai Zhou commented on FLINK-7447:
-

[~Zentol] 
>From the standpoint of the contributor,  I still think that makes sense.
Maybe we can refer to other open source projects:
hadoop: http://hadoop.apache.org/who.html
tez: http://tez.apache.org/team-list.html
hbase: http://hbase.apache.org/team-list.html
spark: http://spark.apache.org/committers.html
...
:)


> Hope add more committer information to "Community & Project Info" page.
> ---
>
> Key: FLINK-7447
> URL: https://issues.apache.org/jira/browse/FLINK-7447
> Project: Flink
>  Issue Type: Wish
>  Components: Project Website
>Reporter: Hai Zhou
>
> I wish add the "organization" and "time zone" information to committer 
> introduction, while using the mail instead of Apache ID.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)