[jira] [Commented] (FLINK-3034) Redis SInk Connector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376320#comment-15376320
 ] 

ASF GitHub Bot commented on FLINK-3034:
---

Github user subhankarb commented on the issue:

https://github.com/apache/flink/pull/1813
  
Hi @kl0u, IMO that is the expected behavior. The sink would not know that 
if the Redis is down or not unless it tries to send the next data to the Redis. 
When ever a new message reaches the sink it tries to use the connection pool, 
then an then only it can throw exception that it can not send the data to Redis.
You can build a heartbeat mechanism to check periodically that Redis serve 
is up or down, and can cancel the job if Redis is down. 
@mjsax please correct me if my understanding is wrong.  


> Redis SInk Connector
> 
>
> Key: FLINK-3034
> URL: https://issues.apache.org/jira/browse/FLINK-3034
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming Connectors
>Reporter: Matthias J. Sax
>Assignee: Subhankar Biswas
>Priority: Minor
> Fix For: 1.1.0
>
>
> Flink does not provide a sink connector for Redis.
> See FLINK-3033



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #1813: [FLINK-3034] Redis Sink Connector

[subhankarb]

Github user subhankarb commented on the issue:

https://github.com/apache/flink/pull/1813
  
Hi @kl0u, IMO that is the expected behavior. The sink would not know that 
if the Redis is down or not unless it tries to send the next data to the Redis. 
When ever a new message reaches the sink it tries to use the connection pool, 
then an then only it can throw exception that it can not send the data to Redis.
You can build a heartbeat mechanism to check periodically that Redis serve 
is up or down, and can cancel the job if Redis is down. 
@mjsax please correct me if my understanding is wrong.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-4214) JobExceptionsHandler will return all exceptions

[Sumit Chawla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sumit Chawla updated FLINK-4214:

Priority: Minor  (was: Major)

> JobExceptionsHandler will return all exceptions
> ---
>
> Key: FLINK-4214
> URL: https://issues.apache.org/jira/browse/FLINK-4214
> Project: Flink
>  Issue Type: Bug
>Reporter: Sumit Chawla
>Priority: Minor
>
> JobExceptionsHandler will return all exceptions and is not incrementing the 
> integer to track the exceptions being serialized



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4214) JobExceptionsHandler will return all exceptions

[Sumit Chawla (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376269#comment-15376269
 ] 

Sumit Chawla commented on FLINK-4214:
-

Change added in https://github.com/apache/flink/pull/2242

> JobExceptionsHandler will return all exceptions
> ---
>
> Key: FLINK-4214
> URL: https://issues.apache.org/jira/browse/FLINK-4214
> Project: Flink
>  Issue Type: Bug
>Reporter: Sumit Chawla
>
> JobExceptionsHandler will return all exceptions and is not incrementing the 
> integer to track the exceptions being serialized



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4214) JobExceptionsHandler will return all exceptions

[Sumit Chawla (JIRA)]

Sumit Chawla created FLINK-4214:
---

 Summary: JobExceptionsHandler will return all exceptions
 Key: FLINK-4214
 URL: https://issues.apache.org/jira/browse/FLINK-4214
 Project: Flink
  Issue Type: Bug
Reporter: Sumit Chawla


JobExceptionsHandler will return all exceptions and is not incrementing the 
integer to track the exceptions being serialized



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4213) Provide CombineHint in Gelly algorithms

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-4213:
-

 Summary: Provide CombineHint in Gelly algorithms
 Key: FLINK-4213
 URL: https://issues.apache.org/jira/browse/FLINK-4213
 Project: Flink
  Issue Type: Improvement
  Components: Gelly
Affects Versions: 1.1.0
Reporter: Greg Hogan
Assignee: Greg Hogan


Many graph algorithms will see better {{reduce}} performance with the 
hash-combine compared with the still default sort-combine, e.g. HITS and 
LocalClusteringCoefficient.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3477) Add hash-based combine strategy for ReduceFunction

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan closed FLINK-3477.
-
Resolution: Implemented

Implemented in 52e191a5067322e82192314c16e70ae9e937ae2c

> Add hash-based combine strategy for ReduceFunction
> --
>
> Key: FLINK-3477
> URL: https://issues.apache.org/jira/browse/FLINK-3477
> Project: Flink
>  Issue Type: Sub-task
>  Components: Local Runtime
>Reporter: Fabian Hueske
>Assignee: Gabor Gevay
>
> This issue is about adding a hash-based combine strategy for ReduceFunctions.
> The interface of the {{reduce()}} method is as follows:
> {code}
> public T reduce(T v1, T v2)
> {code}
> Input type and output type are identical and the function returns only a 
> single value. A Reduce function is incrementally applied to compute a final 
> aggregated value. This allows to hold the preaggregated value in a hash-table 
> and update it with each function call. 
> The hash-based strategy requires special implementation of an in-memory hash 
> table. The hash table should support in place updates of elements (if the 
> updated value has the same size as the new value) but also appending updates 
> with invalidation of the old value (if the binary length of the new value 
> differs). The hash table needs to be able to evict and emit all elements if 
> it runs out-of-memory.
> We should also add {{HASH}} and {{SORT}} compiler hints to 
> {{DataSet.reduce()}} and {{Grouping.reduce()}} to allow users to pick the 
> execution strategy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-2246) Add chained combine driver strategy for ReduceFunction

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan closed FLINK-2246.
-
Resolution: Implemented

Implemented in 0db804b936efd8631f1a08db37753dad7f1f71ea

> Add chained combine driver strategy for ReduceFunction
> --
>
> Key: FLINK-2246
> URL: https://issues.apache.org/jira/browse/FLINK-2246
> Project: Flink
>  Issue Type: Improvement
>  Components: Local Runtime
>Affects Versions: 0.9, 0.10.0
>Reporter: Ufuk Celebi
>Assignee: Gabor Gevay
>Priority: Minor
>
> Running the WordCount example with a text file input/output results and a 
> manual reduce function (instead of the sum(1)) results in a combiner, which 
> is not chained.
> Replace sum(1) with the following to reproduce and use a text file as input:
> {code}
> fileOutput = true;
> textPath = "...";
> outputPath = "...";
> {code}
> {code}
> .reduce(new ReduceFunction>() {
> @Override
> public Tuple2 reduce(Tuple2 value1, 
> Tuple2 value2) throws Exception {
> return new Tuple2(value1.f0, value1.f1 + value2.f1);
> }
> });
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3477) Add hash-based combine strategy for ReduceFunction

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376126#comment-15376126
 ] 

ASF GitHub Bot commented on FLINK-3477:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1517


> Add hash-based combine strategy for ReduceFunction
> --
>
> Key: FLINK-3477
> URL: https://issues.apache.org/jira/browse/FLINK-3477
> Project: Flink
>  Issue Type: Sub-task
>  Components: Local Runtime
>Reporter: Fabian Hueske
>Assignee: Gabor Gevay
>
> This issue is about adding a hash-based combine strategy for ReduceFunctions.
> The interface of the {{reduce()}} method is as follows:
> {code}
> public T reduce(T v1, T v2)
> {code}
> Input type and output type are identical and the function returns only a 
> single value. A Reduce function is incrementally applied to compute a final 
> aggregated value. This allows to hold the preaggregated value in a hash-table 
> and update it with each function call. 
> The hash-based strategy requires special implementation of an in-memory hash 
> table. The hash table should support in place updates of elements (if the 
> updated value has the same size as the new value) but also appending updates 
> with invalidation of the old value (if the binary length of the new value 
> differs). The hash table needs to be able to evict and emit all elements if 
> it runs out-of-memory.
> We should also add {{HASH}} and {{SORT}} compiler hints to 
> {{DataSet.reduce()}} and {{Grouping.reduce()}} to allow users to pick the 
> execution strategy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #1517: [FLINK-3477] [runtime] Add hash-based combine stra...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1517


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3874) Add a Kafka TableSink with JSON serialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375822#comment-15375822
 ] 

ASF GitHub Bot commented on FLINK-3874:
---

Github user mushketyk commented on the issue:

https://github.com/apache/flink/pull/2244
  
I tried to inherit TableSink trait in Java code but it seems that it is 
impossible to inherit traits with vars in Java, therefor I had to change class 
structures there somewhat. 


> Add a Kafka TableSink with JSON serialization
> -
>
> Key: FLINK-3874
> URL: https://issues.apache.org/jira/browse/FLINK-3874
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Fabian Hueske
>Priority: Minor
>
> Add a TableSink that writes JSON serialized data to Kafka.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2244: [FLINK-3874] Add a Kafka TableSink with JSON serializatio...

[mushketyk]

Github user mushketyk commented on the issue:

https://github.com/apache/flink/pull/2244
  
I tried to inherit TableSink trait in Java code but it seems that it is 
impossible to inherit traits with vars in Java, therefor I had to change class 
structures there somewhat. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2244: Kafka json

[mushketyk]

GitHub user mushketyk opened a pull request:

https://github.com/apache/flink/pull/2244

Kafka json

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mushketyk/flink kafka-json

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2244.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2244


commit 9ff33698e44afc005360d8acb10fdbf2ccba814b
Author: Ivan Mushketyk 
Date:   2016-07-05T21:00:18Z

[FLINK-3874] Implement KafkaJsonTableSink

commit 3eeb1dcd0f4febe37f92725bc94f3d3b13e3368f
Author: Ivan Mushketyk 
Date:   2016-07-13T21:43:13Z

[FLINK-3874] Implement tests for CsvTableSink




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3477) Add hash-based combine strategy for ReduceFunction

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375658#comment-15375658
 ] 

ASF GitHub Bot commented on FLINK-3477:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/1517
  
CI tests are passing. I've been testing Gelly algorithms with this without 
error. I will merge this ...


> Add hash-based combine strategy for ReduceFunction
> --
>
> Key: FLINK-3477
> URL: https://issues.apache.org/jira/browse/FLINK-3477
> Project: Flink
>  Issue Type: Sub-task
>  Components: Local Runtime
>Reporter: Fabian Hueske
>Assignee: Gabor Gevay
>
> This issue is about adding a hash-based combine strategy for ReduceFunctions.
> The interface of the {{reduce()}} method is as follows:
> {code}
> public T reduce(T v1, T v2)
> {code}
> Input type and output type are identical and the function returns only a 
> single value. A Reduce function is incrementally applied to compute a final 
> aggregated value. This allows to hold the preaggregated value in a hash-table 
> and update it with each function call. 
> The hash-based strategy requires special implementation of an in-memory hash 
> table. The hash table should support in place updates of elements (if the 
> updated value has the same size as the new value) but also appending updates 
> with invalidation of the old value (if the binary length of the new value 
> differs). The hash table needs to be able to evict and emit all elements if 
> it runs out-of-memory.
> We should also add {{HASH}} and {{SORT}} compiler hints to 
> {{DataSet.reduce()}} and {{Grouping.reduce()}} to allow users to pick the 
> execution strategy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #1517: [FLINK-3477] [runtime] Add hash-based combine strategy fo...

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/1517
  
CI tests are passing. I've been testing Gelly algorithms with this without 
error. I will merge this ...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2243: [FLINK-4196] [runtime] Remove recovery timestamp f...

[StephanEwen]

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/2243

[FLINK-4196] [runtime] Remove recovery timestamp from checkpoint restores

The 'recoveryTimestamp' was an unsafe wall clock timestamp attached by the 
master upon recovery. Because this timestamp cannot be relied upon in 
distributed setups, it is removed here.

If we need something like this in the future, we should try and get a 
globally progress counter or logical timestamp instead.

No code in the core Flink repository is affected by this change.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink 
remove_recovery_timestamp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2243.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2243


commit ae65ef4b8ce35aadd089be46b52e7fddb5a3ef85
Author: Stephan Ewen 
Date:   2016-07-05T08:18:38Z

[hotfix] [kafka connector] Minor code cleanups in the Kafka Producer

commit c738fcd9f6031becc405c17cc479b9c2340c2414
Author: Stephan Ewen 
Date:   2016-07-11T18:36:44Z

[hotfix] [runtim] Minor code cleanups.

commit 116321241923194e9fa6db556681b333197fceed
Author: Stephan Ewen 
Date:   2016-07-13T15:31:35Z

[FLINK-4196] [runtime] Remove the 'recoveryTimestamp' from checkpoint 
restores.

The 'recoveryTimestamp' was an unsafe wall clock timestamp attached by the 
master
upon recovery. This this timestamp cannot be relied upon in distributed 
setups,
it is removed.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4196) Remove "recoveryTimestamp"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375630#comment-15375630
 ] 

ASF GitHub Bot commented on FLINK-4196:
---

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/2243

[FLINK-4196] [runtime] Remove recovery timestamp from checkpoint restores

The 'recoveryTimestamp' was an unsafe wall clock timestamp attached by the 
master upon recovery. Because this timestamp cannot be relied upon in 
distributed setups, it is removed here.

If we need something like this in the future, we should try and get a 
globally progress counter or logical timestamp instead.

No code in the core Flink repository is affected by this change.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink 
remove_recovery_timestamp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2243.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2243


commit ae65ef4b8ce35aadd089be46b52e7fddb5a3ef85
Author: Stephan Ewen 
Date:   2016-07-05T08:18:38Z

[hotfix] [kafka connector] Minor code cleanups in the Kafka Producer

commit c738fcd9f6031becc405c17cc479b9c2340c2414
Author: Stephan Ewen 
Date:   2016-07-11T18:36:44Z

[hotfix] [runtim] Minor code cleanups.

commit 116321241923194e9fa6db556681b333197fceed
Author: Stephan Ewen 
Date:   2016-07-13T15:31:35Z

[FLINK-4196] [runtime] Remove the 'recoveryTimestamp' from checkpoint 
restores.

The 'recoveryTimestamp' was an unsafe wall clock timestamp attached by the 
master
upon recovery. This this timestamp cannot be relied upon in distributed 
setups,
it is removed.




> Remove "recoveryTimestamp"
> --
>
> Key: FLINK-4196
> URL: https://issues.apache.org/jira/browse/FLINK-4196
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.0.3
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> I think we should remove the {{recoveryTimestamp}} that is attached on state 
> restore calls.
> Given that this is a wall clock timestamp from a master node, which may 
> change when clocks are adjusted, and between different master nodes during 
> leader change, this is an unsafe concept.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2242: ExceptionHandler keep count of exceptions

[sumitchawla]

GitHub user sumitchawla opened a pull request:

https://github.com/apache/flink/pull/2242

ExceptionHandler keep count of exceptions

This is just a bug identified while going through exception handler. There 
is no JIRA ticket for this.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sumitchawla/flink master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2242.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2242


commit 35a75cd672db8c40d591d2121f246fc1e642ae7c
Author: Sumit Chawla 
Date:   2016-07-13T19:16:01Z

increment the exceptions counter

commit 2afb0b32f750c5d77ed7da8f86e7ec78da27f3d5
Author: Sumit Chawla 
Date:   2016-07-13T19:18:12Z

Merge pull request #1 from sumitchawla/sumitchawla-exceptions-handler-fix

increment the exceptions counter




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2125) String delimiter for SocketTextStream

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375530#comment-15375530
 ] 

ASF GitHub Bot commented on FLINK-2125:
---

Github user delding commented on the issue:

https://github.com/apache/flink/pull/2233
  
Hi @zentol , thanks for your comments. I have updated this PR. Please let 
me know if there are further improvements that need to be done.


> String delimiter for SocketTextStream
> -
>
> Key: FLINK-2125
> URL: https://issues.apache.org/jira/browse/FLINK-2125
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: 0.9
>Reporter: Márton Balassi
>Priority: Minor
>  Labels: starter
>
> The SocketTextStreamFunction uses a character delimiter, despite other parts 
> of the API using String delimiter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2233: [FLINK-2125][streaming] Delimiter change from char to str...

[delding]

Github user delding commented on the issue:

https://github.com/apache/flink/pull/2233
  
Hi @zentol , thanks for your comments. I have updated this PR. Please let 
me know if there are further improvements that need to be done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2237: [FLINK-4206][metrics] Remove alphanumeric name restrictio...

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2237
  
Actually, I just wrote a more efficient version of a check that was there 
before.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4206) Metric names should allow special characters

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375505#comment-15375505
 ] 

ASF GitHub Bot commented on FLINK-4206:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2237
  
Actually, I just wrote a more efficient version of a check that was there 
before.


> Metric names should allow special characters
> 
>
> Key: FLINK-4206
> URL: https://issues.apache.org/jira/browse/FLINK-4206
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.1.0
>
>
> Currently, the name of the metric is restricted to alphanumeric characters. 
> This restriction was originally put in place to circumvent issues due to 
> systems not supporting certain characters.
> However, this restriction does not make a lot of sense since for group names 
> we don't enforce such a restriction.
> This also affects the integration of the Kafka metrics, so i suggest removing 
> the restriction.
> From now on it will be the responsibility of the reporter to make sure that 
> the metric identifier is supported by the external system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2228: [FLINK-4170][kinesis-connector] Simplify Kinesis connecte...

[tzulitai]

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/2228
  
Rebased again on the new endpoint config merged in 
https://github.com/apache/flink/pull/2227.
This PR should be ready for a final review now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4170) Remove `CONFIG_` prefix from KinesisConfigConstants variables

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375464#comment-15375464
 ] 

ASF GitHub Bot commented on FLINK-4170:
---

Github user tzulitai commented on the issue:

https://github.com/apache/flink/pull/2228
  
Rebased again on the new endpoint config merged in 
https://github.com/apache/flink/pull/2227.
This PR should be ready for a final review now.


> Remove `CONFIG_` prefix from KinesisConfigConstants variables
> -
>
> Key: FLINK-4170
> URL: https://issues.apache.org/jira/browse/FLINK-4170
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming Connectors
>Reporter: Ufuk Celebi
>Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.1.0
>
>
> I find the static variable names verbose. I think it's clear from context 
> that they refer to the Kinesis configuration since they are all gathered in 
> that class.
> Therefore would like to remove the {{CONFIG_}} prefix before the release, so 
> that we have
> {code}
> conf.put(KinesisConfigConstants.AWS_REGION, "")
> {code}
> instead of 
> {code}
> conf.put(KinesisConfigConstants.CONFIG_AWS_REGION, "")
> {code}
> For longer variables it becomes even longer otherwise.
> ---
> Some basic variable names that might be accessed frequently are also very 
> long:
> {code}
> CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_SECRETKEY
> CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_ACCESSKEYID
> {code}
> It might suffice to just have:
> {code}
> AWS_SECRET_KEY
> AWS_ACCESS_KEY
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3466) Job might get stuck in restoreState() from HDFS due to interrupt

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375397#comment-15375397
 ] 

Stephan Ewen commented on FLINK-3466:
-

Here is a Unit test that minimally reproduces getting stuck in interrupt 
sensitive state handles (like those reading from HDFS)

{code}
public class InterruptSensitiveRestoreTest {

private static final OneShotLatch IN_RESTORE_LATCH = new OneShotLatch();

@Test
public void testRestoreWithInterrupt() throws Exception {

Configuration taskConfig = new Configuration();
StreamConfig cfg = new StreamConfig(taskConfig);
cfg.setTimeCharacteristic(TimeCharacteristic.ProcessingTime);

TaskDeploymentDescriptor tdd = 
createTaskDeploymentDescriptor(taskConfig, new InterruptLockingStateHandle());
Task task = createTask(tdd);

// start the task and wait until it is in "restore"
task.startTaskThread();
IN_RESTORE_LATCH.await();

// trigger cancellation and signal to continue
task.cancelExecution();

task.getExecutingThread().join(3);

if (task.getExecutionState() == ExecutionState.CANCELING) {
fail("Task is stuck and not canceling");
}

assertEquals(ExecutionState.CANCELED, task.getExecutionState());
assertNull(task.getFailureCause());
}

// 

//  Utilities
// 


private static TaskDeploymentDescriptor createTaskDeploymentDescriptor(
Configuration taskConfig,
StateHandle state) throws IOException {
return new TaskDeploymentDescriptor(
new JobID(),
"test job name",
new JobVertexID(),
new ExecutionAttemptID(),
new SerializedValue<>(new ExecutionConfig()),
"test task name",
0, 1, 0,
new Configuration(),
taskConfig,
SourceStreamTask.class.getName(),

Collections.emptyList(),

Collections.emptyList(),
Collections.emptyList(),
Collections.emptyList(),
0,
new SerializedValue(state));
}

private static Task createTask(TaskDeploymentDescriptor tdd) throws 
IOException {
return new Task(
tdd,
mock(MemoryManager.class),
mock(IOManager.class),
mock(NetworkEnvironment.class),
mock(BroadcastVariableManager.class),
mock(ActorGateway.class),
mock(ActorGateway.class),
new FiniteDuration(10, TimeUnit.SECONDS),
new FallbackLibraryCacheManager(),
new FileCache(new Configuration()),
new TaskManagerRuntimeInfo(
"localhost", new 
Configuration(), EnvironmentInformation.getTemporaryFileDirectory()),
mock(TaskMetricGroup.class));

}

@SuppressWarnings("serial")
private static class InterruptLockingStateHandle extends 
StreamTaskStateList {

public InterruptLockingStateHandle() throws Exception {
super(new StreamTaskState[0]);
}

@Override
public StreamTaskState[] getState(ClassLoader 
userCodeClassLoader) {
IN_RESTORE_LATCH.trigger();

// this mimics what happens in the HDFS client code.
// an interrupt on a waiting object leads to an 
infinite loop
try {
synchronized (this) {
wait();
}
}
catch (InterruptedException e) {
while (true) {
try {
synchronized (this) {
 

[GitHub] flink issue #2237: [FLINK-4206][metrics] Remove alphanumeric name restrictio...

[rmetzger]

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2237
  
I stumbled across this and I found the limitation quite annoying, so if 
there are no good reasons for the check, I agree to remove it.
Since @StephanEwen wrote this, it might be good if he could confirm.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4206) Metric names should allow special characters

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375377#comment-15375377
 ] 

ASF GitHub Bot commented on FLINK-4206:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2237
  
I stumbled across this and I found the limitation quite annoying, so if 
there are no good reasons for the check, I agree to remove it.
Since @StephanEwen wrote this, it might be good if he could confirm.


> Metric names should allow special characters
> 
>
> Key: FLINK-4206
> URL: https://issues.apache.org/jira/browse/FLINK-4206
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.1.0
>
>
> Currently, the name of the metric is restricted to alphanumeric characters. 
> This restriction was originally put in place to circumvent issues due to 
> systems not supporting certain characters.
> However, this restriction does not make a lot of sense since for group names 
> we don't enforce such a restriction.
> This also affects the integration of the Kafka metrics, so i suggest removing 
> the restriction.
> From now on it will be the responsibility of the reporter to make sure that 
> the metric identifier is supported by the external system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2234: [hotfix][kinesis-connector] Remove duplicate info ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2234


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2234: [hotfix][kinesis-connector] Remove duplicate info in Kine...

[rmetzger]

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2234
  
Merging ...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (FLINK-4197) Allow Kinesis Endpoint to be Overridden via Config

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-4197.
---
   Resolution: Fixed
 Assignee: Scott Kidder
Fix Version/s: (was: 1.0.4)
   1.1.0

Resolved in http://git-wip-us.apache.org/repos/asf/flink/commit/bc3a96f5

Thank you for the contribution [~skidder].

> Allow Kinesis Endpoint to be Overridden via Config
> --
>
> Key: FLINK-4197
> URL: https://issues.apache.org/jira/browse/FLINK-4197
> Project: Flink
>  Issue Type: Improvement
>  Components: Kinesis Connector
>Affects Versions: 1.0.3
>Reporter: Scott Kidder
>Assignee: Scott Kidder
>Priority: Minor
>  Labels: easyfix
> Fix For: 1.1.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> I perform local testing of my application stack with Flink configured as a 
> consumer on a Kinesis stream provided by Kinesalite, an implementation of 
> Kinesis built on LevelDB. This requires me to override the AWS endpoint to 
> refer to my local Kinesalite server rather than reference the real AWS 
> endpoint. I'd like to add a configuration property to the Kinesis streaming 
> connector that allows the AWS endpoint to be specified explicitly.
> This should be a fairly small change and provide a lot of flexibility to 
> people looking to integrate Flink with Kinesis in a non-production setup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2227: [FLINK-4197] Allow Kinesis endpoint to be overridd...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2227


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4197) Allow Kinesis Endpoint to be Overridden via Config

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375360#comment-15375360
 ] 

ASF GitHub Bot commented on FLINK-4197:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2227


> Allow Kinesis Endpoint to be Overridden via Config
> --
>
> Key: FLINK-4197
> URL: https://issues.apache.org/jira/browse/FLINK-4197
> Project: Flink
>  Issue Type: Improvement
>  Components: Kinesis Connector
>Affects Versions: 1.0.3
>Reporter: Scott Kidder
>Priority: Minor
>  Labels: easyfix
> Fix For: 1.0.4
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> I perform local testing of my application stack with Flink configured as a 
> consumer on a Kinesis stream provided by Kinesalite, an implementation of 
> Kinesis built on LevelDB. This requires me to override the AWS endpoint to 
> refer to my local Kinesalite server rather than reference the real AWS 
> endpoint. I'd like to add a configuration property to the Kinesis streaming 
> connector that allows the AWS endpoint to be specified explicitly.
> This should be a fairly small change and provide a lot of flexibility to 
> people looking to integrate Flink with Kinesis in a non-production setup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-4152) TaskManager registration exponential backoff doesn't work

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375296#comment-15375296
 ] 

Till Rohrmann edited comment on FLINK-4152 at 7/13/16 4:24 PM:
---

The restarted registration attempts are the observable symptoms caused by a 
different problem. 

The actual problem is that the {{YarnFlinkRessourceManager}} forgets about the 
registered task managers if the job manager loses its leadership. Each task 
manager has a resource ID with which it registers at the resource manager. The 
{{YarnFlinkResourceManager}} has two states for allocated resources: 
{{containersInLaunch}} and {{registeredWorkers}}. A container can only go from 
{{containersInLaunch}} to {{registeredWorkers}}. This also works for the 
initial registration. However, when the job manager loses its leadership and 
the {{registeredWorkers}} list is cleared, there is no longer an container in 
launch associated with the respective resource ID. Consequently, when the old 
task manager is being re-registered by the new leader, the registration is 
rejected.

This rejection is then sent to the task manager. Upon receiving a rejection, 
the task manager reschedules another registration attempt after waiting for 
some time. Here the problem is that the old registration attempts are not 
cancelled. Consequently, one will have multiple registration attempts taking 
place at the "same" time/concurrently. That's the reason why you observe many 
registration attempt messages in the log.

I think the symptom can be fixed by cancelling all currently active 
registration attempts when you want to restart the registration.

It is a bit unclear to me what the expected behaviour of the 
FlinkYarnResourceManager should be. In the {{jobManagerLostLeadership}} method 
where the {{registeredWorkers}} list is cleared, a comment says "all currently 
registered TaskManagers are put under "awaiting registration"". But there is no 
such state. Furthermore, I'm not sure whether registered TaskManagers have to 
re-register if only the job manager has failed.

Thus, I see two solutions. Either not clearing {{registeredWorkers}} or 
introducing a new state "awaiting registration" which keeps all formerly 
registered task managers which can be re-registered.

Maybe [~mxm] can give some input.


was (Author: till.rohrmann):
[~mxm]The restarted registration attempts are the observable symptoms caused by 
a different problem. 

The actual problem is that the {{YarnFlinkRessourceManager}} forgets about the 
registered task managers if the job manager loses its leadership. Each task 
manager has a resource ID with which it registers at the resource manager. The 
{{YarnFlinkResourceManager}} has two states for allocated resources: 
{{containersInLaunch}} and {{registeredWorkers}}. A container can only go from 
{{containersInLaunch}} to {{registeredWorkers}}. This also works for the 
initial registration. However, when the job manager loses its leadership and 
the {{registeredWorkers}} list is cleared, there is no longer an container in 
launch associated with the respective resource ID. Consequently, when the old 
task manager is being re-registered by the new leader, the registration is 
rejected.

This rejection is then sent to the task manager. Upon receiving a rejection, 
the task manager reschedules another registration attempt after waiting for 
some time. Here the problem is that the old registration attempts are not 
cancelled. Consequently, one will have multiple registration attempts taking 
place at the "same" time/concurrently. That's the reason why you observe many 
registration attempt messages in the log.

I think the symptom can be fixed by cancelling all currently active 
registration attempts when you want to restart the registration.

It is a bit unclear to me what the expected behaviour of the 
FlinkYarnResourceManager should be. In the {{jobManagerLostLeadership}} method 
where the {{registeredWorkers}} list is cleared, a comment says "all currently 
registered TaskManagers are put under "awaiting registration"". But there is no 
such state. Furthermore, I'm not sure whether registered TaskManagers have to 
re-register if only the job manager has failed.

Thus, I see two solutions. Either not clearing {{registeredWorkers}} or 
introducing a new state "awaiting registration" which keeps all formerly 
registered task managers which can be re-registered.

Maybe [~mxm] can give some input.

> TaskManager registration exponential backoff doesn't work
> -
>
> Key: FLINK-4152
> URL: https://issues.apache.org/jira/browse/FLINK-4152
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination, TaskManager, YARN Client
>Reporter: Robert Metzger
>Assignee: Till Rohrmann
> 

[jira] [Commented] (FLINK-4152) TaskManager registration exponential backoff doesn't work

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375296#comment-15375296
 ] 

Till Rohrmann commented on FLINK-4152:
--

[~mxm]The restarted registration attempts are the observable symptoms caused by 
a different problem. 

The actual problem is that the {{YarnFlinkRessourceManager}} forgets about the 
registered task managers if the job manager loses its leadership. Each task 
manager has a resource ID with which it registers at the resource manager. The 
{{YarnFlinkResourceManager}} has two states for allocated resources: 
{{containersInLaunch}} and {{registeredWorkers}}. A container can only go from 
{{containersInLaunch}} to {{registeredWorkers}}. This also works for the 
initial registration. However, when the job manager loses its leadership and 
the {{registeredWorkers}} list is cleared, there is no longer an container in 
launch associated with the respective resource ID. Consequently, when the old 
task manager is being re-registered by the new leader, the registration is 
rejected.

This rejection is then sent to the task manager. Upon receiving a rejection, 
the task manager reschedules another registration attempt after waiting for 
some time. Here the problem is that the old registration attempts are not 
cancelled. Consequently, one will have multiple registration attempts taking 
place at the "same" time/concurrently. That's the reason why you observe many 
registration attempt messages in the log.

I think the symptom can be fixed by cancelling all currently active 
registration attempts when you want to restart the registration.

It is a bit unclear to me what the expected behaviour of the 
FlinkYarnResourceManager should be. In the {{jobManagerLostLeadership}} method 
where the {{registeredWorkers}} list is cleared, a comment says "all currently 
registered TaskManagers are put under "awaiting registration"". But there is no 
such state. Furthermore, I'm not sure whether registered TaskManagers have to 
re-register if only the job manager has failed.

Thus, I see two solutions. Either not clearing {{registeredWorkers}} or 
introducing a new state "awaiting registration" which keeps all formerly 
registered task managers which can be re-registered.

Maybe [~mxm] can give some input.

> TaskManager registration exponential backoff doesn't work
> -
>
> Key: FLINK-4152
> URL: https://issues.apache.org/jira/browse/FLINK-4152
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination, TaskManager, YARN Client
>Reporter: Robert Metzger
>Assignee: Till Rohrmann
> Attachments: logs.tgz
>
>
> While testing Flink 1.1 I've found that the TaskManagers are logging many 
> messages when registering at the JobManager.
> This is the log file: 
> https://gist.github.com/rmetzger/0cebe0419cdef4507b1e8a42e33ef294
> Its logging more than 3000 messages in less than a minute. I don't think that 
> this is the expected behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4212) Lock on pid file when starting daemons

[Greg Hogan (JIRA)]

Greg Hogan created FLINK-4212:
-

 Summary: Lock on pid file when starting daemons
 Key: FLINK-4212
 URL: https://issues.apache.org/jira/browse/FLINK-4212
 Project: Flink
  Issue Type: Improvement
  Components: Startup Shell Scripts
Affects Versions: 1.1.0
Reporter: Greg Hogan
Assignee: Greg Hogan


As noted on the mailing list (0), when multiple TaskManagers are started in 
parallel (using pdsh) there is a race condition on updating the pid: 1) the pid 
file is first read to parse the process' index, 2) the process is started, and 
3) on success the daemon pid is appended to the pid file.

We could use a tool such as {{flock}} to lock on the pid file while starting 
the Flink daemon.

0: 
http://mail-archives.apache.org/mod_mbox/flink-user/201607.mbox/%3CCA%2BssbKXw954Bz_sBRwP6db0FntWyGWzTyP7wJZ5nhOeQnof3kg%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2239: [FLINK-4208] Support Running Flink processes in foregroun...

[iemejia]

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2239
  
Hi, Yes, this is exactly the situation, in a previous pull request I was 
optimizing the flink docker image, however I found that the image used 
supervisord to catch and keep alive those daemons, so I wanted to remove this 
dependency (because it adds around 40MB to the image + python and some extra 
stuff).

Can you give me some hints on the best way to address this ? Or how can I 
improve my current approach (notice that I took the start-foreground idea from 
zookeeper).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4208) Support Running Flink processes in foreground mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375216#comment-15375216
 ] 

ASF GitHub Bot commented on FLINK-4208:
---

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2239
  
Hi, Yes, this is exactly the situation, in a previous pull request I was 
optimizing the flink docker image, however I found that the image used 
supervisord to catch and keep alive those daemons, so I wanted to remove this 
dependency (because it adds around 40MB to the image + python and some extra 
stuff).

Can you give me some hints on the best way to address this ? Or how can I 
improve my current approach (notice that I took the start-foreground idea from 
zookeeper).


> Support Running Flink processes in foreground mode
> --
>
> Key: FLINK-4208
> URL: https://issues.apache.org/jira/browse/FLINK-4208
> Project: Flink
>  Issue Type: Improvement
>Reporter: Ismaël Mejía
>Priority: Minor
>
> Flink clusters are started automatically in daemon mode, this is definitely 
> the default case, however if we want to start containers based on flinks, the 
> execution context gets lost. Running flink as foreground processes can fix 
> this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2239: [FLINK-4208] Support Running Flink processes in foregroun...

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2239
  
Hi @iemejia, is the situation with Docker that if the Flink processes are 
started as daemons and the script returns that Docker assumes the process has 
terminated?

Skipping the pid file might work fine for a container where one wouldn't 
start multiple TaskManagers but would cause logfile issues otherwise. We do 
need to add locking to the pid file.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4208) Support Running Flink processes in foreground mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375204#comment-15375204
 ] 

ASF GitHub Bot commented on FLINK-4208:
---

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/2239
  
Hi @iemejia, is the situation with Docker that if the Flink processes are 
started as daemons and the script returns that Docker assumes the process has 
terminated?

Skipping the pid file might work fine for a container where one wouldn't 
start multiple TaskManagers but would cause logfile issues otherwise. We do 
need to add locking to the pid file.


> Support Running Flink processes in foreground mode
> --
>
> Key: FLINK-4208
> URL: https://issues.apache.org/jira/browse/FLINK-4208
> Project: Flink
>  Issue Type: Improvement
>Reporter: Ismaël Mejía
>Priority: Minor
>
> Flink clusters are started automatically in daemon mode, this is definitely 
> the default case, however if we want to start containers based on flinks, the 
> execution context gets lost. Running flink as foreground processes can fix 
> this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4149) Fix Serialization of NFA in AbstractKeyedCEPPatternOperator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375195#comment-15375195
 ] 

ASF GitHub Bot commented on FLINK-4149:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2202
  
Thanks for the review @tillrohrmann! The `commons.io` was just an IDE 
mess-up. The `equals` I will fix.


> Fix Serialization of NFA in AbstractKeyedCEPPatternOperator
> ---
>
> Key: FLINK-4149
> URL: https://issues.apache.org/jira/browse/FLINK-4149
> Project: Flink
>  Issue Type: Bug
>  Components: CEP
>Affects Versions: 1.0.0, 1.1.0, 1.0.1, 1.0.2, 1.0.3
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.1.0
>
>
> A job that uses CEP fails upon restore with a {{NullPointerException}} in 
> {{NFA.process()}}. The reason seems to be that field {{computationStates}} is 
> {{null}}. This field is transient and read in a custom {{readObject()}} 
> method.
> In {{AbstractKeyedCEPPatternOperator}} this snipped is used to construct a 
> {{StateDescriptor}} for an {{NFA}} state:
> {code}
> new ValueStateDescriptor(
> NFA_OPERATOR_STATE_NAME,
> new KryoSerializer((Class) (Class) NFA.class, 
> getExecutionConfig()),
> null)
> {code}
> It seems Kryo does not invoke {{readObject}}/{{writeObject}}. We probably 
> need a custom {{TypeSerializer}} for {{NFA}} to solve the problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2202: [FLINK-4149] Fix Serialization of NFA in AbstractKeyedCEP...

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2202
  
Thanks for the review @tillrohrmann! The `commons.io` was just an IDE 
mess-up. The `equals` I will fix.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-4152) TaskManager registration exponential backoff doesn't work

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann reassigned FLINK-4152:


Assignee: Till Rohrmann

> TaskManager registration exponential backoff doesn't work
> -
>
> Key: FLINK-4152
> URL: https://issues.apache.org/jira/browse/FLINK-4152
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Coordination, TaskManager, YARN Client
>Reporter: Robert Metzger
>Assignee: Till Rohrmann
> Attachments: logs.tgz
>
>
> While testing Flink 1.1 I've found that the TaskManagers are logging many 
> messages when registering at the JobManager.
> This is the log file: 
> https://gist.github.com/rmetzger/0cebe0419cdef4507b1e8a42e33ef294
> Its logging more than 3000 messages in less than a minute. I don't think that 
> this is the expected behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4167) TaskMetricGroup does not close IOMetricGroup

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375178#comment-15375178
 ] 

ASF GitHub Bot commented on FLINK-4167:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2210


> TaskMetricGroup does not close IOMetricGroup
> 
>
> Key: FLINK-4167
> URL: https://issues.apache.org/jira/browse/FLINK-4167
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.1.0
>
>
> The {{TaskMetricGroup}} does not close the {{ioMetrics}} metric group. This 
> causes that metrics registered under the {{ioMetrics}} are not deregistered 
> after the termination of a job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2220: [FLINK-4184] [metrics] Replace invalid characters in Sche...

[tillrohrmann]

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2220
  
I rebased on the latest master and introduced a `CharacterFilter` 
interface. The `CharacterFilter` allows to filter out invalid characters while 
generating the fully qualified metric name. 

In order to do this, the `AbstractMetricGroup#generateMetricName` takes a 
`CharacterFilter` as argument. The `AbstractReporter` and the 
`ScheduledDropwizardReporter` implement this interface to filter out reporter 
specific characters.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4184) Ganglia and GraphiteReporter report metric names with invalid characters

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375162#comment-15375162
 ] 

ASF GitHub Bot commented on FLINK-4184:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2220
  
I rebased on the latest master and introduced a `CharacterFilter` 
interface. The `CharacterFilter` allows to filter out invalid characters while 
generating the fully qualified metric name. 

In order to do this, the `AbstractMetricGroup#generateMetricName` takes a 
`CharacterFilter` as argument. The `AbstractReporter` and the 
`ScheduledDropwizardReporter` implement this interface to filter out reporter 
specific characters.


> Ganglia and GraphiteReporter report metric names with invalid characters
> 
>
> Key: FLINK-4184
> URL: https://issues.apache.org/jira/browse/FLINK-4184
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.1.0
>
>
> Flink's {{GangliaReporter}} and {{GraphiteReporter}} report metrics with 
> names which contain invalid characters. For example, quotes are not filtered 
> out which can be problematic for Ganglia. Moreover, dots are not replaced 
> which causes Graphite to think that an IP address is actually a scoped metric 
> name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4167) TaskMetricGroup does not close IOMetricGroup

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-4167.

Resolution: Fixed

Fixed via d17fe4f636b56f2200444c59a0dead9010dfaa5d

> TaskMetricGroup does not close IOMetricGroup
> 
>
> Key: FLINK-4167
> URL: https://issues.apache.org/jira/browse/FLINK-4167
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.1.0
>
>
> The {{TaskMetricGroup}} does not close the {{ioMetrics}} metric group. This 
> causes that metrics registered under the {{ioMetrics}} are not deregistered 
> after the termination of a job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2210: [FLINK-4167] [metrics] Close IOMetricGroup in Task...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2210


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4167) TaskMetricGroup does not close IOMetricGroup

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375173#comment-15375173
 ] 

ASF GitHub Bot commented on FLINK-4167:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2210
  
Merging...


> TaskMetricGroup does not close IOMetricGroup
> 
>
> Key: FLINK-4167
> URL: https://issues.apache.org/jira/browse/FLINK-4167
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.1.0
>
>
> The {{TaskMetricGroup}} does not close the {{ioMetrics}} metric group. This 
> causes that metrics registered under the {{ioMetrics}} are not deregistered 
> after the termination of a job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2210: [FLINK-4167] [metrics] Close IOMetricGroup in TaskMetricG...

[tillrohrmann]

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2210
  
Merging...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4211) Dynamic Properties not working for jobs submitted to Yarn session

[Stefan Richter (JIRA)]

Stefan Richter created FLINK-4211:
-

 Summary: Dynamic Properties not working for jobs submitted to Yarn 
session
 Key: FLINK-4211
 URL: https://issues.apache.org/jira/browse/FLINK-4211
 Project: Flink
  Issue Type: Bug
  Components: YARN Client
Reporter: Stefan Richter


The command line argument for dynamic properties (-D) is not working when 
submitting jobs to a flink session.


Example:
{code}
bin/flink run -p 4 myJob.jar -D recovery.zookeeper.path.root=/flink/xyz
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4035) Bump Kafka producer in Kafka sink to Kafka 0.10.0.0

[Radoslaw Gruchalski (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375102#comment-15375102
 ] 

Radoslaw Gruchalski commented on FLINK-4035:


The tests are passing.

> Bump Kafka producer in Kafka sink to Kafka 0.10.0.0
> ---
>
> Key: FLINK-4035
> URL: https://issues.apache.org/jira/browse/FLINK-4035
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 1.0.3
>Reporter: Elias Levy
>Priority: Minor
>
> Kafka 0.10.0.0 introduced protocol changes related to the producer.  
> Published messages now include timestamps and compressed messages now include 
> relative offsets.  As it is now, brokers must decompress publisher compressed 
> messages, assign offset to them, and recompress them, which is wasteful and 
> makes it less likely that compression will be used at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-4196) Remove "recoveryTimestamp"

[Stephan Ewen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephan Ewen reassigned FLINK-4196:
---

Assignee: Stephan Ewen

> Remove "recoveryTimestamp"
> --
>
> Key: FLINK-4196
> URL: https://issues.apache.org/jira/browse/FLINK-4196
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.0.3
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>
> I think we should remove the {{recoveryTimestamp}} that is attached on state 
> restore calls.
> Given that this is a wall clock timestamp from a master node, which may 
> change when clocks are adjusted, and between different master nodes during 
> leader change, this is an unsafe concept.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3666) Remove Nephele references

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375078#comment-15375078
 ] 

ASF GitHub Bot commented on FLINK-3666:
---

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/2241

[FLINK-3666] Remove all remaining Nephele references



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 3666_bye_bye_nephele

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2241.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2241


commit f1d22ed2bfb2b4ec62a1ee86c201a40a0fa97573
Author: zentol 
Date:   2016-07-13T14:15:28Z

[FLINK-3666] Remove all remaining Nephele references




> Remove Nephele references
> -
>
> Key: FLINK-3666
> URL: https://issues.apache.org/jira/browse/FLINK-3666
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
>
> There still exist a few references to nephele which should be removed:
> {code}
> flink\docs\setup\local_setup.md:
>79  $ tail log/flink-*-jobmanager-*.log
>80  INFO ... - Initializing memory manager with 409 megabytes of memory
>81: INFO ... - Trying to load 
> org.apache.flinknephele.jobmanager.scheduler.local.LocalScheduler as scheduler
>82  INFO ... - Setting up web info server, using web-root directory ...
>83: INFO ... - Web info server will display information about nephele 
> job-manager on localhost, port 8081.
>84  INFO ... - Starting web info server for JobManager on port 8081
>85  ~~~
>..
>   118  $ cd flink
>   119  $ bin/start-local.sh
>   120: Starting Nephele job manager
>   121  ~~~
> {code}
> {code}
> flink\flink-runtime\src\main\java\org\apache\flink\runtime\operators\TaskContext.java:
>70:AbstractInvokable getOwningNepheleTask();
> {code}
> {code}
> flink\flink-runtime\src\main\java\org\apache\flink\runtime\operators\BatchTask.java:
>  1149  * @param message The main message for the log.
>  1150  * @param taskName The name of the task.
>  1151: * @param parent The nephele task that contains the code 
> producing the message.
>  1152  *
>  1153  * @return The string for logging.
>  
>  1254  */
>  1255 @SuppressWarnings("unchecked")
>  1256:public static  Collector initOutputs(AbstractInvokable 
> nepheleTask, ClassLoader cl, TaskConfig config,
>  1257 
> List chainedTasksTarget,
>  1258 
> List eventualOutputs,
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-3666) Remove Nephele references

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler reassigned FLINK-3666:
---

Assignee: Chesnay Schepler

> Remove Nephele references
> -
>
> Key: FLINK-3666
> URL: https://issues.apache.org/jira/browse/FLINK-3666
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
>
> There still exist a few references to nephele which should be removed:
> {code}
> flink\docs\setup\local_setup.md:
>79  $ tail log/flink-*-jobmanager-*.log
>80  INFO ... - Initializing memory manager with 409 megabytes of memory
>81: INFO ... - Trying to load 
> org.apache.flinknephele.jobmanager.scheduler.local.LocalScheduler as scheduler
>82  INFO ... - Setting up web info server, using web-root directory ...
>83: INFO ... - Web info server will display information about nephele 
> job-manager on localhost, port 8081.
>84  INFO ... - Starting web info server for JobManager on port 8081
>85  ~~~
>..
>   118  $ cd flink
>   119  $ bin/start-local.sh
>   120: Starting Nephele job manager
>   121  ~~~
> {code}
> {code}
> flink\flink-runtime\src\main\java\org\apache\flink\runtime\operators\TaskContext.java:
>70:AbstractInvokable getOwningNepheleTask();
> {code}
> {code}
> flink\flink-runtime\src\main\java\org\apache\flink\runtime\operators\BatchTask.java:
>  1149  * @param message The main message for the log.
>  1150  * @param taskName The name of the task.
>  1151: * @param parent The nephele task that contains the code 
> producing the message.
>  1152  *
>  1153  * @return The string for logging.
>  
>  1254  */
>  1255 @SuppressWarnings("unchecked")
>  1256:public static  Collector initOutputs(AbstractInvokable 
> nepheleTask, ClassLoader cl, TaskConfig config,
>  1257 
> List chainedTasksTarget,
>  1258 
> List eventualOutputs,
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2241: [FLINK-3666] Remove all remaining Nephele referenc...

[zentol]

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/2241

[FLINK-3666] Remove all remaining Nephele references



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 3666_bye_bye_nephele

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2241.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2241


commit f1d22ed2bfb2b4ec62a1ee86c201a40a0fa97573
Author: zentol 
Date:   2016-07-13T14:15:28Z

[FLINK-3666] Remove all remaining Nephele references




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2239: [FLINK-4208] Support Running Flink processes in foregroun...

[iemejia]

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2239
  
I hope this gets into 1.1.0 so I can push the changes to the docker image 
once this is available in the official binary distribution.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4207) WindowOperator becomes very slow with allowed lateness

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375072#comment-15375072
 ] 

Aljoscha Krettek commented on FLINK-4207:
-

It is somehow caused by the new cleanup timers. If the code in 
{{registerCleanupTimer}} is commented out then it performs as before.

> WindowOperator becomes very slow with allowed lateness
> --
>
> Key: FLINK-4207
> URL: https://issues.apache.org/jira/browse/FLINK-4207
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.1.0
>Reporter: Aljoscha Krettek
>Priority: Blocker
>
> In this simple example the throughput (as measured by the count the window 
> emits) becomes very low when an allowed lateness is set:
> {code}
> public class WindowWordCount {
>   public static void main(String[] args) throws Exception {
>   final StreamExecutionEnvironment env = 
> StreamExecutionEnvironment.getExecutionEnvironment();
>   
> env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
>   env.setParallelism(1);
>   env.addSource(new InfiniteTupleSource(100_000))
>   .keyBy(0)
>   .timeWindow(Time.seconds(3))
>   .allowedLateness(Time.seconds(1))
>   .reduce(new ReduceFunction Integer>>() {
>   @Override
>   public Tuple2 
> reduce(Tuple2 value1,
>   Tuple2 
> value2) throws Exception {
>   return Tuple2.of(value1.f0, 
> value1.f1 + value2.f1);
>   }
>   })
>   .filter(new FilterFunction Integer>>() {
>   private static final long 
> serialVersionUID = 1L;
>   @Override
>   public boolean filter(Tuple2 Integer> value) throws Exception {
>   return 
> value.f0.startsWith("Tuple 0");
>   }
>   })
>   .print();
>   // execute program
>   env.execute("WindowWordCount");
>   }
>   public static class InfiniteTupleSource implements 
> ParallelSourceFunction> {
>   private static final long serialVersionUID = 1L;
>   private int numGroups;
>   public InfiniteTupleSource(int numGroups) {
>   this.numGroups = numGroups;
>   }
>   @Override
>   public void run(SourceContext> out) 
> throws Exception {
>   long index = 0;
>   while (true) {
>   Tuple2 tuple = new 
> Tuple2<>("Tuple " + (index % numGroups), 1);
>   out.collect(tuple);
>   index++;
>   }
>   }
>   @Override
>   public void cancel() {
>   }
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4209) Docker image breaks with multiple NICs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375058#comment-15375058
 ] 

ASF GitHub Bot commented on FLINK-4209:
---

GitHub user iemejia opened a pull request:

https://github.com/apache/flink/pull/2240

[FLINK-4209] Docker image breaks with multiple NICs

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/flink FLINK-4209

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2240.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2240


commit 1af7c871a1aa9d446436df96f4b1e2d9115aee34
Author: Ismaël Mejía 
Date:   2016-07-05T02:16:07Z

[FLINK-4209] Change hostname resolution from IP to name

This solves issues when a host has multiple IPs

commit 385cb7f2ec2d629912b2a7af03f1ba1c76f49483
Author: Ismaël Mejía 
Date:   2016-07-05T02:17:09Z

[FLINK-4209] Separate build dependencies in the docker image and remove 
them once it is ready

commit 88102e54fde4c967831ae1cc3e4025543e5fed57
Author: Ismaël Mejía 
Date:   2016-07-05T07:23:46Z

[FLINK-4209] Add debug information of the build steps

commit b365fbd47db025dcf9d8d6d797ed7bdd85093b3f
Author: Ismaël Mejía 
Date:   2016-07-11T15:23:17Z

[FLINK-4209] Simplify docker-compose script (volumes are now local)




> Docker image breaks with multiple NICs
> --
>
> Key: FLINK-4209
> URL: https://issues.apache.org/jira/browse/FLINK-4209
> Project: Flink
>  Issue Type: Improvement
>  Components: flink-contrib
>Reporter: Ismaël Mejía
>Priority: Minor
>
> The resolution of the host is done by IP today in the docker image scripts, 
> this is an issue when the system has multiple network cards, if the hostname 
> resolution is done by name, this is fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4208) Support Running Flink processes in foreground mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375063#comment-15375063
 ] 

ASF GitHub Bot commented on FLINK-4208:
---

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2239
  
@aljoscha As discussed in FLINK-4118 I am doing this to support this use 
case and eventually remove the dependency on python and supervisord from the 
docker image.


> Support Running Flink processes in foreground mode
> --
>
> Key: FLINK-4208
> URL: https://issues.apache.org/jira/browse/FLINK-4208
> Project: Flink
>  Issue Type: Improvement
>Reporter: Ismaël Mejía
>Priority: Minor
>
> Flink clusters are started automatically in daemon mode, this is definitely 
> the default case, however if we want to start containers based on flinks, the 
> execution context gets lost. Running flink as foreground processes can fix 
> this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4208) Support Running Flink processes in foreground mode

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375068#comment-15375068
 ] 

ASF GitHub Bot commented on FLINK-4208:
---

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2239
  
I hope this gets into 1.1.0 so I can push the changes to the docker image 
once this is available in the official binary distribution.


> Support Running Flink processes in foreground mode
> --
>
> Key: FLINK-4208
> URL: https://issues.apache.org/jira/browse/FLINK-4208
> Project: Flink
>  Issue Type: Improvement
>Reporter: Ismaël Mejía
>Priority: Minor
>
> Flink clusters are started automatically in daemon mode, this is definitely 
> the default case, however if we want to start containers based on flinks, the 
> execution context gets lost. Running flink as foreground processes can fix 
> this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2239: [FLINK-4208] Support Running Flink processes in foregroun...

[iemejia]

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2239
  
@aljoscha As discussed in FLINK-4118 I am doing this to support this use 
case and eventually remove the dependency on python and supervisord from the 
docker image.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4209) Docker image breaks with multiple NICs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375062#comment-15375062
 ] 

ASF GitHub Bot commented on FLINK-4209:
---

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2240
  
I added some additional fixes for example to exclude build dependencies 
from the final image, and to make the docker-compose script simpler.
@aljoscha This is a set of small fixes that I missed at FLINK-4118.


> Docker image breaks with multiple NICs
> --
>
> Key: FLINK-4209
> URL: https://issues.apache.org/jira/browse/FLINK-4209
> Project: Flink
>  Issue Type: Improvement
>  Components: flink-contrib
>Reporter: Ismaël Mejía
>Priority: Minor
>
> The resolution of the host is done by IP today in the docker image scripts, 
> this is an issue when the system has multiple network cards, if the hostname 
> resolution is done by name, this is fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2240: [FLINK-4209] Docker image breaks with multiple NICs

[iemejia]

Github user iemejia commented on the issue:

https://github.com/apache/flink/pull/2240
  
I added some additional fixes for example to exclude build dependencies 
from the final image, and to make the docker-compose script simpler.
@aljoscha This is a set of small fixes that I missed at FLINK-4118.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2240: [FLINK-4209] Docker image breaks with multiple NIC...

[iemejia]

GitHub user iemejia opened a pull request:

https://github.com/apache/flink/pull/2240

[FLINK-4209] Docker image breaks with multiple NICs

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/flink FLINK-4209

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2240.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2240


commit 1af7c871a1aa9d446436df96f4b1e2d9115aee34
Author: Ismaël Mejía 
Date:   2016-07-05T02:16:07Z

[FLINK-4209] Change hostname resolution from IP to name

This solves issues when a host has multiple IPs

commit 385cb7f2ec2d629912b2a7af03f1ba1c76f49483
Author: Ismaël Mejía 
Date:   2016-07-05T02:17:09Z

[FLINK-4209] Separate build dependencies in the docker image and remove 
them once it is ready

commit 88102e54fde4c967831ae1cc3e4025543e5fed57
Author: Ismaël Mejía 
Date:   2016-07-05T07:23:46Z

[FLINK-4209] Add debug information of the build steps

commit b365fbd47db025dcf9d8d6d797ed7bdd85093b3f
Author: Ismaël Mejía 
Date:   2016-07-11T15:23:17Z

[FLINK-4209] Simplify docker-compose script (volumes are now local)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2239: [FLINK-4208] Support Running Flink processes in fo...

[iemejia]

GitHub user iemejia opened a pull request:

https://github.com/apache/flink/pull/2239

[FLINK-4208] Support Running Flink processes in foreground mode

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/flink FLINK-4208

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2239.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2239


commit af9ac6cdb3f6601d6248abe82df4fd44de4453e5
Author: Ismaël Mejía 
Date:   2016-07-13T13:55:13Z

[FLINK-4208] Add support to run flink as a foreground process

commit 5a27c3ce0bfaf7cf862aba3929f7712a3897bc19
Author: Ismaël Mejía 
Date:   2016-07-13T13:56:13Z

[FLINK-4208] Make the flink daemon a real daemon
nohup ignores the HUP terminal signals, so the process is still alive
even after the terminal ends




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2125) String delimiter for SocketTextStream

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375020#comment-15375020
 ] 

ASF GitHub Bot commented on FLINK-2125:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2233
  
Only had 2 small comments, otherwise this looks good to me.


> String delimiter for SocketTextStream
> -
>
> Key: FLINK-2125
> URL: https://issues.apache.org/jira/browse/FLINK-2125
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: 0.9
>Reporter: Márton Balassi
>Priority: Minor
>  Labels: starter
>
> The SocketTextStreamFunction uses a character delimiter, despite other parts 
> of the API using String delimiter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2233: [FLINK-2125][streaming] Delimiter change from char to str...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2233
  
Only had 2 small comments, otherwise this looks good to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4210) Move close()/isClosed() out of MetricGroup interface

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-4210:
---

 Summary: Move close()/isClosed() out of MetricGroup interface
 Key: FLINK-4210
 URL: https://issues.apache.org/jira/browse/FLINK-4210
 Project: Flink
  Issue Type: Improvement
  Components: Metrics
Affects Versions: 1.1.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
Priority: Minor
 Fix For: 1.1.0


The (user-facing) MetricGroup interface currently exposes a close() and 
isClosed() method which generally users shouldn't need to call. They are an 
internal thing, and thus should be moved into the AbstractMetricGroup class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4208) Support Running Flink processes in foreground mode

[JIRA]

Ismaël Mejía created FLINK-4208:
---

 Summary: Support Running Flink processes in foreground mode
 Key: FLINK-4208
 URL: https://issues.apache.org/jira/browse/FLINK-4208
 Project: Flink
  Issue Type: Improvement
Reporter: Ismaël Mejía
Priority: Minor


Flink clusters are started automatically in daemon mode, this is definitely the 
default case, however if we want to start containers based on flinks, the 
execution context gets lost. Running flink as foreground processes can fix this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4209) Docker image breaks with multiple NICs

[JIRA]

Ismaël Mejía created FLINK-4209:
---

 Summary: Docker image breaks with multiple NICs
 Key: FLINK-4209
 URL: https://issues.apache.org/jira/browse/FLINK-4209
 Project: Flink
  Issue Type: Improvement
  Components: flink-contrib
Reporter: Ismaël Mejía
Priority: Minor


The resolution of the host is done by IP today in the docker image scripts, 
this is an issue when the system has multiple network cards, if the hostname 
resolution is done by name, this is fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4192) Move Metrics API to separate module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374919#comment-15374919
 ] 

ASF GitHub Bot commented on FLINK-4192:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2226
  
Alright, although I am curious why this wasn't an issue when we merged the 
metric system initially.

Since you mentioned that we should/could add additional helper methods, 
please provide a full list of methods that you think are necessary. Otherwise 
this will turn into longer back-and-forth than necessary.


> Move Metrics API to separate module
> ---
>
> Key: FLINK-4192
> URL: https://issues.apache.org/jira/browse/FLINK-4192
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.1.0
>
>
> All metrics code currently resides in flink-core. If a user implements a 
> reporter and wants a fat jar it will now have to include the entire 
> flink-core module.
> Instead, we could move several interfaces into a separate module.
> These interfaces to move include:
> * Counter, Gauge, Histogram(Statistics)
> * MetricGroup
> * MetricReporter, Scheduled, AbstractReporter
> In addition a new MetricRegistry interface will be required as well as a 
> replacement for the Configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2226: [FLINK-4192] - Move Metrics API to separate module

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2226
  
Alright, although I am curious why this wasn't an issue when we merged the 
metric system initially.

Since you mentioned that we should/could add additional helper methods, 
please provide a full list of methods that you think are necessary. Otherwise 
this will turn into longer back-and-forth than necessary.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2125) String delimiter for SocketTextStream

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374911#comment-15374911
 ] 

ASF GitHub Bot commented on FLINK-2125:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2233#discussion_r70614809
  
--- Diff: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/SocketTextStreamFunction.java
 ---
@@ -96,19 +96,21 @@ public void run(SourceContext ctx) throws 
Exception {
socket.connect(new InetSocketAddress(hostname, 
port), CONNECTION_TIMEOUT_TIME);
BufferedReader reader = new BufferedReader(new 
InputStreamReader(socket.getInputStream()));
 
-   int data;
-   while (isRunning && (data = reader.read()) != 
-1) {
-   // check if the string is complete
-   if (data != delimiter) {
-   buffer.append((char) data);
-   }
-   else {
+   char[] cbuf = new char[8192];
+   int byteRead;
--- End diff --

This variable should be named `bytesRead` .


> String delimiter for SocketTextStream
> -
>
> Key: FLINK-2125
> URL: https://issues.apache.org/jira/browse/FLINK-2125
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: 0.9
>Reporter: Márton Balassi
>Priority: Minor
>  Labels: starter
>
> The SocketTextStreamFunction uses a character delimiter, despite other parts 
> of the API using String delimiter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2233: [FLINK-2125][streaming] Delimiter change from char...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2233#discussion_r70614809
  
--- Diff: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/SocketTextStreamFunction.java
 ---
@@ -96,19 +96,21 @@ public void run(SourceContext ctx) throws 
Exception {
socket.connect(new InetSocketAddress(hostname, 
port), CONNECTION_TIMEOUT_TIME);
BufferedReader reader = new BufferedReader(new 
InputStreamReader(socket.getInputStream()));
 
-   int data;
-   while (isRunning && (data = reader.read()) != 
-1) {
-   // check if the string is complete
-   if (data != delimiter) {
-   buffer.append((char) data);
-   }
-   else {
+   char[] cbuf = new char[8192];
+   int byteRead;
--- End diff --

This variable should be named `bytesRead` .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4192) Move Metrics API to separate module

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374910#comment-15374910
 ] 

ASF GitHub Bot commented on FLINK-4192:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2226
  
The point of using `Properties` is that users of the API can use a class 
they are familiar with and maybe even reuse configurations and tools they 
internally that work with `Properties` types. Adding yet another config type 
means users may have to integrate yet another config type with their own tools 
and utils.


> Move Metrics API to separate module
> ---
>
> Key: FLINK-4192
> URL: https://issues.apache.org/jira/browse/FLINK-4192
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
> Fix For: 1.1.0
>
>
> All metrics code currently resides in flink-core. If a user implements a 
> reporter and wants a fat jar it will now have to include the entire 
> flink-core module.
> Instead, we could move several interfaces into a separate module.
> These interfaces to move include:
> * Counter, Gauge, Histogram(Statistics)
> * MetricGroup
> * MetricReporter, Scheduled, AbstractReporter
> In addition a new MetricRegistry interface will be required as well as a 
> replacement for the Configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4167) TaskMetricGroup does not close IOMetricGroup

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374907#comment-15374907
 ] 

ASF GitHub Bot commented on FLINK-4167:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/2210#discussion_r70614548
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/metrics/groups/ProxyMetricGroup.java 
---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.groups;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.util.Preconditions;
+
+/**
+ * Metric group which forwards all registration calls to its parent metric 
group.
+ *
+ * @param  Type of the parent metric group
+ */
+@Internal
+public class ProxyMetricGroup implements 
MetricGroup {
+   private final P parentMetricGroup;
+
+   public ProxyMetricGroup(P parentMetricGroup) {
+   this.parentMetricGroup = 
Preconditions.checkNotNull(parentMetricGroup);
+   }
+
+   @Override
+   public final void close() {
+   parentMetricGroup.close();
--- End diff --

Hmm you're right, it is a bit tricky. You don't want to close the complete 
parent metric group. But I guess you would want to unregister the metrics 
registered by the `IOMetricGroup` if you call `IOMetricGroup#close`. However, 
this is not really possible at the moment. I will remove the `parent.close` 
call then.

I agree that `close` and `isClosed` should only be used internally and not 
being exposed to the user. This could be a follow-up task.


> TaskMetricGroup does not close IOMetricGroup
> 
>
> Key: FLINK-4167
> URL: https://issues.apache.org/jira/browse/FLINK-4167
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.1.0
>
>
> The {{TaskMetricGroup}} does not close the {{ioMetrics}} metric group. This 
> causes that metrics registered under the {{ioMetrics}} are not deregistered 
> after the termination of a job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2226: [FLINK-4192] - Move Metrics API to separate module

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2226
  
The point of using `Properties` is that users of the API can use a class 
they are familiar with and maybe even reuse configurations and tools they 
internally that work with `Properties` types. Adding yet another config type 
means users may have to integrate yet another config type with their own tools 
and utils.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2210: [FLINK-4167] [metrics] Close IOMetricGroup in Task...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/2210#discussion_r70614548
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/metrics/groups/ProxyMetricGroup.java 
---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.groups;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.util.Preconditions;
+
+/**
+ * Metric group which forwards all registration calls to its parent metric 
group.
+ *
+ * @param  Type of the parent metric group
+ */
+@Internal
+public class ProxyMetricGroup implements 
MetricGroup {
+   private final P parentMetricGroup;
+
+   public ProxyMetricGroup(P parentMetricGroup) {
+   this.parentMetricGroup = 
Preconditions.checkNotNull(parentMetricGroup);
+   }
+
+   @Override
+   public final void close() {
+   parentMetricGroup.close();
--- End diff --

Hmm you're right, it is a bit tricky. You don't want to close the complete 
parent metric group. But I guess you would want to unregister the metrics 
registered by the `IOMetricGroup` if you call `IOMetricGroup#close`. However, 
this is not really possible at the moment. I will remove the `parent.close` 
call then.

I agree that `close` and `isClosed` should only be used internally and not 
being exposed to the user. This could be a follow-up task.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2233: [FLINK-2125][streaming] Delimiter change from char...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2233#discussion_r70614465
  
--- Diff: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/SocketTextStreamFunction.java
 ---
@@ -96,19 +96,21 @@ public void run(SourceContext ctx) throws 
Exception {
socket.connect(new InetSocketAddress(hostname, 
port), CONNECTION_TIMEOUT_TIME);
BufferedReader reader = new BufferedReader(new 
InputStreamReader(socket.getInputStream()));
 
-   int data;
-   while (isRunning && (data = reader.read()) != 
-1) {
-   // check if the string is complete
-   if (data != delimiter) {
-   buffer.append((char) data);
-   }
-   else {
+   char[] cbuf = new char[8192];
+   int byteRead;
+   while (isRunning && (byteRead = 
reader.read(cbuf)) != -1) {
+   buffer.append(cbuf, 0, byteRead);
+   int delimPos;
+   while (buffer.length() >= 
delimiter.length() && (delimPos = buffer.indexOf(delimiter)) != -1) {
+   String token = 
buffer.substring(0, delimPos);
// truncate trailing carriage 
return
-   if (delimiter == '\n' && 
buffer.length() > 0 && buffer.charAt(buffer.length() - 1) == '\r') {
-   
buffer.setLength(buffer.length() - 1);
+   if (delimiter.equals("\n") && 
token.endsWith("\r")) {
+   token = 
token.substring(0, token.length() - 1);
+   }
+   if (!token.isEmpty()) {
+   ctx.collect(token);
--- End diff --

This changes existing behavior. Empty strings are filtered out, which 
wasn't done before.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2125) String delimiter for SocketTextStream

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374902#comment-15374902
 ] 

ASF GitHub Bot commented on FLINK-2125:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2233#discussion_r70614465
  
--- Diff: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/SocketTextStreamFunction.java
 ---
@@ -96,19 +96,21 @@ public void run(SourceContext ctx) throws 
Exception {
socket.connect(new InetSocketAddress(hostname, 
port), CONNECTION_TIMEOUT_TIME);
BufferedReader reader = new BufferedReader(new 
InputStreamReader(socket.getInputStream()));
 
-   int data;
-   while (isRunning && (data = reader.read()) != 
-1) {
-   // check if the string is complete
-   if (data != delimiter) {
-   buffer.append((char) data);
-   }
-   else {
+   char[] cbuf = new char[8192];
+   int byteRead;
+   while (isRunning && (byteRead = 
reader.read(cbuf)) != -1) {
+   buffer.append(cbuf, 0, byteRead);
+   int delimPos;
+   while (buffer.length() >= 
delimiter.length() && (delimPos = buffer.indexOf(delimiter)) != -1) {
+   String token = 
buffer.substring(0, delimPos);
// truncate trailing carriage 
return
-   if (delimiter == '\n' && 
buffer.length() > 0 && buffer.charAt(buffer.length() - 1) == '\r') {
-   
buffer.setLength(buffer.length() - 1);
+   if (delimiter.equals("\n") && 
token.endsWith("\r")) {
+   token = 
token.substring(0, token.length() - 1);
+   }
+   if (!token.isEmpty()) {
+   ctx.collect(token);
--- End diff --

This changes existing behavior. Empty strings are filtered out, which 
wasn't done before.


> String delimiter for SocketTextStream
> -
>
> Key: FLINK-2125
> URL: https://issues.apache.org/jira/browse/FLINK-2125
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: 0.9
>Reporter: Márton Balassi
>Priority: Minor
>  Labels: starter
>
> The SocketTextStreamFunction uses a character delimiter, despite other parts 
> of the API using String delimiter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-4207) WindowOperator becomes very slow with allowed lateness

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374897#comment-15374897
 ] 

Stephan Ewen edited comment on FLINK-4207 at 7/13/16 12:20 PM:
---

Is the overhead in the
  - "onElement()" processing?
  - the window triggering?
  - the window cleanups?


was (Author: stephanewen):
What is happening different now than before?

> WindowOperator becomes very slow with allowed lateness
> --
>
> Key: FLINK-4207
> URL: https://issues.apache.org/jira/browse/FLINK-4207
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.1.0
>Reporter: Aljoscha Krettek
>Priority: Blocker
>
> In this simple example the throughput (as measured by the count the window 
> emits) becomes very low when an allowed lateness is set:
> {code}
> public class WindowWordCount {
>   public static void main(String[] args) throws Exception {
>   final StreamExecutionEnvironment env = 
> StreamExecutionEnvironment.getExecutionEnvironment();
>   
> env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
>   env.setParallelism(1);
>   env.addSource(new InfiniteTupleSource(100_000))
>   .keyBy(0)
>   .timeWindow(Time.seconds(3))
>   .allowedLateness(Time.seconds(1))
>   .reduce(new ReduceFunction Integer>>() {
>   @Override
>   public Tuple2 
> reduce(Tuple2 value1,
>   Tuple2 
> value2) throws Exception {
>   return Tuple2.of(value1.f0, 
> value1.f1 + value2.f1);
>   }
>   })
>   .filter(new FilterFunction Integer>>() {
>   private static final long 
> serialVersionUID = 1L;
>   @Override
>   public boolean filter(Tuple2 Integer> value) throws Exception {
>   return 
> value.f0.startsWith("Tuple 0");
>   }
>   })
>   .print();
>   // execute program
>   env.execute("WindowWordCount");
>   }
>   public static class InfiniteTupleSource implements 
> ParallelSourceFunction> {
>   private static final long serialVersionUID = 1L;
>   private int numGroups;
>   public InfiniteTupleSource(int numGroups) {
>   this.numGroups = numGroups;
>   }
>   @Override
>   public void run(SourceContext> out) 
> throws Exception {
>   long index = 0;
>   while (true) {
>   Tuple2 tuple = new 
> Tuple2<>("Tuple " + (index % numGroups), 1);
>   out.collect(tuple);
>   index++;
>   }
>   }
>   @Override
>   public void cancel() {
>   }
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4207) WindowOperator becomes very slow with allowed lateness

[Stephan Ewen (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374897#comment-15374897
 ] 

Stephan Ewen commented on FLINK-4207:
-

What is happening different now than before?

> WindowOperator becomes very slow with allowed lateness
> --
>
> Key: FLINK-4207
> URL: https://issues.apache.org/jira/browse/FLINK-4207
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.1.0
>Reporter: Aljoscha Krettek
>Priority: Blocker
>
> In this simple example the throughput (as measured by the count the window 
> emits) becomes very low when an allowed lateness is set:
> {code}
> public class WindowWordCount {
>   public static void main(String[] args) throws Exception {
>   final StreamExecutionEnvironment env = 
> StreamExecutionEnvironment.getExecutionEnvironment();
>   
> env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
>   env.setParallelism(1);
>   env.addSource(new InfiniteTupleSource(100_000))
>   .keyBy(0)
>   .timeWindow(Time.seconds(3))
>   .allowedLateness(Time.seconds(1))
>   .reduce(new ReduceFunction Integer>>() {
>   @Override
>   public Tuple2 
> reduce(Tuple2 value1,
>   Tuple2 
> value2) throws Exception {
>   return Tuple2.of(value1.f0, 
> value1.f1 + value2.f1);
>   }
>   })
>   .filter(new FilterFunction Integer>>() {
>   private static final long 
> serialVersionUID = 1L;
>   @Override
>   public boolean filter(Tuple2 Integer> value) throws Exception {
>   return 
> value.f0.startsWith("Tuple 0");
>   }
>   })
>   .print();
>   // execute program
>   env.execute("WindowWordCount");
>   }
>   public static class InfiniteTupleSource implements 
> ParallelSourceFunction> {
>   private static final long serialVersionUID = 1L;
>   private int numGroups;
>   public InfiniteTupleSource(int numGroups) {
>   this.numGroups = numGroups;
>   }
>   @Override
>   public void run(SourceContext> out) 
> throws Exception {
>   long index = 0;
>   while (true) {
>   Tuple2 tuple = new 
> Tuple2<>("Tuple " + (index % numGroups), 1);
>   out.collect(tuple);
>   index++;
>   }
>   }
>   @Override
>   public void cancel() {
>   }
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2235: [hotfix] removed duplicated code

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2235


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3729) Several SQL tests fail on Windows OS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374889#comment-15374889
 ] 

ASF GitHub Bot commented on FLINK-3729:
---

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/2238

[FLINK-3729] Fix Several SQL tests on Windows OS 

- fixed ExplainTest by replacing all occurrences of `\r\n` with `\n` before 
comparing the output
- replaced usages of `\n` in BatchTableEnvironment#explain() with 
System.lineSeparator
- fixed TableSinkITCase by generating the qualidifed path using 
`File#toUri()` and `URI#toString()`
- removed a lot of unused imports in TableSinkITCase

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 3729_sql_tests_windows

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2238.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2238


commit 0694fde595289b4ae145544fb39598fee63e8bf5
Author: zentol 
Date:   2016-07-13T11:51:05Z

[FLINK-3729][table] Fix ExplainTest on Windows OS

commit d79866a78b30d6794c6c96431ac53d6a5119890d
Author: zentol 
Date:   2016-07-13T11:51:34Z

[FLINK-3729][table] Fix TableSinkITCase on Windows OS

commit 31c121c25cae70262c0187e43b41a144a326ab33
Author: zentol 
Date:   2016-07-13T11:51:53Z

[hotfix][table] Remove unused imports in TableSinkITCase




> Several SQL tests fail on Windows OS
> 
>
> Key: FLINK-3729
> URL: https://issues.apache.org/jira/browse/FLINK-3729
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.0.1
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>
> The Table API SqlExplain(Test/ITCase) fail categorically on Windows due to 
> different line-endings. These tests generate an string representation of an 
> abstract syntax tree; problem is there is a difference in line-endings.
> The expected ones contain LF, the actual one CRLF.
> The tests should be either changed to either
> * include CRLF line-endings in the expected string when run on windows
> * always use LF line-endings regardless of OS
> * use a compare method that is aware of this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2235: [hotfix] removed duplicated code

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2235
  
merging


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2238: [FLINK-3729] Fix Several SQL tests on Windows OS

[zentol]

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/2238

[FLINK-3729] Fix Several SQL tests on Windows OS 

- fixed ExplainTest by replacing all occurrences of `\r\n` with `\n` before 
comparing the output
- replaced usages of `\n` in BatchTableEnvironment#explain() with 
System.lineSeparator
- fixed TableSinkITCase by generating the qualidifed path using 
`File#toUri()` and `URI#toString()`
- removed a lot of unused imports in TableSinkITCase

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 3729_sql_tests_windows

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2238.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2238


commit 0694fde595289b4ae145544fb39598fee63e8bf5
Author: zentol 
Date:   2016-07-13T11:51:05Z

[FLINK-3729][table] Fix ExplainTest on Windows OS

commit d79866a78b30d6794c6c96431ac53d6a5119890d
Author: zentol 
Date:   2016-07-13T11:51:34Z

[FLINK-3729][table] Fix TableSinkITCase on Windows OS

commit 31c121c25cae70262c0187e43b41a144a326ab33
Author: zentol 
Date:   2016-07-13T11:51:53Z

[hotfix][table] Remove unused imports in TableSinkITCase




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4204) Clean up gelly-examples

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374877#comment-15374877
 ] 

Greg Hogan commented on FLINK-4204:
---

We should also add a {{Main-Class}} to {{flink-gelly-examples}} to print usage 
for running the drivers and example programs. Currently the only means to 
discover the available classes are to read the source or list classes in the 
jar.

> Clean up gelly-examples
> ---
>
> Key: FLINK-4204
> URL: https://issues.apache.org/jira/browse/FLINK-4204
> Project: Flink
>  Issue Type: Improvement
>  Components: Gelly
>Affects Versions: 1.1.0
>Reporter: Vasia Kalavri
>
> The gelly-examples has grown quite big (14 examples) and contains several 
> examples that illustrate the same functionality. Examples should help users 
> understand how to use the API and ideally show how to use 1-2 features.
> Also, it is helpful to state the purpose of each example in the comments.
> We should keep the example set small and move everything that does not fit 
> there to the library.
> I propose to remove the following:
> - ClusteringCoefficient: the functionality already exists as a library method.
> - HITS: the functionality already exists as a library method.
> - JaccardIndex: the functionality already exists as a library method.
> - SingleSourceShortestPaths: the example shows how to use scatter-gather 
> iterations. HITSAlgorithm shows the same feature plus the use of aggregators. 
> I propose we keep this one instead.
> - TriangleListing: the functionality already exists as a library method



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-3729) Several SQL tests fail on Windows OS

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler reassigned FLINK-3729:
---

Assignee: Chesnay Schepler

> Several SQL tests fail on Windows OS
> 
>
> Key: FLINK-3729
> URL: https://issues.apache.org/jira/browse/FLINK-3729
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Affects Versions: 1.0.1
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>
> The Table API SqlExplain(Test/ITCase) fail categorically on Windows due to 
> different line-endings. These tests generate an string representation of an 
> abstract syntax tree; problem is there is a difference in line-endings.
> The expected ones contain LF, the actual one CRLF.
> The tests should be either changed to either
> * include CRLF line-endings in the expected string when run on windows
> * always use LF line-endings regardless of OS
> * use a compare method that is aware of this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4167) TaskMetricGroup does not close IOMetricGroup

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374776#comment-15374776
 ] 

ASF GitHub Bot commented on FLINK-4167:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2210
  
Only had a small comment, otherwise +1.


> TaskMetricGroup does not close IOMetricGroup
> 
>
> Key: FLINK-4167
> URL: https://issues.apache.org/jira/browse/FLINK-4167
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.1.0
>
>
> The {{TaskMetricGroup}} does not close the {{ioMetrics}} metric group. This 
> causes that metrics registered under the {{ioMetrics}} are not deregistered 
> after the termination of a job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2210: [FLINK-4167] [metrics] Close IOMetricGroup in TaskMetricG...

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2210
  
Only had a small comment, otherwise +1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4167) TaskMetricGroup does not close IOMetricGroup

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374764#comment-15374764
 ] 

ASF GitHub Bot commented on FLINK-4167:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2210#discussion_r70597751
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/metrics/groups/ProxyMetricGroup.java 
---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.groups;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.util.Preconditions;
+
+/**
+ * Metric group which forwards all registration calls to its parent metric 
group.
+ *
+ * @param  Type of the parent metric group
+ */
+@Internal
+public class ProxyMetricGroup implements 
MetricGroup {
+   private final P parentMetricGroup;
+
+   public ProxyMetricGroup(P parentMetricGroup) {
+   this.parentMetricGroup = 
Preconditions.checkNotNull(parentMetricGroup);
+   }
+
+   @Override
+   public final void close() {
+   parentMetricGroup.close();
--- End diff --

It is safer to not do anything here. `close()` should only be called by the 
Task on the `TaskMetricGroup`, this would open us to the possibility of 
components closing the TaskMG as well.

There's also the looming StackOverflow when someone puts 
`ioMetrics.close()` into the `TaskMetricGroup#close()`.

Now that i think about it i believe `close()` (and by extension, 
`isClosed()`) has no business being in the MetricGroup interface in the first 
place, as users actually don't need to call it.


> TaskMetricGroup does not close IOMetricGroup
> 
>
> Key: FLINK-4167
> URL: https://issues.apache.org/jira/browse/FLINK-4167
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.1.0
>
>
> The {{TaskMetricGroup}} does not close the {{ioMetrics}} metric group. This 
> causes that metrics registered under the {{ioMetrics}} are not deregistered 
> after the termination of a job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2210: [FLINK-4167] [metrics] Close IOMetricGroup in Task...

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/2210#discussion_r70597751
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/metrics/groups/ProxyMetricGroup.java 
---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.metrics.groups;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.Gauge;
+import org.apache.flink.metrics.Histogram;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.util.Preconditions;
+
+/**
+ * Metric group which forwards all registration calls to its parent metric 
group.
+ *
+ * @param  Type of the parent metric group
+ */
+@Internal
+public class ProxyMetricGroup implements 
MetricGroup {
+   private final P parentMetricGroup;
+
+   public ProxyMetricGroup(P parentMetricGroup) {
+   this.parentMetricGroup = 
Preconditions.checkNotNull(parentMetricGroup);
+   }
+
+   @Override
+   public final void close() {
+   parentMetricGroup.close();
--- End diff --

It is safer to not do anything here. `close()` should only be called by the 
Task on the `TaskMetricGroup`, this would open us to the possibility of 
components closing the TaskMG as well.

There's also the looming StackOverflow when someone puts 
`ioMetrics.close()` into the `TaskMetricGroup#close()`.

Now that i think about it i believe `close()` (and by extension, 
`isClosed()`) has no business being in the MetricGroup interface in the first 
place, as users actually don't need to call it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4184) Ganglia and GraphiteReporter report metric names with invalid characters

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374763#comment-15374763
 ] 

ASF GitHub Bot commented on FLINK-4184:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2220
  
True, it conflicts with your proposed changes for the definable metric 
group delimiter. I will rebase and adapt this PR wrt #2219.


> Ganglia and GraphiteReporter report metric names with invalid characters
> 
>
> Key: FLINK-4184
> URL: https://issues.apache.org/jira/browse/FLINK-4184
> Project: Flink
>  Issue Type: Bug
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.1.0
>
>
> Flink's {{GangliaReporter}} and {{GraphiteReporter}} report metrics with 
> names which contain invalid characters. For example, quotes are not filtered 
> out which can be problematic for Ganglia. Moreover, dots are not replaced 
> which causes Graphite to think that an IP address is actually a scoped metric 
> name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2220: [FLINK-4184] [metrics] Replace invalid characters in Sche...

[tillrohrmann]

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2220
  
True, it conflicts with your proposed changes for the definable metric 
group delimiter. I will rebase and adapt this PR wrt #2219.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4143) Configurable delimiter for metric identifier

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374733#comment-15374733
 ] 

ASF GitHub Bot commented on FLINK-4143:
---

Github user zentol closed the pull request at:

https://github.com/apache/flink/pull/2219


> Configurable delimiter for metric identifier
> 
>
> Key: FLINK-4143
> URL: https://issues.apache.org/jira/browse/FLINK-4143
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Minor
>
> The metric identifier is currently hard-coded to separate components with a 
> dot.
> We should make this configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-4143) Configurable delimiter for metric identifier

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-4143.
---
Resolution: Fixed

Implemented in 790a654c5e08e0e54f3e02499be4dd8c4006227a

> Configurable delimiter for metric identifier
> 
>
> Key: FLINK-4143
> URL: https://issues.apache.org/jira/browse/FLINK-4143
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Minor
>
> The metric identifier is currently hard-coded to separate components with a 
> dot.
> We should make this configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2219: [FLINK-4143][metrics] Configurable delimiter

[zentol]

Github user zentol closed the pull request at:

https://github.com/apache/flink/pull/2219


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4143) Configurable delimiter for metric identifier

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374725#comment-15374725
 ] 

ASF GitHub Bot commented on FLINK-4143:
---

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2219
  
merging


> Configurable delimiter for metric identifier
> 
>
> Key: FLINK-4143
> URL: https://issues.apache.org/jira/browse/FLINK-4143
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Minor
>
> The metric identifier is currently hard-coded to separate components with a 
> dot.
> We should make this configurable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #2219: [FLINK-4143][metrics] Configurable delimiter

[zentol]

Github user zentol commented on the issue:

https://github.com/apache/flink/pull/2219
  
merging


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4159) Quickstart poms exclude unused dependencies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374721#comment-15374721
 ] 

ASF GitHub Bot commented on FLINK-4159:
---

Github user zentol closed the pull request at:

https://github.com/apache/flink/pull/2217


> Quickstart poms exclude unused dependencies
> ---
>
> Key: FLINK-4159
> URL: https://issues.apache.org/jira/browse/FLINK-4159
> Project: Flink
>  Issue Type: Bug
>  Components: Quickstarts
>Affects Versions: 1.1.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Minor
> Fix For: 1.1.0
>
>
> The Quickstart poms exclude several dependencies from being packaged into the 
> fat-jar, even though they aren't used by Flink according to `mvn 
> dependency:tree`.
> com.amazonaws:aws-java-sdk
> com.twitter:chill-avro_*
> com.twitter:chill-bijection_*
> com.twitter:bijection-core_*
> com.twitter:bijection-avro_*
> de.javakaffee:kryo-serializers



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)