date:20160221

[jira] [Commented] (FLINK-1502) Expose metrics to graphite, ganglia and JMX.

2016-02-21 Thread Dongwon Kim (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156330#comment-15156330
 ] 

Dongwon Kim commented on FLINK-1502:


[~jgrier] Thank you for sharing the link :-) The page said that ganglia already 
stores counters when the slope type of a metric is set to Slope.POSITIVE, which 
I wasn't aware of. 
I found the following explanation about Slope.POSITIVE:
"Using the value positive for the slope of a new metric will cause the 
corresponding RRD file to be generated as a COUNTER, with delta values being 
displayed instead of the actual metric values."
You're right. We actually didn't need to do that.

I also don't have any idea regarding the query language.
If Ganglia supports it, we could store metrics as GAUGE and just change a way 
to draw graphs.

> Expose metrics to graphite, ganglia and JMX.
> 
>
> Key: FLINK-1502
> URL: https://issues.apache.org/jira/browse/FLINK-1502
> Project: Flink
>  Issue Type: Sub-task
>  Components: JobManager, TaskManager
>Affects Versions: 0.9
>Reporter: Robert Metzger
>Assignee: Dongwon Kim
>Priority: Minor
> Fix For: pre-apache
>
>
> The metrics library allows to expose collected metrics easily to other 
> systems such as graphite, ganglia or Java's JVM (VisualVM).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [WIP][FLINK-3422][streaming][api-breaking] Scr...

2016-02-21 Thread StephanEwen

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1685#issuecomment-186948795
  
It is pretty crucial that different hash functions are used for the 
partitioning across machines, and the internal partitioning of data structures. 
If the same hash function is used for both, many internal data structure 
partitions will be empty.

So far we divided it the following way (admittedly not documented)
  - murmur hash across machines
  - Jenkins hash internally in data structures

How about we stick with that division and use Murmur Hash in the streaming 
partitioner as well?




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3422) Scramble HashPartitioner hashes

2016-02-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156306#comment-15156306
 ] 

ASF GitHub Bot commented on FLINK-3422:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1685#issuecomment-186948795
  
It is pretty crucial that different hash functions are used for the 
partitioning across machines, and the internal partitioning of data structures. 
If the same hash function is used for both, many internal data structure 
partitions will be empty.

So far we divided it the following way (admittedly not documented)
  - murmur hash across machines
  - Jenkins hash internally in data structures

How about we stick with that division and use Murmur Hash in the streaming 
partitioner as well?




> Scramble HashPartitioner hashes
> ---
>
> Key: FLINK-3422
> URL: https://issues.apache.org/jira/browse/FLINK-3422
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: 0.10.2
>Reporter: Stephan Ewen
>Assignee: Gabor Horvath
>Priority: Critical
> Fix For: 1.0.0
>
>
> The {{HashPartitioner}} used by the streaming API does not apply any hash 
> scrambling against bad user hash functions.
> We should apply a murmor or jenkins hash on top of the hash code, similar as 
> in the {{DataSet}} API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3422) Scramble HashPartitioner hashes

2016-02-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156257#comment-15156257
 ] 

ASF GitHub Bot commented on FLINK-3422:
---

GitHub user Xazax-hun opened a pull request:

https://github.com/apache/flink/pull/1685

[WIP][FLINK-3422][streaming][api-breaking] Scramble HashPartitioner hashes.

This pull request contains a fix for FLINK-3422. Some of the tests are 
failing at the moment, because they utilized prior knowledge about the user 
hash function. Fixing those tests require knowledge about the internals of 
Flink that I do not possess yet, so Marton Balassi helps me.

The Jira ticket mentions both Murmur and Jenkins hash.
Murmur hash is already used in the batch implementation: 
https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/operators/shipping/OutputEmitter.java#L187

My approach was to move Jenkins hash from CompactingHashTable to MathUtils 
and use that in HashPartitioner. In case you think it is better to use murmur 
hash here, or it has some value to be consistent in this regard with the batch 
implementation, please let me know. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Xazax-hun/flink HashPartitioner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1685.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1685


commit afaa069483423e0bbb448f773cdcb4992689745e
Author: Gabor Horvath 
Date:   2016-02-21T13:54:44Z

[FLINK-3422][streaming][api-breaking] Scramble HashPartitioner hashes.

commit 102053618e11e0de784d4d02152dc439a1e274ca
Author: Márton Balassi 
Date:   2016-02-21T22:01:00Z

[WIP][FLINK-3422][streaming][api-breaking] Update tests reliant on hashing




> Scramble HashPartitioner hashes
> ---
>
> Key: FLINK-3422
> URL: https://issues.apache.org/jira/browse/FLINK-3422
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: 0.10.2
>Reporter: Stephan Ewen
>Assignee: Gabor Horvath
>Priority: Critical
> Fix For: 1.0.0
>
>
> The {{HashPartitioner}} used by the streaming API does not apply any hash 
> scrambling against bad user hash functions.
> We should apply a murmor or jenkins hash on top of the hash code, similar as 
> in the {{DataSet}} API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [WIP][FLINK-3422][streaming][api-breaking] Scr...

2016-02-21 Thread Xazax-hun

GitHub user Xazax-hun opened a pull request:

https://github.com/apache/flink/pull/1685

[WIP][FLINK-3422][streaming][api-breaking] Scramble HashPartitioner hashes.

This pull request contains a fix for FLINK-3422. Some of the tests are 
failing at the moment, because they utilized prior knowledge about the user 
hash function. Fixing those tests require knowledge about the internals of 
Flink that I do not possess yet, so Marton Balassi helps me.

The Jira ticket mentions both Murmur and Jenkins hash.
Murmur hash is already used in the batch implementation: 
https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/operators/shipping/OutputEmitter.java#L187

My approach was to move Jenkins hash from CompactingHashTable to MathUtils 
and use that in HashPartitioner. In case you think it is better to use murmur 
hash here, or it has some value to be consistent in this regard with the batch 
implementation, please let me know. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Xazax-hun/flink HashPartitioner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1685.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1685


commit afaa069483423e0bbb448f773cdcb4992689745e
Author: Gabor Horvath 
Date:   2016-02-21T13:54:44Z

[FLINK-3422][streaming][api-breaking] Scramble HashPartitioner hashes.

commit 102053618e11e0de784d4d02152dc439a1e274ca
Author: MÃ¡rton Balassi 
Date:   2016-02-21T22:01:00Z

[WIP][FLINK-3422][streaming][api-breaking] Update tests reliant on hashing




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3416) [py] .bat files fail when path contains spaces

2016-02-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156221#comment-15156221
 ] 

ASF GitHub Bot commented on FLINK-3416:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1674#discussion_r53573047
  
--- Diff: flink-dist/src/main/flink-bin/bin/pyflink2.bat ---
@@ -22,4 +22,4 @@ setlocal EnableDelayedExpansion
 SET bin=%~dp0
 SET FLINK_ROOT_DIR=%bin%..
 
-%FLINK_ROOT_DIR%\bin\flink run -v %FLINK_ROOT_DIR%\lib\flink-python*.jar 2 
%*
\ No newline at end of file
+"%FLINK_ROOT_DIR%\bin\flink" run -v 
"%FLINK_ROOT_DIR%"\lib\flink-python*.jar 2 %*
--- End diff --

no, the jar name will never contain spaces


> [py] .bat files fail when path contains spaces
> --
>
> Key: FLINK-3416
> URL: https://issues.apache.org/jira/browse/FLINK-3416
> Project: Flink
>  Issue Type: Bug
>  Components: Python API
>Affects Versions: 1.0.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Minor
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3416] [py] Support for spaces in flink ...

2016-02-21 Thread zentol

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1674#discussion_r53573047
  
--- Diff: flink-dist/src/main/flink-bin/bin/pyflink2.bat ---
@@ -22,4 +22,4 @@ setlocal EnableDelayedExpansion
 SET bin=%~dp0
 SET FLINK_ROOT_DIR=%bin%..
 
-%FLINK_ROOT_DIR%\bin\flink run -v %FLINK_ROOT_DIR%\lib\flink-python*.jar 2 
%*
\ No newline at end of file
+"%FLINK_ROOT_DIR%\bin\flink" run -v 
"%FLINK_ROOT_DIR%"\lib\flink-python*.jar 2 %*
--- End diff --

no, the jar name will never contain spaces


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3416) [py] .bat files fail when path contains spaces

2016-02-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156179#comment-15156179
 ] 

ASF GitHub Bot commented on FLINK-3416:
---

Github user greghogan commented on a diff in the pull request:

https://github.com/apache/flink/pull/1674#discussion_r53571627
  
--- Diff: flink-dist/src/main/flink-bin/bin/pyflink2.bat ---
@@ -22,4 +22,4 @@ setlocal EnableDelayedExpansion
 SET bin=%~dp0
 SET FLINK_ROOT_DIR=%bin%..
 
-%FLINK_ROOT_DIR%\bin\flink run -v %FLINK_ROOT_DIR%\lib\flink-python*.jar 2 
%*
\ No newline at end of file
+"%FLINK_ROOT_DIR%\bin\flink" run -v 
"%FLINK_ROOT_DIR%"\lib\flink-python*.jar 2 %*
--- End diff --

Does the wildcard argument also need double quotes?


> [py] .bat files fail when path contains spaces
> --
>
> Key: FLINK-3416
> URL: https://issues.apache.org/jira/browse/FLINK-3416
> Project: Flink
>  Issue Type: Bug
>  Components: Python API
>Affects Versions: 1.0.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Minor
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3416] [py] Support for spaces in flink ...

2016-02-21 Thread greghogan

Github user greghogan commented on a diff in the pull request:

https://github.com/apache/flink/pull/1674#discussion_r53571627
  
--- Diff: flink-dist/src/main/flink-bin/bin/pyflink2.bat ---
@@ -22,4 +22,4 @@ setlocal EnableDelayedExpansion
 SET bin=%~dp0
 SET FLINK_ROOT_DIR=%bin%..
 
-%FLINK_ROOT_DIR%\bin\flink run -v %FLINK_ROOT_DIR%\lib\flink-python*.jar 2 
%*
\ No newline at end of file
+"%FLINK_ROOT_DIR%\bin\flink" run -v 
"%FLINK_ROOT_DIR%"\lib\flink-python*.jar 2 %*
--- End diff --

Does the wildcard argument also need double quotes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3387) Replace all ByteArrayOutputStream and DataOutputStream use in State Backends by DataOutputSerializer

2016-02-21 Thread Subhobrata Dey (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-3387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156147#comment-15156147
 ] 

Subhobrata Dey commented on FLINK-3387:
---

Hello [~StephanEwen], 

I see that each class is creating an instance of `DataInputDeserializer`. So, 
from what I understand, you want a static method like `getInstance` in class 
`DataInputDeserializer` which will be be used by all the classes for calling 
`deserializer.setBuffer(bytes, 0, bytes.length)`. Is this understanding 
correct? 

> Replace all ByteArrayOutputStream and DataOutputStream use in State Backends 
> by DataOutputSerializer
> 
>
> Key: FLINK-3387
> URL: https://issues.apache.org/jira/browse/FLINK-3387
> Project: Flink
>  Issue Type: Bug
>  Components: state backends
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Subhobrata Dey
>
> Currently, the {{byte[]}} to object conversion and vice versa is always done 
> by creating new byte array streams and Data Input/Output streams. That is 
> pretty wasteful, we can optimize this with Flink's own classes:
>   - {{DataInputSerializer}}
>   - {{DataOutputSerializer}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (FLINK-3440) Kafka should also checkpoint partitions where no initial offset was retrieved

2016-02-21 Thread Robert Metzger (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger reassigned FLINK-3440:
-

Assignee: Robert Metzger

> Kafka should also checkpoint partitions where no initial offset was retrieved
> -
>
> Key: FLINK-3440
> URL: https://issues.apache.org/jira/browse/FLINK-3440
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector
>Affects Versions: 0.10.2
>Reporter: Stephan Ewen
>Assignee: Robert Metzger
>Priority: Blocker
> Fix For: 1.0.0
>
>
> For the discussion, see here: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-partition-alignment-for-event-time-td4782.html#a4998



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3455) Bump Kafka 0.9 connector dependency to Kafka 0.9.0.1

2016-02-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156049#comment-15156049
 ] 

ASF GitHub Bot commented on FLINK-3455:
---

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/1684

[FLINK-3455] Bump Kafka to 0.9.0.1 and 0.8.2.2



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink flink3455

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1684.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1684


commit 43f987a201d40f1d9c11ba9eb46d2b98d60584f5
Author: Robert Metzger 
Date:   2016-02-19T19:59:58Z

[FLINK-3455] Bump Kafka to 0.9.0.1 and 0.8.2.2




> Bump Kafka 0.9 connector dependency to Kafka 0.9.0.1
> 
>
> Key: FLINK-3455
> URL: https://issues.apache.org/jira/browse/FLINK-3455
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>
> The Kafka project released the 0.9.0.1 version.
> I saw some issues (in our integration tests) while developing the code. My 
> hope is that the upgraded version will improve stability.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3455] Bump Kafka to 0.9.0.1 and 0.8.2.2

2016-02-21 Thread rmetzger

GitHub user rmetzger opened a pull request:

https://github.com/apache/flink/pull/1684

[FLINK-3455] Bump Kafka to 0.9.0.1 and 0.8.2.2



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rmetzger/flink flink3455

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1684.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1684


commit 43f987a201d40f1d9c11ba9eb46d2b98d60584f5
Author: Robert Metzger 
Date:   2016-02-19T19:59:58Z

[FLINK-3455] Bump Kafka to 0.9.0.1 and 0.8.2.2




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: FLINK-2213 Makes the number of vcores per YARN...

2016-02-21 Thread rmetzger

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1588#issuecomment-186835009
  
The expected output of the test is the following

```
Test testQueryCluster(org.apache.flink.yarn.YARNSessionFIFOITCase) is 
running.


20:12:25,692 INFO  org.apache.hadoop.yarn.client.RMProxy
 - Connecting to ResourceManager at 
testing-worker-linux-docker-8281db7b-3371-linux-16/172.17.6.245:8032
20:12:25,701 INFO  org.apache.hadoop.yarn.webapp.WebApps
 - Registered webapp guice modules
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl  - Registering 
with RM using finished containers :[]
20:12:25,712 INFO  org.apache.hadoop.yarn.util.RackResolver 
 - Resolved 
testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org to 
/default-rack
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService  - 
NodeManager from node 
testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org(cmPort: 
59877 httpPort: 39611) registered with capability: , 
assigned nodeId 
testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:59877
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl  - 
testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:59877 
Node Transitioned from NEW to RUNNING
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager
  - Rolling master-key for container-tokens, got key with id 1069868518
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM  - 
Rolling master-key for nm-tokens, got key with id :-1168902475
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl  - Registered 
with ResourceManager as 
testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:59877 
with total resource of 
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl  - Notifying 
ContainerManager to unblock new container-requests
20:12:25,939 INFO  org.apache.flink.yarn.YARNSessionFIFOITCase  
 - Starting testQueryCluster()
20:12:25,939 INFO  org.apache.flink.yarn.YarnTestBase   
 - Running with args [-q]
20:12:25,993 INFO  org.apache.hadoop.yarn.client.RMProxy
 - Connecting to ResourceManager at /0.0.0.0:8032
20:12:26,940 INFO  org.apache.flink.yarn.YarnTestBase   
 - Found expected output in redirected streams
20:12:26,940 INFO  org.apache.flink.yarn.YarnTestBase   
 - RunWithArgs: request runner to stop
20:12:26,940 WARN  org.apache.flink.yarn.YarnTestBase   
 - RunWithArgs runner stopped.
20:12:26,940 INFO  org.apache.flink.yarn.YarnTestBase   
 - Sending stdout content through logger: 

NodeManagers in the Cluster 2|Property |Value  
+---+
|NodeID   
|testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:59877 
|Memory   |4096 MB  
|vCores   |666  
|HealthReport | 
|Containers   |0
+---+
|NodeID   
|testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:44161 
|Memory   |4096 MB  
|vCores   |666  
|HealthReport | 
|Containers   |0
+---+
Summary: totalMemory 8192 totalCores 1332




20:12:26,940 INFO  org.apache.flink.yarn.YarnTestBase   
 - Sending stderr content through logger: 




20:12:26,940 INFO  org.apache.flink.yarn.YarnTestBase   
 - Test was successful
20:12:26,940 INFO  org.apache.flink.yarn.YARNSessionFIFOITCase  
 - Finished testQueryCluster()
20:12:27,443 INFO  org.apache.flink.yarn.YARNSessionFIFOITCase  
 - 
```

but when its failing, its outputting

```
Test testQueryCluster(org.apache.flink.yarn.YARNSessionFIFOITCase) is 
running.


21:50:20,684 INFO  org.apache.hadoop.yarn.client.RMProxy
 - Connecting to ResourceManager at 
testing-worker-linux-docker-1305e497-3358-linux-10/172.17.4.146:8032
21:50:22,041 INFO  org.apache.flink.yarn.YARNSessionFIFOITCase  
 - Starting

[jira] [Commented] (FLINK-2213) Configure number of vcores

2016-02-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156048#comment-15156048
 ] 

ASF GitHub Bot commented on FLINK-2213:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1588#issuecomment-186835009
  
The expected output of the test is the following

```
Test testQueryCluster(org.apache.flink.yarn.YARNSessionFIFOITCase) is 
running.


20:12:25,692 INFO  org.apache.hadoop.yarn.client.RMProxy
 - Connecting to ResourceManager at 
testing-worker-linux-docker-8281db7b-3371-linux-16/172.17.6.245:8032
20:12:25,701 INFO  org.apache.hadoop.yarn.webapp.WebApps
 - Registered webapp guice modules
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl  - Registering 
with RM using finished containers :[]
20:12:25,712 INFO  org.apache.hadoop.yarn.util.RackResolver 
 - Resolved 
testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org to 
/default-rack
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService  - 
NodeManager from node 
testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org(cmPort: 
59877 httpPort: 39611) registered with capability: , 
assigned nodeId 
testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:59877
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl  - 
testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:59877 
Node Transitioned from NEW to RUNNING
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager
  - Rolling master-key for container-tokens, got key with id 1069868518
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM  - 
Rolling master-key for nm-tokens, got key with id :-1168902475
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl  - Registered 
with ResourceManager as 
testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:59877 
with total resource of 
20:12:25,712 INFO  
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl  - Notifying 
ContainerManager to unblock new container-requests
20:12:25,939 INFO  org.apache.flink.yarn.YARNSessionFIFOITCase  
 - Starting testQueryCluster()
20:12:25,939 INFO  org.apache.flink.yarn.YarnTestBase   
 - Running with args [-q]
20:12:25,993 INFO  org.apache.hadoop.yarn.client.RMProxy
 - Connecting to ResourceManager at /0.0.0.0:8032
20:12:26,940 INFO  org.apache.flink.yarn.YarnTestBase   
 - Found expected output in redirected streams
20:12:26,940 INFO  org.apache.flink.yarn.YarnTestBase   
 - RunWithArgs: request runner to stop
20:12:26,940 WARN  org.apache.flink.yarn.YarnTestBase   
 - RunWithArgs runner stopped.
20:12:26,940 INFO  org.apache.flink.yarn.YarnTestBase   
 - Sending stdout content through logger: 

NodeManagers in the Cluster 2|Property |Value  
+---+
|NodeID   
|testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:59877 
|Memory   |4096 MB  
|vCores   |666  
|HealthReport | 
|Containers   |0
+---+
|NodeID   
|testing-worker-linux-docker-8281db7b-3371-linux-16.prod.travis-ci.org:44161 
|Memory   |4096 MB  
|vCores   |666  
|HealthReport | 
|Containers   |0
+---+
Summary: totalMemory 8192 totalCores 1332




20:12:26,940 INFO  org.apache.flink.yarn.YarnTestBase   
 - Sending stderr content through logger: 




20:12:26,940 INFO  org.apache.flink.yarn.YarnTestBase   
 - Test was successful
20:12:26,940 INFO  org.apache.flink.yarn.YARNSessionFIFOITCase  
 - Finished testQueryCluster()
20:12:27,443 INFO  org.apache.flink.yarn.YARNSessionFIFOITCase  
 - 
```

but when its failing, its outputting

```
Test testQueryCluster(org.apache.flink.yarn.YARNSessionFIFOITCase) is 
running.


21:50:20,684 INFO  org.apache.hadoop.yarn.client.RMProxy

[jira] [Commented] (FLINK-3385) Fix outer join skipping unprobed partitions

2016-02-21 Thread Greg Hogan (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156041#comment-15156041
 ] 

Greg Hogan commented on FLINK-3385:
---

Pull request is [here|https://github.com/apache/flink/pull/1680]. Had to rename 
the pull request, looks like if there are multiple commits the default name is 
the branch name rather than the commit log.

> Fix outer join skipping unprobed partitions
> ---
>
> Key: FLINK-3385
> URL: https://issues.apache.org/jira/browse/FLINK-3385
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Blocker
> Fix For: 1.0.0
>
>
> {{MutableHashTable.nextRecord}} performs three steps for a build-side outer 
> join:
> {code}
>   public boolean nextRecord() throws IOException {
>   if (buildSideOuterJoin) {
>   return processProbeIter() || 
> processUnmatchedBuildIter() || prepareNextPartition();
>   } else {
>   return processProbeIter() || prepareNextPartition();
>   }
>   }
> {code}
> {{MutableHashTable.processUnmatchedBuildIter}} eventually calls through to 
> {{MutableHashTable.moveToNextBucket}} which is unable to process spilled 
> partitions:
> {code}
>   if (p.isInMemory()) {
>   ...
>   } else {
>   return false;
>   }
> {code}
> {{MutableHashTable.prepareNextPartition}} calls 
> {{HashPartition.finalizeProbePhase}} which only spills the partition (to be 
> read and processed in the next instantiation of {{MutableHashTable}}) if 
> probe-side records were spilled. In an equi-join this is fine but with an 
> outer join the unmatched build-side records must still be retained (though no 
> further probing is necessary, so could this be short-circuited when loaded by 
> the next {{MutableHashTable}}?).
> {code}
>   if (isInMemory()) {
>   ...
>   }
>   else if (this.probeSideRecordCounter == 0) {
>   // partition is empty, no spilled buffers
>   // return the memory buffer
>   
> freeMemory.add(this.probeSideBuffer.getCurrentSegment());
>   // delete the spill files
>   this.probeSideChannel.close();
>   this.buildSideChannel.deleteChannel();
>   this.probeSideChannel.deleteChannel();
>   return 0;
>   }
>   else {
>   // flush the last probe side buffer and register this 
> partition as pending
>   this.probeSideBuffer.close();
>   this.probeSideChannel.close();
>   spilledPartitions.add(this);
>   return 1;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: FLINK-2213 Makes the number of vcores per YARN...

2016-02-21 Thread kl0u

Github user kl0u commented on the pull request:

https://github.com/apache/flink/pull/1588#issuecomment-186832973
  
@rmetzger I am trying to reproduce it but so far I cannot. I will keep you 
posted.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2213) Configure number of vcores

2016-02-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156039#comment-15156039
 ] 

ASF GitHub Bot commented on FLINK-2213:
---

Github user kl0u commented on the pull request:

https://github.com/apache/flink/pull/1588#issuecomment-186832973
  
@rmetzger I am trying to reproduce it but so far I cannot. I will keep you 
posted.


> Configure number of vcores
> --
>
> Key: FLINK-2213
> URL: https://issues.apache.org/jira/browse/FLINK-2213
> Project: Flink
>  Issue Type: Improvement
>  Components: YARN Client
>Affects Versions: 0.10.0
>Reporter: Ufuk Celebi
>Assignee: Klou
> Fix For: 1.0.0
>
>
> Currently, the number of vcores per YARN container is set to 1.
> It is desirable to allow configuring this value. As a simple heuristic it 
> makes sense to at least set it to the number of slots per container.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3443) JobManager cancel and clear everything fails jobs instead of cancelling

2016-02-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156021#comment-15156021
 ] 

ASF GitHub Bot commented on FLINK-3443:
---

Github user uce commented on a diff in the pull request:

https://github.com/apache/flink/pull/1669#discussion_r5359
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -1487,7 +1487,7 @@ class JobManager(
   }
 }
 
-eg.fail(cause)
+eg.cancel()
--- End diff --

Yes, that would work during shutdown, but there will be a chance that a 
`fail` right before `cancelAndClearEverything` will still result in the 
restarting behaviour, because multiple calls to `fail` are ignored when the job 
status is `FAILING`. `cancel` makes sure that this does not happen, because 
cancellation "overwrites" failing behaviour.

If we say that this is OK as a corner case, we can keep the `fail` on 
`cancelAndClearEverything` and wrap the Exception to suppress restarts in the 
common case.


> JobManager cancel and clear everything fails jobs instead of cancelling
> ---
>
> Key: FLINK-3443
> URL: https://issues.apache.org/jira/browse/FLINK-3443
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>
> When the job manager is shut down, it calls {{cancelAndClearEverything}}. 
> This method does not {{cancel}} the {{ExecutionGraph}} instances, but 
> {{fail}}s them, which can lead to {{ExecutionGraph}} restart.
> I've noticed this in tests, where old graph got into a loop of restarts.
> What I don't understand is why the futures etc. are not cancelled when the 
> executor service is shut down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3443] [runtime] Prevent cancelled jobs ...

2016-02-21 Thread uce

Github user uce commented on a diff in the pull request:

https://github.com/apache/flink/pull/1669#discussion_r5359
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -1487,7 +1487,7 @@ class JobManager(
   }
 }
 
-eg.fail(cause)
+eg.cancel()
--- End diff --

Yes, that would work during shutdown, but there will be a chance that a 
`fail` right before `cancelAndClearEverything` will still result in the 
restarting behaviour, because multiple calls to `fail` are ignored when the job 
status is `FAILING`. `cancel` makes sure that this does not happen, because 
cancellation "overwrites" failing behaviour.

If we say that this is OK as a corner case, we can keep the `fail` on 
`cancelAndClearEverything` and wrap the Exception to suppress restarts in the 
common case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3460) Make flink-streaming-connectors' flink dependencies provided

2016-02-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156020#comment-15156020
 ] 

ASF GitHub Bot commented on FLINK-3460:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/1683

[FLINK-3460] [build] Set Flink dependencies in flink-streaming-connectors 
to provided

The flink-streaming-connectors all depend on flink-streaming-java in 
compile scope.
This entails that this dependency is always pulled in, when you build a fat 
jar. By
setting this dependency to provided, this will be avoided. Furthermore, the 
connectors
are always used in conjunction with flink-streaming-java. This means that 
you will
always have an explicit dependency import of flink-streaming-java in your 
build script.
This allows you to run your program locally but it is also easy to exclude 
the dependency
from being included in the fat jar by setting it to provided, as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink providedDeps

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1683.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1683


commit fdfc046d5d349afc395cd09a682365b6746aedb3
Author: Till Rohrmann 
Date:   2016-02-18T10:23:05Z

[FLINK-3460] [build] Set Flink dependencies in flink-streaming-connectors 
to provided

The flink-streaming-connectors all depend on flink-streaming-java in 
compile scope.
This entails that this dependency is always pulled in, when you build a fat 
jar. By
setting this dependency to provided, this will be avoided. Furthermore, the 
connectors
are always used in conjunction with flink-streaming-java. This means that 
you will
always have an explicit dependency import of flink-streaming-java in your 
build script.
This allows you to run your program locally but it is also easy to exclude 
the dependency
from being included in the fat jar by setting it to provided, as well.




> Make flink-streaming-connectors' flink dependencies provided
> 
>
> Key: FLINK-3460
> URL: https://issues.apache.org/jira/browse/FLINK-3460
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.0.0
>
>
> All of the flink-streaming-connectors depend on flink-streaming-java (compile 
> scope). This entails that whenever you have a project which depends on the a 
> connector this will also pull in the heavy-weight flink-streaming-java 
> dependency. If you now build a fat-jar, you have to use exclusion rules to 
> filter them out again.
> It would be more natural to simply declare the dependencies as provided which 
> will automatically avoid the inclusion in the fat jar with respect to the 
> connector dependency. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3460] [build] Set Flink dependencies in...

2016-02-21 Thread tillrohrmann

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/1683

[FLINK-3460] [build] Set Flink dependencies in flink-streaming-connectors 
to provided

The flink-streaming-connectors all depend on flink-streaming-java in 
compile scope.
This entails that this dependency is always pulled in, when you build a fat 
jar. By
setting this dependency to provided, this will be avoided. Furthermore, the 
connectors
are always used in conjunction with flink-streaming-java. This means that 
you will
always have an explicit dependency import of flink-streaming-java in your 
build script.
This allows you to run your program locally but it is also easy to exclude 
the dependency
from being included in the fat jar by setting it to provided, as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink providedDeps

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1683.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1683


commit fdfc046d5d349afc395cd09a682365b6746aedb3
Author: Till Rohrmann 
Date:   2016-02-18T10:23:05Z

[FLINK-3460] [build] Set Flink dependencies in flink-streaming-connectors 
to provided

The flink-streaming-connectors all depend on flink-streaming-java in 
compile scope.
This entails that this dependency is always pulled in, when you build a fat 
jar. By
setting this dependency to provided, this will be avoided. Furthermore, the 
connectors
are always used in conjunction with flink-streaming-java. This means that 
you will
always have an explicit dependency import of flink-streaming-java in your 
build script.
This allows you to run your program locally but it is also easy to exclude 
the dependency
from being included in the fat jar by setting it to provided, as well.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3459) Make build SBT compatible

2016-02-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156019#comment-15156019
 ] 

ASF GitHub Bot commented on FLINK-3459:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/1682

[FLINK-3459] [build] Fix conflicting dependencies commons-collections 
commons-beanutils and commons-beanutils-core

The Hadoop dependencies have a dependency on commons-configuration which 
pulls in transitively the commons-collection, commons-beanutils and 
common-beanutils-core depedencies. Commons-beanutils and commons-collection 
contain classes which live in the same namespace. They are also binary 
compatible but not binary identical. This is a problem for the sbt assembly 
plugin which checks for binary identity. In order to solve the problem, we bump 
the commons-configuration version to 1.7 so that only commons-beanutils is 
pulled in. This is necessary, because the transitive promotion of dependencies 
of the shade plugin only excludes the commons-beanutils dependency only from 
the directly depending dependency. All parent dependencies won't have the 
exclusion. This is a problem for SBT which will pull the dependency as part of 
one of the parents, then. Moreover, we replace commons-beanutils by 
commons-beanutils-bean-collections which contains only the non-conflicting 
classes wrt commons-collections.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink makeSbtCompatible

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1682.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1682


commit 872a5cc8b40142feb6313549e1ed2b2029ce4d68
Author: Till Rohrmann 
Date:   2016-02-17T14:41:43Z

[FLINK-3459] [build] Fix conflicting dependencies commons-collections, 
commons-beanutils and commons-beanutils-core

The Hadoop dependencies have a dependency on commons-configuration which 
pulls in transitively the commons-collection, commons-beanutils and 
common-beanutils-core depedencies. Commons-beanutils and commons-collection 
contain classes which live in the same namespace. They are also binary 
compatible but not binary identical. This is a problem for the sbt assembly 
plugin which checks for binary identity. In order to solve the problem, we bump 
the commons-configuration version to 1.7 so that only commons-beanutils is 
pulled in. This is necessary, because the transitive promotion of dependencies 
of the shade plugin only excludes the commons-beanutils dependency only from 
the directly depending dependency. All parent dependencies won't have the 
exclusion. This is a problem for SBT which will pull the dependency as part of 
one of the parents, then. Moreover, we replace commons-beanutils by 
commons-beanutils-bean-collections which contains only the non-conflicting 
classes wrt commons-collections.




> Make build SBT compatible
> -
>
> Key: FLINK-3459
> URL: https://issues.apache.org/jira/browse/FLINK-3459
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.0.0
>
>
> The current flink-shaded-hadoop2 dependency is not SBT compatible because it 
> contains {{commons-collection:3.2.1}}, {{commons-beanutils:1.7}} and 
> {{commons-beanutils-core:1.8}} dependencies. These dependencies contain 
> overlapping classes which are binary compatible but not binary identical. 
> This is a problem for the SBT assembly plugin which requires binary identity 
> for merging them. Otherwise, the build is aborted. I propose to exclude the 
> conflicting dependencies and use instead the 
> {{commons-beanutils-bean-collection}} class which contains only then 
> non-conflicting classes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3460) Make flink-streaming-connectors' flink dependencies provided

2016-02-21 Thread Till Rohrmann (JIRA)

Till Rohrmann created FLINK-3460:


 Summary: Make flink-streaming-connectors' flink dependencies 
provided
 Key: FLINK-3460
 URL: https://issues.apache.org/jira/browse/FLINK-3460
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.0.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Priority: Minor
 Fix For: 1.0.0


All of the flink-streaming-connectors depend on flink-streaming-java (compile 
scope). This entails that whenever you have a project which depends on the a 
connector this will also pull in the heavy-weight flink-streaming-java 
dependency. If you now build a fat-jar, you have to use exclusion rules to 
filter them out again.

It would be more natural to simply declare the dependencies as provided which 
will automatically avoid the inclusion in the fat jar with respect to the 
connector dependency. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3459] [build] Fix conflicting dependenc...

2016-02-21 Thread tillrohrmann

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/1682

[FLINK-3459] [build] Fix conflicting dependencies commons-collections 
commons-beanutils and commons-beanutils-core

The Hadoop dependencies have a dependency on commons-configuration which 
pulls in transitively the commons-collection, commons-beanutils and 
common-beanutils-core depedencies. Commons-beanutils and commons-collection 
contain classes which live in the same namespace. They are also binary 
compatible but not binary identical. This is a problem for the sbt assembly 
plugin which checks for binary identity. In order to solve the problem, we bump 
the commons-configuration version to 1.7 so that only commons-beanutils is 
pulled in. This is necessary, because the transitive promotion of dependencies 
of the shade plugin only excludes the commons-beanutils dependency only from 
the directly depending dependency. All parent dependencies won't have the 
exclusion. This is a problem for SBT which will pull the dependency as part of 
one of the parents, then. Moreover, we replace commons-beanutils by 
commons-beanutils-bean-collections which contains only the non-conflicting 
classes wrt commons-col
 lections.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink makeSbtCompatible

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1682.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1682


commit 872a5cc8b40142feb6313549e1ed2b2029ce4d68
Author: Till Rohrmann 
Date:   2016-02-17T14:41:43Z

[FLINK-3459] [build] Fix conflicting dependencies commons-collections, 
commons-beanutils and commons-beanutils-core

The Hadoop dependencies have a dependency on commons-configuration which 
pulls in transitively the commons-collection, commons-beanutils and 
common-beanutils-core depedencies. Commons-beanutils and commons-collection 
contain classes which live in the same namespace. They are also binary 
compatible but not binary identical. This is a problem for the sbt assembly 
plugin which checks for binary identity. In order to solve the problem, we bump 
the commons-configuration version to 1.7 so that only commons-beanutils is 
pulled in. This is necessary, because the transitive promotion of dependencies 
of the shade plugin only excludes the commons-beanutils dependency only from 
the directly depending dependency. All parent dependencies won't have the 
exclusion. This is a problem for SBT which will pull the dependency as part of 
one of the parents, then. Moreover, we replace commons-beanutils by 
commons-beanutils-bean-collections which contains only the non-conflicting 
classes wrt commons-col
 lections.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3459) Make build SBT compatible

2016-02-21 Thread Till Rohrmann (JIRA)

Till Rohrmann created FLINK-3459:


 Summary: Make build SBT compatible
 Key: FLINK-3459
 URL: https://issues.apache.org/jira/browse/FLINK-3459
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.0.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Priority: Minor
 Fix For: 1.0.0


The current flink-shaded-hadoop2 dependency is not SBT compatible because it 
contains {{commons-collection:3.2.1}}, {{commons-beanutils:1.7}} and 
{{commons-beanutils-core:1.8}} dependencies. These dependencies contain 
overlapping classes which are binary compatible but not binary identical. This 
is a problem for the SBT assembly plugin which requires binary identity for 
merging them. Otherwise, the build is aborted. I propose to exclude the 
conflicting dependencies and use instead the 
{{commons-beanutils-bean-collection}} class which contains only then 
non-conflicting classes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3458] [build] Disable shade-flink execu...

2016-02-21 Thread tillrohrmann

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/1681

[FLINK-3458] [build] Disable shade-flink execution in flink-shaded-hadoop

The `shade-flink` execution of the parent pom caused the problem that the 
guava
dependencies were relocated twice in the flink-shaded-hadoop jar. In order 
to
avoid this, this patch disables the `shade-flink` execution.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink fixShading

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1681.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1681


commit 834d352d62b1502d36bdf67d885fd874a70cb073
Author: Till Rohrmann 
Date:   2016-02-17T13:59:16Z

[FLINK-3458] [build] Disable shade-flink execution in flink-shaded-hadoop

The shade-flink execution of the parent pom caused the problem that the 
guava
dependencies were relocated twice in the flink-shaded-hadoop jar. In order 
to
avoid this, this patch disables the shade-flink execution.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3458) Shading broken in flink-shaded-hadoop

2016-02-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-3458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15156017#comment-15156017
 ] 

ASF GitHub Bot commented on FLINK-3458:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/1681

[FLINK-3458] [build] Disable shade-flink execution in flink-shaded-hadoop

The `shade-flink` execution of the parent pom caused the problem that the 
guava
dependencies were relocated twice in the flink-shaded-hadoop jar. In order 
to
avoid this, this patch disables the `shade-flink` execution.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink fixShading

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1681.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1681


commit 834d352d62b1502d36bdf67d885fd874a70cb073
Author: Till Rohrmann 
Date:   2016-02-17T13:59:16Z

[FLINK-3458] [build] Disable shade-flink execution in flink-shaded-hadoop

The shade-flink execution of the parent pom caused the problem that the 
guava
dependencies were relocated twice in the flink-shaded-hadoop jar. In order 
to
avoid this, this patch disables the shade-flink execution.




> Shading broken in flink-shaded-hadoop
> -
>
> Key: FLINK-3458
> URL: https://issues.apache.org/jira/browse/FLINK-3458
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.0.0
>
>
> The shading in flink-shaded-hadoop relocates the guava dependencies of Hadoop 
> twice.
> Once under {{org.apache.flink.shaded.hadoop.com.google}} and 
> {{org.apache.flink.shaded.com.google}}. This causes merge resolution problems 
> when building a fat jar including the flink-shaded-hadoop jar.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3458) Shading broken in flink-shaded-hadoop

2016-02-21 Thread Till Rohrmann (JIRA)

Till Rohrmann created FLINK-3458:


 Summary: Shading broken in flink-shaded-hadoop
 Key: FLINK-3458
 URL: https://issues.apache.org/jira/browse/FLINK-3458
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Priority: Minor
 Fix For: 1.0.0


The shading in flink-shaded-hadoop relocates the guava dependencies of Hadoop 
twice.
Once under {{org.apache.flink.shaded.hadoop.com.google}} and 
{{org.apache.flink.shaded.com.google}}. This causes merge resolution problems 
when building a fat jar including the flink-shaded-hadoop jar.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3457) Link to Apache Flink meetups from the 'Community' section of the website

2016-02-21 Thread Slim Baltagi (JIRA)

Slim Baltagi created FLINK-3457:
---

 Summary: Link to Apache Flink meetups from the 'Community' section 
of the website
 Key: FLINK-3457
 URL: https://issues.apache.org/jira/browse/FLINK-3457
 Project: Flink
  Issue Type: Task
  Components: Documentation
Reporter: Slim Baltagi
Priority: Trivial


Now with the number of Apache Flink meetups increasing worldwide, it is helpful 
to add a link to Apache Flink meetups 
http://www.meetup.com/topics/apache-flink/ to the community section of 
https://flink.apache.org/community.html so visitors can conveniently find them  
right from the Apache Flink website. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3443) JobManager cancel and clear everything fails jobs instead of cancelling

2016-02-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15155982#comment-15155982
 ] 

ASF GitHub Bot commented on FLINK-3443:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1669#discussion_r53564976
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -1487,7 +1487,7 @@ class JobManager(
   }
 }
 
-eg.fail(cause)
+eg.cancel()
--- End diff --

What if we let the job fail with an UnrecoverableException upon JobManager
termination?
On Feb 20, 2016 12:08 AM, "Ufuk Celebi"  wrote:

> In
> 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
> :
>
> > @@ -1487,7 +1487,7 @@ class JobManager(
> >}
> >  }
> >
> > -eg.fail(cause)
> > +eg.cancel()
>
> Good point with the TM logs.
>
> My main reason was that calls to fail (for example a shutdown
> cancelAndClearEverything or shutdown of the InstanceManager) can lead to
> the execution graph being restarted even though the job manager is shut
> down. The cancel call ensures that this does not happen and the execution
> graph eventually enters a terminal state.
>
> The main thing that triggered this change was the following: when you
> start a test cluster and shut it down while a job with a restart strategy
> is running and you *don't* immediately kill the process and have logging
> enabled, you see that the ExecutionGraph is still attempting to recover
> the job.
>
> What I don't understand is how this even happens when we shut down the
> ExecutorService. Any idea?
>
> Do you think there is another way to prevent this behaviour? I would be
> happy to keep the failure cause as before, but couldn't think of any other
> way.
> --
>
> This has been changed as well: a fail will be ignored when the job is
> cancelling or cancelled. That's OK, right?
>
> —
> Reply to this email directly or view it on GitHub
> .
>



> JobManager cancel and clear everything fails jobs instead of cancelling
> ---
>
> Key: FLINK-3443
> URL: https://issues.apache.org/jira/browse/FLINK-3443
> Project: Flink
>  Issue Type: Bug
>  Components: Distributed Runtime
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>
> When the job manager is shut down, it calls {{cancelAndClearEverything}}. 
> This method does not {{cancel}} the {{ExecutionGraph}} instances, but 
> {{fail}}s them, which can lead to {{ExecutionGraph}} restart.
> I've noticed this in tests, where old graph got into a loop of restarts.
> What I don't understand is why the futures etc. are not cancelled when the 
> executor service is shut down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3443] [runtime] Prevent cancelled jobs ...

2016-02-21 Thread tillrohrmann

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1669#discussion_r53564976
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -1487,7 +1487,7 @@ class JobManager(
   }
 }
 
-eg.fail(cause)
+eg.cancel()
--- End diff --

What if we let the job fail with an UnrecoverableException upon JobManager
termination?
On Feb 20, 2016 12:08 AM, "Ufuk Celebi"  wrote:

> In
> 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
> :
>
> > @@ -1487,7 +1487,7 @@ class JobManager(
> >}
> >  }
> >
> > -eg.fail(cause)
> > +eg.cancel()
>
> Good point with the TM logs.
>
> My main reason was that calls to fail (for example a shutdown
> cancelAndClearEverything or shutdown of the InstanceManager) can lead to
> the execution graph being restarted even though the job manager is shut
> down. The cancel call ensures that this does not happen and the execution
> graph eventually enters a terminal state.
>
> The main thing that triggered this change was the following: when you
> start a test cluster and shut it down while a job with a restart strategy
> is running and you *don't* immediately kill the process and have logging
> enabled, you see that the ExecutionGraph is still attempting to recover
> the job.
>
> What I don't understand is how this even happens when we shut down the
> ExecutorService. Any idea?
>
> Do you think there is another way to prevent this behaviour? I would be
> happy to keep the failure cause as before, but couldn't think of any other
> way.
> --
>
> This has been changed as well: a fail will be ignored when the job is
> cancelling or cancelled. That's OK, right?
>
> â
> Reply to this email directly or view it on GitHub
> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3238) EventTimeAllWindowCheckpointingITCase.testSlidingTimeWindow()

2016-02-21 Thread Robert Metzger (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15155979#comment-15155979
 ] 

Robert Metzger commented on FLINK-3238:
---

Again: https://s3.amazonaws.com/archive.travis-ci.org/jobs/110477863/log.txt

> EventTimeAllWindowCheckpointingITCase.testSlidingTimeWindow()
> -
>
> Key: FLINK-3238
> URL: https://issues.apache.org/jira/browse/FLINK-3238
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Reporter: Matthias J. Sax
>  Labels: test-stability
>
> "Maven produced no output for 300 seconds."
> https://travis-ci.org/mjsax/flink/jobs/102475719



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-1502) Expose metrics to graphite, ganglia and JMX.

[GitHub] flink pull request: [WIP][FLINK-3422][streaming][api-breaking] Scr...

[jira] [Commented] (FLINK-3422) Scramble HashPartitioner hashes

[jira] [Commented] (FLINK-3422) Scramble HashPartitioner hashes

[GitHub] flink pull request: [WIP][FLINK-3422][streaming][api-breaking] Scr...

[jira] [Commented] (FLINK-3416) [py] .bat files fail when path contains spaces

[GitHub] flink pull request: [FLINK-3416] [py] Support for spaces in flink ...

[jira] [Commented] (FLINK-3416) [py] .bat files fail when path contains spaces

[GitHub] flink pull request: [FLINK-3416] [py] Support for spaces in flink ...

[jira] [Commented] (FLINK-3387) Replace all ByteArrayOutputStream and DataOutputStream use in State Backends by DataOutputSerializer

[jira] [Assigned] (FLINK-3440) Kafka should also checkpoint partitions where no initial offset was retrieved

[jira] [Commented] (FLINK-3455) Bump Kafka 0.9 connector dependency to Kafka 0.9.0.1

[GitHub] flink pull request: [FLINK-3455] Bump Kafka to 0.9.0.1 and 0.8.2.2

[GitHub] flink pull request: FLINK-2213 Makes the number of vcores per YARN...

[jira] [Commented] (FLINK-2213) Configure number of vcores

[jira] [Commented] (FLINK-3385) Fix outer join skipping unprobed partitions

[GitHub] flink pull request: FLINK-2213 Makes the number of vcores per YARN...

[jira] [Commented] (FLINK-2213) Configure number of vcores

[jira] [Commented] (FLINK-3443) JobManager cancel and clear everything fails jobs instead of cancelling

[GitHub] flink pull request: [FLINK-3443] [runtime] Prevent cancelled jobs ...

[jira] [Commented] (FLINK-3460) Make flink-streaming-connectors' flink dependencies provided

[GitHub] flink pull request: [FLINK-3460] [build] Set Flink dependencies in...

[jira] [Commented] (FLINK-3459) Make build SBT compatible

[jira] [Created] (FLINK-3460) Make flink-streaming-connectors' flink dependencies provided

[GitHub] flink pull request: [FLINK-3459] [build] Fix conflicting dependenc...

[jira] [Created] (FLINK-3459) Make build SBT compatible

[GitHub] flink pull request: [FLINK-3458] [build] Disable shade-flink execu...

[jira] [Commented] (FLINK-3458) Shading broken in flink-shaded-hadoop

[jira] [Created] (FLINK-3458) Shading broken in flink-shaded-hadoop

[jira] [Created] (FLINK-3457) Link to Apache Flink meetups from the 'Community' section of the website

[jira] [Commented] (FLINK-3443) JobManager cancel and clear everything fails jobs instead of cancelling

[GitHub] flink pull request: [FLINK-3443] [runtime] Prevent cancelled jobs ...

[jira] [Commented] (FLINK-3238) EventTimeAllWindowCheckpointingITCase.testSlidingTimeWindow()

33 matches

Site Navigation

Mail list logo

Footer information