[jira] [Commented] (FLINK-3501) flink-runtime-0.10.2-hadoop-1 jar problem
[ https://issues.apache.org/jira/browse/FLINK-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168480#comment-15168480 ] Paul Harris commented on FLINK-3501: This seems to be an OSX issue. I have tried with Oracle JDK7 u75, JDK7 u79 and JDK8 u84. There is more information here http://stackoverflow.com/questions/10522835/hadoop-java-io-ioexception-mkdirs-failed-to-create-some-path it relates to the jar containing a file META-INF/LICENSE and also a directory called META-INF/license. If I edit the archive with {code:none} zip -d flink-runtime-0.10.2-hadoop1.jar META_INF/license {code} then unjar with {code:none} jar xf flink-runtime-0.10.2-hadoop1.jar {code} everything works ok, but this means editing my gradle cache which is a bit of a pain. > flink-runtime-0.10.2-hadoop-1 jar problem > - > > Key: FLINK-3501 > URL: https://issues.apache.org/jira/browse/FLINK-3501 > Project: Flink > Issue Type: Bug > Components: Distributed Runtime >Affects Versions: 0.10.2 >Reporter: Paul Harris > > Unjaring flink-runtime-0.10.2-hadoop-1 results in an error. > Steps to reproduce > {code:none} > jar xf flink-runtime-0.10.2-hadoop1.jar > {code} > Result... > {code:none} > java.io.IOException: META-INF/license : could not create directory > at sun.tools.jar.Main.extractFile(Main.java:934) > at sun.tools.jar.Main.extract(Main.java:877) > at sun.tools.jar.Main.run(Main.java:263) > at sun.tools.jar.Main.main(Main.java:1177) > {code} > This is the result of the jar containing a file 'license' and a directory > license'. The file is unpacked first, preventing the directory being unpacked > to the same name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3461) Remove duplicate condition check in ZooKeeperLeaderElectionService
[ https://issues.apache.org/jira/browse/FLINK-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168434#comment-15168434 ] sahitya pavurala commented on FLINK-3461: - I submitted a pull request for this, can you please review it. Thanks. > Remove duplicate condition check in ZooKeeperLeaderElectionService > -- > > Key: FLINK-3461 > URL: https://issues.apache.org/jira/browse/FLINK-3461 > Project: Flink > Issue Type: Improvement >Reporter: Ufuk Celebi >Priority: Trivial > > In {{ZooKeeperLeaderElectionService}}: > {code} > @Override > public boolean hasLeadership() { > if(leaderLatch.getState().equals(LeaderLatch.State.STARTED)) { > return leaderLatch.hasLeadership(); > } else { > return false; > } > } > {code} > Looking into the leader {{leaderLatch.hasLeadership()}}, the state gets > checked already/again. > {code} > return (state.get() == State.STARTED) && hasLeadership.get(); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLINK-3501) flink-runtime-0.10.2-hadoop-1 jar problem
[ https://issues.apache.org/jira/browse/FLINK-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168206#comment-15168206 ] Paul Harris edited comment on FLINK-3501 at 2/26/16 1:20 AM: - I am on a mac with {code:none} java -version java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) {code} When I download the jar from Apache I can untar fine. When I pull via gradle from mvnrepository I have the issue. I am bringing in the following dependencies {code:none} 'org.apache.flink:flink-clients:0.10.2-hadoop1' 'org.apache.flink:flink-scala:0.10.2-hadoop1' {code} The file in the gradle cache is the one with the issue (I have deleted the cached files and allowed gradle to pull again) . This sounds more like a gradle-ism than a jar'ing problem. I think we can mark this as resolved. I'll re-open if I continue to have issues. was (Author: trickbooter): I am on a mac with {code:none} java -version java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) {code} When I download the jar from Apache I can untar fine. When I pull via gradle from mvnrepository I have the issue. I am bringing in the following dependencies {code:none} 'org.apache.flink:flink-clients:0.10.2-hadoop1' 'org.apache.flink:flink-scala:0.10.2-hadoop1' {code} The file in the gradle cache is the one with the issue (I have deleted the cached files and allowed gradle to pull again) . I don't know much mvn packaging I am afraid, so I don't know if the folllowing is relevent, but... {code:none} cd /path/to/gradle/cache md5 flink-runtime-0.10.2-hadoop1.jar MD5 (flink-runtime-0.10.2-hadoop1.jar) = dc747ec3da5ddc8556aa707b3c5fc9a2 cd /path/to/mvn/direct/download MD5 (/Users/paul/dev/tmp/flink-core-0.10.2-hadoop1.jar) = 54a4c422a57baecb48d609e7187448d8 {code} This sounds more like a gradle-ism than a jar'ing problem. I think we can mark this as resolved. I'll re-open if I continue to have issues. > flink-runtime-0.10.2-hadoop-1 jar problem > - > > Key: FLINK-3501 > URL: https://issues.apache.org/jira/browse/FLINK-3501 > Project: Flink > Issue Type: Bug > Components: Distributed Runtime >Affects Versions: 0.10.2 >Reporter: Paul Harris > > Unjaring flink-runtime-0.10.2-hadoop-1 results in an error. > Steps to reproduce > {code:none} > jar xf flink-runtime-0.10.2-hadoop1.jar > {code} > Result... > {code:none} > java.io.IOException: META-INF/license : could not create directory > at sun.tools.jar.Main.extractFile(Main.java:934) > at sun.tools.jar.Main.extract(Main.java:877) > at sun.tools.jar.Main.run(Main.java:263) > at sun.tools.jar.Main.main(Main.java:1177) > {code} > This is the result of the jar containing a file 'license' and a directory > license'. The file is unpacked first, preventing the directory being unpacked > to the same name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLINK-3501) flink-runtime-0.10.2-hadoop-1 jar problem
[ https://issues.apache.org/jira/browse/FLINK-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168206#comment-15168206 ] Paul Harris edited comment on FLINK-3501 at 2/26/16 12:57 AM: -- I am on a mac with {code:none} java -version java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) {code} When I download the jar from Apache I can untar fine. When I pull via gradle from mvnrepository I have the issue. I am bringing in the following dependencies {code:none} 'org.apache.flink:flink-clients:0.10.2-hadoop1' 'org.apache.flink:flink-scala:0.10.2-hadoop1' {code} The file in the gradle cache is the one with the issue (I have deleted the cached files and allowed gradle to pull again) . I don't know much mvn packaging I am afraid, so I don't know if the folllowing is relevent, but... {code:none} cd /path/to/gradle/cache md5 flink-runtime-0.10.2-hadoop1.jar MD5 (flink-runtime-0.10.2-hadoop1.jar) = dc747ec3da5ddc8556aa707b3c5fc9a2 cd /path/to/mvn/direct/download MD5 (/Users/paul/dev/tmp/flink-core-0.10.2-hadoop1.jar) = 54a4c422a57baecb48d609e7187448d8 {code} This sounds more like a gradle-ism than a jar'ing problem. I think we can mark this as resolved. I'll re-open if I continue to have issues. was (Author: trickbooter): I am on a mac with ``` java -version java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) ``` When I download the jar from Apache I can untar fine. When I pull via gradle from mvnrepository I have the issue. I am bringing in the following dependencies ``` 'org.apache.flink:flink-clients:0.10.2-hadoop1' 'org.apache.flink:flink-scala:0.10.2-hadoop1' ``` The file in the gradle cache is the one with the issue (I have deleted the cached files and allowed gradle to pull again) . I don't know much mvn packaging I am afraid, so I don't know if the folllowing is relevent, but... ``` cd /path/to/gradle/cache md5 flink-runtime-0.10.2-hadoop1.jar MD5 (flink-runtime-0.10.2-hadoop1.jar) = dc747ec3da5ddc8556aa707b3c5fc9a2 cd /path/to/mvn/direct/download MD5 (/Users/paul/dev/tmp/flink-core-0.10.2-hadoop1.jar) = 54a4c422a57baecb48d609e7187448d8 ``` This sounds more like a gradle-ism than a jar'ing problem. I think we can mark this as resolved. I'll re-open if I continue to have issues. > flink-runtime-0.10.2-hadoop-1 jar problem > - > > Key: FLINK-3501 > URL: https://issues.apache.org/jira/browse/FLINK-3501 > Project: Flink > Issue Type: Bug > Components: Distributed Runtime >Affects Versions: 0.10.2 >Reporter: Paul Harris > > Unjaring flink-runtime-0.10.2-hadoop-1 results in an error. > Steps to reproduce > {code:none} > jar xf flink-runtime-0.10.2-hadoop1.jar > {code} > Result... > {code:none} > java.io.IOException: META-INF/license : could not create directory > at sun.tools.jar.Main.extractFile(Main.java:934) > at sun.tools.jar.Main.extract(Main.java:877) > at sun.tools.jar.Main.run(Main.java:263) > at sun.tools.jar.Main.main(Main.java:1177) > {code} > This is the result of the jar containing a file 'license' and a directory > license'. The file is unpacked first, preventing the directory being unpacked > to the same name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3501) flink-runtime-0.10.2-hadoop-1 jar problem
[ https://issues.apache.org/jira/browse/FLINK-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168206#comment-15168206 ] Paul Harris commented on FLINK-3501: I am on a mac with ``` java -version java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode) ``` When I download the jar from Apache I can untar fine. When I pull via gradle from mvnrepository I have the issue. I am bringing in the following dependencies ``` 'org.apache.flink:flink-clients:0.10.2-hadoop1' 'org.apache.flink:flink-scala:0.10.2-hadoop1' ``` The file in the gradle cache is the one with the issue (I have deleted the cached files and allowed gradle to pull again) . I don't know much mvn packaging I am afraid, so I don't know if the folllowing is relevent, but... ``` cd /path/to/gradle/cache md5 flink-runtime-0.10.2-hadoop1.jar MD5 (flink-runtime-0.10.2-hadoop1.jar) = dc747ec3da5ddc8556aa707b3c5fc9a2 cd /path/to/mvn/direct/download MD5 (/Users/paul/dev/tmp/flink-core-0.10.2-hadoop1.jar) = 54a4c422a57baecb48d609e7187448d8 ``` This sounds more like a gradle-ism than a jar'ing problem. I think we can mark this as resolved. I'll re-open if I continue to have issues. > flink-runtime-0.10.2-hadoop-1 jar problem > - > > Key: FLINK-3501 > URL: https://issues.apache.org/jira/browse/FLINK-3501 > Project: Flink > Issue Type: Bug > Components: Distributed Runtime >Affects Versions: 0.10.2 >Reporter: Paul Harris > > Unjaring flink-runtime-0.10.2-hadoop-1 results in an error. > Steps to reproduce > {code:none} > jar xf flink-runtime-0.10.2-hadoop1.jar > {code} > Result... > {code:none} > java.io.IOException: META-INF/license : could not create directory > at sun.tools.jar.Main.extractFile(Main.java:934) > at sun.tools.jar.Main.extract(Main.java:877) > at sun.tools.jar.Main.run(Main.java:263) > at sun.tools.jar.Main.main(Main.java:1177) > {code} > This is the result of the jar containing a file 'license' and a directory > license'. The file is unpacked first, preventing the directory being unpacked > to the same name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3485) The SerializedListAccumulator value doest seem to be right
[ https://issues.apache.org/jira/browse/FLINK-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168142#comment-15168142 ] ASF GitHub Bot commented on FLINK-3485: --- GitHub user mxm opened a pull request: https://github.com/apache/flink/pull/1717 [FLINK-3485] implement toString for SerializedListAccumulator You can merge this pull request into a Git repository by running: $ git pull https://github.com/mxm/flink FLINK-3485 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1717.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1717 commit 420b7d164168f8557e82f836ed809f3c81e1983a Author: Maximilian Michels Date: 2016-02-25T23:49:28Z [FLINK-3485] implement toString for SerializedListAccumulator > The SerializedListAccumulator value doest seem to be right > -- > > Key: FLINK-3485 > URL: https://issues.apache.org/jira/browse/FLINK-3485 > Project: Flink > Issue Type: Bug > Components: Web Client >Reporter: sahitya pavurala >Priority: Minor > Attachments: screenshot-1.png > > > When run flink examples, the accumulator value keeps changing, as it shows > the hashcode value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3485] implement toString for Serialized...
GitHub user mxm opened a pull request: https://github.com/apache/flink/pull/1717 [FLINK-3485] implement toString for SerializedListAccumulator You can merge this pull request into a Git repository by running: $ git pull https://github.com/mxm/flink FLINK-3485 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1717.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1717 commit 420b7d164168f8557e82f836ed809f3c81e1983a Author: Maximilian Michels Date: 2016-02-25T23:49:28Z [FLINK-3485] implement toString for SerializedListAccumulator --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3485) The SerializedListAccumulator value doest seem to be right
[ https://issues.apache.org/jira/browse/FLINK-3485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168139#comment-15168139 ] Maximilian Michels commented on FLINK-3485: --- You're right. The accumulator is printed using the {{toString()}} method. What you see is the default Object.toString() method of the {{SerializedListAccumuator}}. I suppose we could change this to something nicer like printing the number of items. The accumulator is used for aggregating the results when you use {{collect()}}. > The SerializedListAccumulator value doest seem to be right > -- > > Key: FLINK-3485 > URL: https://issues.apache.org/jira/browse/FLINK-3485 > Project: Flink > Issue Type: Bug > Components: Web Client >Reporter: sahitya pavurala >Priority: Minor > Attachments: screenshot-1.png > > > When run flink examples, the accumulator value keeps changing, as it shows > the hashcode value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-3510) Pattern class class-level comment misses type argument
[ https://issues.apache.org/jira/browse/FLINK-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ufuk Celebi closed FLINK-3510. -- Resolution: Fixed Fixed in 930a266. > Pattern class class-level comment misses type argument > -- > > Key: FLINK-3510 > URL: https://issues.apache.org/jira/browse/FLINK-3510 > Project: Flink > Issue Type: Bug > Components: CEP >Reporter: Ufuk Celebi >Assignee: Ufuk Celebi >Priority: Trivial > > Class-level comment code example reads: > {code} > * {@code > * Pattern pattern = Pattern.begin("start") > * .next("middle").subtype(F.class) > * .followedBy("end").where(new MyFilterFunction()); > * } > * > {code} > The 2nd type paramter for {{Pattern}} is missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3517) Number of job and task managers not checked in scripts
[ https://issues.apache.org/jira/browse/FLINK-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168098#comment-15168098 ] ASF GitHub Bot commented on FLINK-3517: --- GitHub user uce opened a pull request: https://github.com/apache/flink/pull/1716 [FLINK-3517] [dist] Only count active PIDs in start script ```bash $ bin/start-cluster.sh Starting cluster. Starting jobmanager daemon on host pablo. Starting taskmanager daemon on host pablo. $ bin/taskmanager.sh start [INFO] 1 instance(s) of taskmanager are already running on pablo. Starting taskmanager daemon on host pablo. $ bin/taskmanager.sh start [INFO] 2 instance(s) of taskmanager are already running on pablo. Starting taskmanager daemon on host pablo. $ bin/taskmanager.sh start [INFO] 3 instance(s) of taskmanager are already running on pablo. Starting taskmanager daemon on host pablo. $ jps 27328 TaskManager 27140 TaskManager 26949 TaskManager 26523 JobManager 26716 TaskManager $ kill -9 27140 $ bin/taskmanager.sh start >>> [INFO] 3 instance(s) of taskmanager are already running on pablo <<< Correct now Starting taskmanager daemon on host pablo. $ bin/stop-cluster.sh Stopping taskmanager daemon (pid: 27545) on host pablo. Stopping jobmanager daemon (pid: 26523) on host pablo. $ bin/taskmanager.sh stop Stopping taskmanager daemon (pid: 27328) on host pablo. $ bin/taskmanager.sh stop No taskmanager daemon (pid: 27140) is running anymore on pablo. $ bin/taskmanager.sh stop Stopping taskmanager daemon (pid: 26949) on host pablo. $ bin/taskmanager.sh stop Stopping taskmanager daemon (pid: 26716) on host pablo. $ bin/taskmanager.sh stop No taskmanager daemon to stop on host pablo. ``` We can further improve the stop part by repeatedly the PIDs in the pid file if a value is not matching an active PID. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uce/flink 3517-scripts Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1716.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1716 commit e037c89404704b8f8bd02911e65dc1dd24b1e836 Author: Ufuk Celebi Date: 2016-02-25T23:11:48Z [FLINK-3517] [dist] Only count active PIDs in start script > Number of job and task managers not checked in scripts > -- > > Key: FLINK-3517 > URL: https://issues.apache.org/jira/browse/FLINK-3517 > Project: Flink > Issue Type: Test > Components: Start-Stop Scripts >Reporter: Ufuk Celebi >Assignee: Ufuk Celebi >Priority: Minor > > The start up scripts determine whether a job or task manager is running via a > pids file. If a process, which is part of the pid file, is destroyed (for > example on failure) outside of the scripts, a warning for multiple job > managers are printed even though they are not running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3517] [dist] Only count active PIDs in ...
GitHub user uce opened a pull request: https://github.com/apache/flink/pull/1716 [FLINK-3517] [dist] Only count active PIDs in start script ```bash $ bin/start-cluster.sh Starting cluster. Starting jobmanager daemon on host pablo. Starting taskmanager daemon on host pablo. $ bin/taskmanager.sh start [INFO] 1 instance(s) of taskmanager are already running on pablo. Starting taskmanager daemon on host pablo. $ bin/taskmanager.sh start [INFO] 2 instance(s) of taskmanager are already running on pablo. Starting taskmanager daemon on host pablo. $ bin/taskmanager.sh start [INFO] 3 instance(s) of taskmanager are already running on pablo. Starting taskmanager daemon on host pablo. $ jps 27328 TaskManager 27140 TaskManager 26949 TaskManager 26523 JobManager 26716 TaskManager $ kill -9 27140 $ bin/taskmanager.sh start >>> [INFO] 3 instance(s) of taskmanager are already running on pablo <<< Correct now Starting taskmanager daemon on host pablo. $ bin/stop-cluster.sh Stopping taskmanager daemon (pid: 27545) on host pablo. Stopping jobmanager daemon (pid: 26523) on host pablo. $ bin/taskmanager.sh stop Stopping taskmanager daemon (pid: 27328) on host pablo. $ bin/taskmanager.sh stop No taskmanager daemon (pid: 27140) is running anymore on pablo. $ bin/taskmanager.sh stop Stopping taskmanager daemon (pid: 26949) on host pablo. $ bin/taskmanager.sh stop Stopping taskmanager daemon (pid: 26716) on host pablo. $ bin/taskmanager.sh stop No taskmanager daemon to stop on host pablo. ``` We can further improve the stop part by repeatedly the PIDs in the pid file if a value is not matching an active PID. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uce/flink 3517-scripts Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1716.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1716 commit e037c89404704b8f8bd02911e65dc1dd24b1e836 Author: Ufuk Celebi Date: 2016-02-25T23:11:48Z [FLINK-3517] [dist] Only count active PIDs in start script --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[ https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167956#comment-15167956 ] ASF GitHub Bot commented on FLINK-3340: --- Github user greghogan commented on the pull request: https://github.com/apache/flink/pull/1626#issuecomment-189004909 Pushed. Thanks for the explanation. > Fix object juggling in drivers > -- > > Key: FLINK-3340 > URL: https://issues.apache.org/jira/browse/FLINK-3340 > Project: Flink > Issue Type: Bug > Components: Distributed Runtime >Affects Versions: 1.0.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Critical > Fix For: 1.0.0 > > > {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are > not properly tracking objects for reuse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3340] [runtime] Fix object juggling in ...
Github user greghogan commented on the pull request: https://github.com/apache/flink/pull/1626#issuecomment-189004909 Pushed. Thanks for the explanation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[ https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167871#comment-15167871 ] ASF GitHub Bot commented on FLINK-3340: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1626#issuecomment-188985984 If you can quickly do it, yes. Otherwise whoever merges this tomorrow, please remove the sysout statements. We try to keep the maven output a bit cleaner that way. If you want to output something into the logs for the tests, just use a regular slf4j logger... > Fix object juggling in drivers > -- > > Key: FLINK-3340 > URL: https://issues.apache.org/jira/browse/FLINK-3340 > Project: Flink > Issue Type: Bug > Components: Distributed Runtime >Affects Versions: 1.0.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Critical > Fix For: 1.0.0 > > > {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are > not properly tracking objects for reuse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3340] [runtime] Fix object juggling in ...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1626#issuecomment-188985984 If you can quickly do it, yes. Otherwise whoever merges this tomorrow, please remove the sysout statements. We try to keep the maven output a bit cleaner that way. If you want to output something into the logs for the tests, just use a regular slf4j logger... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[ https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167864#comment-15167864 ] ASF GitHub Bot commented on FLINK-3340: --- Github user greghogan commented on the pull request: https://github.com/apache/flink/pull/1626#issuecomment-188982973 @StephanEwen just to clarify, since time is of the essence, would you like me to update the pull request without the println's? > Fix object juggling in drivers > -- > > Key: FLINK-3340 > URL: https://issues.apache.org/jira/browse/FLINK-3340 > Project: Flink > Issue Type: Bug > Components: Distributed Runtime >Affects Versions: 1.0.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Critical > Fix For: 1.0.0 > > > {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are > not properly tracking objects for reuse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3340] [runtime] Fix object juggling in ...
Github user greghogan commented on the pull request: https://github.com/apache/flink/pull/1626#issuecomment-188982973 @StephanEwen just to clarify, since time is of the essence, would you like me to update the pull request without the println's? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3482) Implement Union Support
[ https://issues.apache.org/jira/browse/FLINK-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167755#comment-15167755 ] ASF GitHub Bot commented on FLINK-3482: --- GitHub user vasia opened a pull request: https://github.com/apache/flink/pull/1715 [FLINK-3482] implement union translation This PR add union support to Table on Calcite. I've implemented a custom `JoinUnionTransposeRule` because Calcite's only matches with LogicalUnion. You can merge this pull request into a Git repository by running: $ git pull https://github.com/vasia/flink union Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1715.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1715 commit 6991e601ceabc63ec9f367c4ad2e4f75468aaf2b Author: vasia Date: 2016-02-25T19:52:19Z [FLINK-3482] implement union translation - implement custom JoinUnionTransposeRules because Calcite's only match with LogicalUnion > Implement Union Support > --- > > Key: FLINK-3482 > URL: https://issues.apache.org/jira/browse/FLINK-3482 > Project: Flink > Issue Type: Sub-task > Components: Table API >Reporter: Vasia Kalavri >Assignee: Vasia Kalavri > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3482] implement union translation
GitHub user vasia opened a pull request: https://github.com/apache/flink/pull/1715 [FLINK-3482] implement union translation This PR add union support to Table on Calcite. I've implemented a custom `JoinUnionTransposeRule` because Calcite's only matches with LogicalUnion. You can merge this pull request into a Git repository by running: $ git pull https://github.com/vasia/flink union Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1715.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1715 commit 6991e601ceabc63ec9f367c4ad2e4f75468aaf2b Author: vasia Date: 2016-02-25T19:52:19Z [FLINK-3482] implement union translation - implement custom JoinUnionTransposeRules because Calcite's only match with LogicalUnion --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3226] Improvements for expected types
Github user twalthr closed the pull request at: https://github.com/apache/flink/pull/1709 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs
[ https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167678#comment-15167678 ] ASF GitHub Bot commented on FLINK-3226: --- Github user twalthr closed the pull request at: https://github.com/apache/flink/pull/1709 > Translate optimized logical Table API plans into physical plans representing > DataSet programs > - > > Key: FLINK-3226 > URL: https://issues.apache.org/jira/browse/FLINK-3226 > Project: Flink > Issue Type: Sub-task > Components: Table API >Reporter: Fabian Hueske >Assignee: Chengxiang Li > > This issue is about translating an (optimized) logical Table API (see > FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 > representation of the DataSet program that will be executed. This means: > - Each Flink RelNode refers to exactly one Flink DataSet or DataStream > operator. > - All (join and grouping) keys of Flink operators are correctly specified. > - The expressions which are to be executed in user-code are identified. > - All fields are referenced with their physical execution-time index. > - Flink type information is available. > - Optional: Add physical execution hints for joins > The translation should be the final part of Calcite's optimization process. > For this task we need to: > - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one > Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all > relevant operator information (keys, user-code expression, strategy hints, > parallelism). > - implement rules to translate optimized Calcite RelNodes into Flink > RelNodes. We start with a straight-forward mapping and later add rules that > merge several relational operators into a single Flink operator, e.g., merge > a join followed by a filter. Timo implemented some rules for the first SQL > implementation which can be used as a starting point. > - Integrate the translation rules into the Calcite optimization process -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs
[ https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167662#comment-15167662 ] ASF GitHub Bot commented on FLINK-3226: --- Github user vasia commented on the pull request: https://github.com/apache/flink/pull/1709#issuecomment-188933506 You can close this now @twalthr. I've merged it. > Translate optimized logical Table API plans into physical plans representing > DataSet programs > - > > Key: FLINK-3226 > URL: https://issues.apache.org/jira/browse/FLINK-3226 > Project: Flink > Issue Type: Sub-task > Components: Table API >Reporter: Fabian Hueske >Assignee: Chengxiang Li > > This issue is about translating an (optimized) logical Table API (see > FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 > representation of the DataSet program that will be executed. This means: > - Each Flink RelNode refers to exactly one Flink DataSet or DataStream > operator. > - All (join and grouping) keys of Flink operators are correctly specified. > - The expressions which are to be executed in user-code are identified. > - All fields are referenced with their physical execution-time index. > - Flink type information is available. > - Optional: Add physical execution hints for joins > The translation should be the final part of Calcite's optimization process. > For this task we need to: > - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one > Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all > relevant operator information (keys, user-code expression, strategy hints, > parallelism). > - implement rules to translate optimized Calcite RelNodes into Flink > RelNodes. We start with a straight-forward mapping and later add rules that > merge several relational operators into a single Flink operator, e.g., merge > a join followed by a filter. Timo implemented some rules for the first SQL > implementation which can be used as a starting point. > - Integrate the translation rules into the Calcite optimization process -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3463) Implement Calc Support
[ https://issues.apache.org/jira/browse/FLINK-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167661#comment-15167661 ] ASF GitHub Bot commented on FLINK-3463: --- Github user vasia closed the pull request at: https://github.com/apache/flink/pull/1696 > Implement Calc Support > -- > > Key: FLINK-3463 > URL: https://issues.apache.org/jira/browse/FLINK-3463 > Project: Flink > Issue Type: Sub-task > Components: Table API >Reporter: Vasia Kalavri >Assignee: Vasia Kalavri > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3226] Improvements for expected types
Github user vasia commented on the pull request: https://github.com/apache/flink/pull/1709#issuecomment-188933506 You can close this now @twalthr. I've merged it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator
[ https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167659#comment-15167659 ] ASF GitHub Bot commented on FLINK-3513: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1712 > Fix interplay of automatic Operator UID and Changing name of WindowOperator > --- > > Key: FLINK-3513 > URL: https://issues.apache.org/jira/browse/FLINK-3513 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.0.0 >Reporter: Aljoscha Krettek >Assignee: Ufuk Celebi >Priority: Blocker > Fix For: 1.0.0 > > > WindowOperator can have a changing name because it has the TypeSerializer > .toString() output in it's name. For some type serializers that don't > implement toString() this means that the name changes. > This means that savepoint restore does not work for the automatically > generated UID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3513] [FLINK-3512] Fix savepoint issues
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1712 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3463] implement calc translation
Github user vasia closed the pull request at: https://github.com/apache/flink/pull/1696 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-3512) Savepoint backend should not revert to "jobmanager"
[ https://issues.apache.org/jira/browse/FLINK-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ufuk Celebi closed FLINK-3512. -- Resolution: Fixed Fix Version/s: 1.0.0 Fixed in c9cba27. > Savepoint backend should not revert to "jobmanager" > --- > > Key: FLINK-3512 > URL: https://issues.apache.org/jira/browse/FLINK-3512 > Project: Flink > Issue Type: Bug >Reporter: Aljoscha Krettek >Assignee: Ufuk Celebi >Priority: Blocker > Fix For: 1.0.0 > > > When not setting a state backend in the config the savepoint coordinator > defaults to use "jobmanager" as savepoint backend, ignoring the setting for > the savepoint backend in the config. > This means that when setting a StateBackend only per job the system always > falls back to "jobmanager" even though it could use "filesystem" to make > savepoints persistent. > See SavepointStoreFactory.java:71 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator
[ https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ufuk Celebi closed FLINK-3513. -- Resolution: Fixed Fix Version/s: 1.0.0 Fixed in d90672f. > Fix interplay of automatic Operator UID and Changing name of WindowOperator > --- > > Key: FLINK-3513 > URL: https://issues.apache.org/jira/browse/FLINK-3513 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.0.0 >Reporter: Aljoscha Krettek >Assignee: Ufuk Celebi >Priority: Blocker > Fix For: 1.0.0 > > > WindowOperator can have a changing name because it has the TypeSerializer > .toString() output in it's name. For some type serializers that don't > implement toString() this means that the name changes. > This means that savepoint restore does not work for the automatically > generated UID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator
[ https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167654#comment-15167654 ] ASF GitHub Bot commented on FLINK-3513: --- Github user uce commented on the pull request: https://github.com/apache/flink/pull/1712#issuecomment-188932532 I'm merging this to master. > Fix interplay of automatic Operator UID and Changing name of WindowOperator > --- > > Key: FLINK-3513 > URL: https://issues.apache.org/jira/browse/FLINK-3513 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.0.0 >Reporter: Aljoscha Krettek >Assignee: Ufuk Celebi >Priority: Blocker > > WindowOperator can have a changing name because it has the TypeSerializer > .toString() output in it's name. For some type serializers that don't > implement toString() this means that the name changes. > This means that savepoint restore does not work for the automatically > generated UID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3513] [FLINK-3512] Fix savepoint issues
Github user uce commented on the pull request: https://github.com/apache/flink/pull/1712#issuecomment-188932532 I'm merging this to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-3517) Number of job and task managers not checked in scripts
Ufuk Celebi created FLINK-3517: -- Summary: Number of job and task managers not checked in scripts Key: FLINK-3517 URL: https://issues.apache.org/jira/browse/FLINK-3517 Project: Flink Issue Type: Test Components: Start-Stop Scripts Reporter: Ufuk Celebi Assignee: Ufuk Celebi Priority: Minor The start up scripts determine whether a job or task manager is running via a pids file. If a process, which is part of the pid file, is destroyed (for example on failure) outside of the scripts, a warning for multiple job managers are printed even though they are not running. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3340) Fix object juggling in drivers
[ https://issues.apache.org/jira/browse/FLINK-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167641#comment-15167641 ] ASF GitHub Bot commented on FLINK-3340: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1626#issuecomment-188929386 This looks good to me. +1 for merging, but removing the System.out.println > Fix object juggling in drivers > -- > > Key: FLINK-3340 > URL: https://issues.apache.org/jira/browse/FLINK-3340 > Project: Flink > Issue Type: Bug > Components: Distributed Runtime >Affects Versions: 1.0.0 >Reporter: Greg Hogan >Assignee: Greg Hogan >Priority: Critical > Fix For: 1.0.0 > > > {{ReduceDriver}}, {{ReduceCombineDriver}}, and {{ChainedAllReduceDriver}} are > not properly tracking objects for reuse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3340] [runtime] Fix object juggling in ...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1626#issuecomment-188929386 This looks good to me. +1 for merging, but removing the System.out.println --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-3516) JobManagerHACheckpointRecoveryITCase testCheckpointedStreamingSumProgram still fails :(
Ufuk Celebi created FLINK-3516: -- Summary: JobManagerHACheckpointRecoveryITCase testCheckpointedStreamingSumProgram still fails :( Key: FLINK-3516 URL: https://issues.apache.org/jira/browse/FLINK-3516 Project: Flink Issue Type: Test Reporter: Ufuk Celebi https://s3.amazonaws.com/archive.travis-ci.org/jobs/111782050/log.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [hotfix] fix PageRank example to run with no a...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1714#issuecomment-188923753 +1 good to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [hotfix] fix PageRank example to run with no a...
GitHub user vasia opened a pull request: https://github.com/apache/flink/pull/1714 [hotfix] fix PageRank example to run with no args; some doc fixes This PR includes a fix for the java PageRank example to retrieve the required parameter `numPages ` when run with no arguments and some documentation fixes. You can merge this pull request into a Git repository by running: $ git pull https://github.com/vasia/flink examples-fixes Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1714.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1714 commit d9c89b26d288e75dfc708b42816cc35015dcbf20 Author: vasia Date: 2016-02-25T16:36:37Z [hotfix] fix PageRank example to run with no args; some doc fixes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3442) Expose savepoint button on web ui
[ https://issues.apache.org/jira/browse/FLINK-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167599#comment-15167599 ] Gyula Fora commented on FLINK-3442: --- I thought that this should be equivalent to calling bin/flink savepoint jobid. So whatever savepoint path is configured for the given job will be used. > Expose savepoint button on web ui > - > > Key: FLINK-3442 > URL: https://issues.apache.org/jira/browse/FLINK-3442 > Project: Flink > Issue Type: Improvement > Components: Streaming, Webfrontend >Reporter: Gyula Fora >Priority: Minor > > Similarly to Cancel there should be a Savepoint button to initiate a > savepoint for streaming jobs. > These 2 buttons should NOT be next to each other :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3277) Use Value types in Gelly API
[ https://issues.apache.org/jira/browse/FLINK-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167586#comment-15167586 ] ASF GitHub Bot commented on FLINK-3277: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1671#issuecomment-188918217 Thanks for checking this out. Do you have an estimate of how this affects Gelly performance for some algorithms? How much does this improve the performance? > Use Value types in Gelly API > > > Key: FLINK-3277 > URL: https://issues.apache.org/jira/browse/FLINK-3277 > Project: Flink > Issue Type: Improvement > Components: Gelly >Affects Versions: 1.0.0 >Reporter: Greg Hogan >Assignee: Greg Hogan > > This would be a breaking change so the discussion needs to happen before the > 1.0.0 release. > I think it would benefit Flink to use {{Value}} types wherever possible. The > {{Graph}} functions {{inDegrees}}, {{outDegrees}}, and {{getDegrees}} each > return {{DataSet>}}. Using {{Long}} creates a new heap object > for every serialization and deserialization. The mutable {{Value}} types do > not suffer from this issue when object reuse is enabled. > I lean towards a preference for conciseness in documentation and performance > in examples and APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3277] Use Value types in Gelly API
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1671#issuecomment-188918217 Thanks for checking this out. Do you have an estimate of how this affects Gelly performance for some algorithms? How much does this improve the performance? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3442) Expose savepoint button on web ui
[ https://issues.apache.org/jira/browse/FLINK-3442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167572#comment-15167572 ] Stephan Ewen commented on FLINK-3442: - This would rely on some pre-configured default safepoint target path? > Expose savepoint button on web ui > - > > Key: FLINK-3442 > URL: https://issues.apache.org/jira/browse/FLINK-3442 > Project: Flink > Issue Type: Improvement > Components: Streaming, Webfrontend >Reporter: Gyula Fora >Priority: Minor > > Similarly to Cancel there should be a Savepoint button to initiate a > savepoint for streaming jobs. > These 2 buttons should NOT be next to each other :) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3515) Make the "file monitoring source" exactly-once
Stephan Ewen created FLINK-3515: --- Summary: Make the "file monitoring source" exactly-once Key: FLINK-3515 URL: https://issues.apache.org/jira/browse/FLINK-3515 Project: Flink Issue Type: Improvement Components: Streaming Affects Versions: 0.10.2 Reporter: Stephan Ewen The stream source that watches directories for changes is currently not "exactly-once". To make it exactly once, the source (that generates files to be read) and the flatMap (that reads the files) need to keep track of where they were at the point of a checkpoint. Assuming that files do not change after creation (HDFS / S3 style), we can make this the following way: - The source can track the files it already emitted downstream via file creation/modification timestamp, assuming that new files always get newer timestamps. - The flatMappers need to always store the path of their current file fragment, plus the byte offset where they were within that file split. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3432) ZookeeperOffsetHandlerTest fails on windows
[ https://issues.apache.org/jira/browse/FLINK-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167557#comment-15167557 ] ASF GitHub Bot commented on FLINK-3432: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1664#issuecomment-188911039 Looks good to my, +1 to merge > ZookeeperOffsetHandlerTest fails on windows > --- > > Key: FLINK-3432 > URL: https://issues.apache.org/jira/browse/FLINK-3432 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.0.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler > Fix For: 1.0.0 > > > org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandlerTest > Time elapsed: 0.351 sec <<< FAILURE! > java.lang.AssertionError: cannot create kafka temp dir > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.prepare(KafkaTestEnvironmentImpl.java:139) > at > org.apache.flink.streaming.connectors.kafka.KafkaTestBase.prepare(KafkaTestBase.java:97) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:483) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) > org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandlerTest > Time elapsed: 0.352 sec <<< ERROR! > java.lang.NullPointerException: null > at > org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.shutdown(KafkaTestEnvironmentImpl.java:187) > at > org.apache.flink.streaming.connectors.kafka.KafkaTestBase.shutDownServices(KafkaTestBase.java:127) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:483) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3432] Fixed ZookeeperOffsethandlerTest ...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1664#issuecomment-188911039 Looks good to my, +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-3514) Add support for slowly changing streaming broadcast variables
Stephan Ewen created FLINK-3514: --- Summary: Add support for slowly changing streaming broadcast variables Key: FLINK-3514 URL: https://issues.apache.org/jira/browse/FLINK-3514 Project: Flink Issue Type: New Feature Components: Streaming Reporter: Stephan Ewen With "slow changing inputs", I mean auxiliary data sets that change over time. Think about dictionaries, set of rules, etc that are updates at slow rates and then applied onto another stream. They are a bit like broadcast variables in the DataSet API, but not constant over the entire lifetime of the job. One can currently approximate this via connected streams: {code} DataStream rules = ... DataStream events = ... rules.broadcast().connect(events).flatMap(...); {code} The disadvantage is that one has no control that the broadcasted "rules" input is materialized in the function before the "events" are delivered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3511) Flink library examples not runnable withput adding dependencies
[ https://issues.apache.org/jira/browse/FLINK-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167506#comment-15167506 ] Greg Hogan commented on FLINK-3511: --- Is the only difference from {{flink-examples-batch}} that {{flink-gelly-examples}} would need to either include Gelly in the single or individual jars, or instruct the user to copy or download a Gelly jar into the lib folder? > Flink library examples not runnable withput adding dependencies > --- > > Key: FLINK-3511 > URL: https://issues.apache.org/jira/browse/FLINK-3511 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.0.0 >Reporter: Márton Balassi > > Recent changes to the build [1] where many libraries got their core > dependencies (the ones included in the flink-dist fat jar) moved to the > provided scope. > The reasoning was that when submitting to the Flink cluster the application > already has these dependencies, while when a user writes a program against > these libraries she will include the core dependencies explicitly anyway. > There is one other case of usage however, namely when someone is trying to > run an application defined in these libraries depending on the core jars. To > give an example if you were to run the Gelly ConnectedComponents example [2] > from an IDE after importing Flink (or running with java -jar without > including the flink fat jar in the classpath) you would receive the following > class not found exception as per the current master: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/flink/api/common/ProgramDescription > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122) > Where the class missing in runtime is in flink-core, a transitive dependency > of the jars moved to the provided scope. > Funny thing is we have tests in place to run our examples, but those add test > scope dependencies re-adding the missing classes, so it is never discovered. > I agree with the original purpose of PR #1683, but also think that the > current state makes for very inconvenient user experience. > I would like to open a discussion on how and when to resolve the issue given > the release of 1.0.0. > 1. Is it a release blocker? > 2. Should the change be reverted or is it sufficient to have proper > documentation around it? Maybe a maven profile for explicitly for developing > Flink without the provided scope? > Note that the issue was originally reported by Gábor Gévay. > [1] https://github.com/apache/flink/pull/1683 > [2] > https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3509) Update Hadoop versions in release script and on travis to the latest minor version
[ https://issues.apache.org/jira/browse/FLINK-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167481#comment-15167481 ] ASF GitHub Bot commented on FLINK-3509: --- GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/1713 [FLINK-3509] Update Hadoop minor versions to latest & sync travis and release scripts You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink3509 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1713.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1713 commit 45344e162f432003c2aded4cbae2841e7c2e4476 Author: Robert Metzger Date: 2016-02-25T15:30:56Z [FLINK-3509] Update Hadoop minor versions to latest & sync travis and release scripts > Update Hadoop versions in release script and on travis to the latest minor > version > -- > > Key: FLINK-3509 > URL: https://issues.apache.org/jira/browse/FLINK-3509 > Project: Flink > Issue Type: Improvement > Components: Build System >Reporter: Robert Metzger >Assignee: Robert Metzger > > The Hadoop versions in the travis.yml and the create-release-files.sh script > are not updated to the latest Hadoop minor releases. Hadoop 2.7.2 and 2.6.4 > were released in the meantime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3509] Update Hadoop minor versions to l...
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/1713 [FLINK-3509] Update Hadoop minor versions to latest & sync travis and release scripts You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink3509 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1713.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1713 commit 45344e162f432003c2aded4cbae2841e7c2e4476 Author: Robert Metzger Date: 2016-02-25T15:30:56Z [FLINK-3509] Update Hadoop minor versions to latest & sync travis and release scripts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3511) Flink library examples not runnable withput adding dependencies
[ https://issues.apache.org/jira/browse/FLINK-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167446#comment-15167446 ] Márton Balassi commented on FLINK-3511: --- Somewhat sensible for me, but it just propagates the problem to flink-gelly-examples then, doesn't? > Flink library examples not runnable withput adding dependencies > --- > > Key: FLINK-3511 > URL: https://issues.apache.org/jira/browse/FLINK-3511 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.0.0 >Reporter: Márton Balassi > > Recent changes to the build [1] where many libraries got their core > dependencies (the ones included in the flink-dist fat jar) moved to the > provided scope. > The reasoning was that when submitting to the Flink cluster the application > already has these dependencies, while when a user writes a program against > these libraries she will include the core dependencies explicitly anyway. > There is one other case of usage however, namely when someone is trying to > run an application defined in these libraries depending on the core jars. To > give an example if you were to run the Gelly ConnectedComponents example [2] > from an IDE after importing Flink (or running with java -jar without > including the flink fat jar in the classpath) you would receive the following > class not found exception as per the current master: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/flink/api/common/ProgramDescription > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122) > Where the class missing in runtime is in flink-core, a transitive dependency > of the jars moved to the provided scope. > Funny thing is we have tests in place to run our examples, but those add test > scope dependencies re-adding the missing classes, so it is never discovered. > I agree with the original purpose of PR #1683, but also think that the > current state makes for very inconvenient user experience. > I would like to open a discussion on how and when to resolve the issue given > the release of 1.0.0. > 1. Is it a release blocker? > 2. Should the change be reverted or is it sufficient to have proper > documentation around it? Maybe a maven profile for explicitly for developing > Flink without the provided scope? > Note that the issue was originally reported by Gábor Gévay. > [1] https://github.com/apache/flink/pull/1683 > [2] > https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLINK-3511) Flink library examples not runnable withput adding dependencies
[ https://issues.apache.org/jira/browse/FLINK-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167446#comment-15167446 ] Márton Balassi edited comment on FLINK-3511 at 2/25/16 5:03 PM: Somewhat sensible for me, but it just propagates the problem to flink-gelly-examples then, doesn't it? was (Author: mbalassi): Somewhat sensible for me, but it just propagates the problem to flink-gelly-examples then, doesn't? > Flink library examples not runnable withput adding dependencies > --- > > Key: FLINK-3511 > URL: https://issues.apache.org/jira/browse/FLINK-3511 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.0.0 >Reporter: Márton Balassi > > Recent changes to the build [1] where many libraries got their core > dependencies (the ones included in the flink-dist fat jar) moved to the > provided scope. > The reasoning was that when submitting to the Flink cluster the application > already has these dependencies, while when a user writes a program against > these libraries she will include the core dependencies explicitly anyway. > There is one other case of usage however, namely when someone is trying to > run an application defined in these libraries depending on the core jars. To > give an example if you were to run the Gelly ConnectedComponents example [2] > from an IDE after importing Flink (or running with java -jar without > including the flink fat jar in the classpath) you would receive the following > class not found exception as per the current master: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/flink/api/common/ProgramDescription > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122) > Where the class missing in runtime is in flink-core, a transitive dependency > of the jars moved to the provided scope. > Funny thing is we have tests in place to run our examples, but those add test > scope dependencies re-adding the missing classes, so it is never discovered. > I agree with the original purpose of PR #1683, but also think that the > current state makes for very inconvenient user experience. > I would like to open a discussion on how and when to resolve the issue given > the release of 1.0.0. > 1. Is it a release blocker? > 2. Should the change be reverted or is it sufficient to have proper > documentation around it? Maybe a maven profile for explicitly for developing > Flink without the provided scope? > Note that the issue was originally reported by Gábor Gévay. > [1] https://github.com/apache/flink/pull/1683 > [2] > https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3463] implement calc translation
Github user vasia commented on the pull request: https://github.com/apache/flink/pull/1696#issuecomment-188878585 Thanks! I'll merge both once Travis passes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3463) Implement Calc Support
[ https://issues.apache.org/jira/browse/FLINK-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167443#comment-15167443 ] ASF GitHub Bot commented on FLINK-3463: --- Github user vasia commented on the pull request: https://github.com/apache/flink/pull/1696#issuecomment-188878585 Thanks! I'll merge both once Travis passes. > Implement Calc Support > -- > > Key: FLINK-3463 > URL: https://issues.apache.org/jira/browse/FLINK-3463 > Project: Flink > Issue Type: Sub-task > Components: Table API >Reporter: Vasia Kalavri >Assignee: Vasia Kalavri > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3511) Flink library examples not runnable withput adding dependencies
[ https://issues.apache.org/jira/browse/FLINK-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167429#comment-15167429 ] Greg Hogan commented on FLINK-3511: --- Then a flink-gelly-examples? > Flink library examples not runnable withput adding dependencies > --- > > Key: FLINK-3511 > URL: https://issues.apache.org/jira/browse/FLINK-3511 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.0.0 >Reporter: Márton Balassi > > Recent changes to the build [1] where many libraries got their core > dependencies (the ones included in the flink-dist fat jar) moved to the > provided scope. > The reasoning was that when submitting to the Flink cluster the application > already has these dependencies, while when a user writes a program against > these libraries she will include the core dependencies explicitly anyway. > There is one other case of usage however, namely when someone is trying to > run an application defined in these libraries depending on the core jars. To > give an example if you were to run the Gelly ConnectedComponents example [2] > from an IDE after importing Flink (or running with java -jar without > including the flink fat jar in the classpath) you would receive the following > class not found exception as per the current master: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/flink/api/common/ProgramDescription > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122) > Where the class missing in runtime is in flink-core, a transitive dependency > of the jars moved to the provided scope. > Funny thing is we have tests in place to run our examples, but those add test > scope dependencies re-adding the missing classes, so it is never discovered. > I agree with the original purpose of PR #1683, but also think that the > current state makes for very inconvenient user experience. > I would like to open a discussion on how and when to resolve the issue given > the release of 1.0.0. > 1. Is it a release blocker? > 2. Should the change be reverted or is it sufficient to have proper > documentation around it? Maybe a maven profile for explicitly for developing > Flink without the provided scope? > Note that the issue was originally reported by Gábor Gévay. > [1] https://github.com/apache/flink/pull/1683 > [2] > https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3399) Count with timeout trigger
[ https://issues.apache.org/jira/browse/FLINK-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167426#comment-15167426 ] ASF GitHub Bot commented on FLINK-3399: --- Github user shikhar closed the pull request at: https://github.com/apache/flink/pull/1636 > Count with timeout trigger > -- > > Key: FLINK-3399 > URL: https://issues.apache.org/jira/browse/FLINK-3399 > Project: Flink > Issue Type: Improvement >Reporter: Shikhar Bhushan >Priority: Minor > > I created an implementation of a trigger that I'd like to contribute, > https://gist.github.com/shikhar/2cb9f1b792be31b7c16e > An example application - if a sink function operates more efficiently if it > is writing in a batched fashion, then the windowing mechanism + this trigger > can be used. Count to have an upper bound on batch size & better control on > memory usage, and timeout to ensure timeliness of the outputs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3399] CountWithTimeoutTrigger
Github user shikhar closed the pull request at: https://github.com/apache/flink/pull/1636 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3399) Count with timeout trigger
[ https://issues.apache.org/jira/browse/FLINK-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167425#comment-15167425 ] ASF GitHub Bot commented on FLINK-3399: --- Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1636#issuecomment-188874847 I would prefer to just add the compassable way, yes. Sorry that you now already put effort into this. > Count with timeout trigger > -- > > Key: FLINK-3399 > URL: https://issues.apache.org/jira/browse/FLINK-3399 > Project: Flink > Issue Type: Improvement >Reporter: Shikhar Bhushan >Priority: Minor > > I created an implementation of a trigger that I'd like to contribute, > https://gist.github.com/shikhar/2cb9f1b792be31b7c16e > An example application - if a sink function operates more efficiently if it > is writing in a batched fashion, then the windowing mechanism + this trigger > can be used. Count to have an upper bound on batch size & better control on > memory usage, and timeout to ensure timeliness of the outputs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3399] CountWithTimeoutTrigger
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1636#issuecomment-188874847 I would prefer to just add the compassable way, yes. Sorry that you now already put effort into this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3399) Count with timeout trigger
[ https://issues.apache.org/jira/browse/FLINK-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167417#comment-15167417 ] ASF GitHub Bot commented on FLINK-3399: --- Github user shikhar commented on the pull request: https://github.com/apache/flink/pull/1636#issuecomment-188874026 Thanks @StephanEwen, @aljoscha noted on the JIRA that it may be better to create a composable mechanism like `Any.of(,)` Until that exists, I do think this is useful functionality, but let me know if we'd prefer to just have the composable way and I should close this PR. > Count with timeout trigger > -- > > Key: FLINK-3399 > URL: https://issues.apache.org/jira/browse/FLINK-3399 > Project: Flink > Issue Type: Improvement >Reporter: Shikhar Bhushan >Priority: Minor > > I created an implementation of a trigger that I'd like to contribute, > https://gist.github.com/shikhar/2cb9f1b792be31b7c16e > An example application - if a sink function operates more efficiently if it > is writing in a batched fashion, then the windowing mechanism + this trigger > can be used. Count to have an upper bound on batch size & better control on > memory usage, and timeout to ensure timeliness of the outputs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3399] CountWithTimeoutTrigger
Github user shikhar commented on the pull request: https://github.com/apache/flink/pull/1636#issuecomment-188874026 Thanks @StephanEwen, @aljoscha noted on the JIRA that it may be better to create a composable mechanism like `Any.of(,)` Until that exists, I do think this is useful functionality, but let me know if we'd prefer to just have the composable way and I should close this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3322) MemoryManager creates too much GC pressure with iterative jobs
[ https://issues.apache.org/jira/browse/FLINK-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167415#comment-15167415 ] Gabor Gevay commented on FLINK-3322: It's making it even worse: false, 500m: 30s false, 5000m: 115s true, 500m: 8s true, 5000m: 10s > MemoryManager creates too much GC pressure with iterative jobs > -- > > Key: FLINK-3322 > URL: https://issues.apache.org/jira/browse/FLINK-3322 > Project: Flink > Issue Type: Bug > Components: Distributed Runtime >Affects Versions: 1.0.0 >Reporter: Gabor Gevay >Priority: Critical > Fix For: 1.0.0 > > > When taskmanager.memory.preallocate is false (the default), released memory > segments are not added to a pool, but the GC is expected to take care of > them. This puts too much pressure on the GC with iterative jobs, where the > operators reallocate all memory at every superstep. > See the following discussion on the mailing list: > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Memory-manager-behavior-in-iterative-jobs-tt10066.html > Reproducing the issue: > https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc > The class to start is malom.Solver. If you increase the memory given to the > JVM from 1 to 50 GB, performance gradually degrades by more than 10 times. > (It will generate some lookuptables to /tmp on first run for a few minutes.) > (I think the slowdown might also depend somewhat on > taskmanager.memory.fraction, because more unused non-managed memory results > in rarer GCs.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3511) Flink library examples not runnable withput adding dependencies
[ https://issues.apache.org/jira/browse/FLINK-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167416#comment-15167416 ] Márton Balassi commented on FLINK-3511: --- I am against moving the examples of a library out of the library. It would also mean that flink-examples-batch would need to depend on gelly, which would add gelly automatically as a dependency to flink-dist, something that we would like to avoid as long as it is beta. And even if we did, it would only heal one single symptom instead of curing the underlying problem. With the current solution any example in any library is a potential victim of this issue. > Flink library examples not runnable withput adding dependencies > --- > > Key: FLINK-3511 > URL: https://issues.apache.org/jira/browse/FLINK-3511 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.0.0 >Reporter: Márton Balassi > > Recent changes to the build [1] where many libraries got their core > dependencies (the ones included in the flink-dist fat jar) moved to the > provided scope. > The reasoning was that when submitting to the Flink cluster the application > already has these dependencies, while when a user writes a program against > these libraries she will include the core dependencies explicitly anyway. > There is one other case of usage however, namely when someone is trying to > run an application defined in these libraries depending on the core jars. To > give an example if you were to run the Gelly ConnectedComponents example [2] > from an IDE after importing Flink (or running with java -jar without > including the flink fat jar in the classpath) you would receive the following > class not found exception as per the current master: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/flink/api/common/ProgramDescription > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122) > Where the class missing in runtime is in flink-core, a transitive dependency > of the jars moved to the provided scope. > Funny thing is we have tests in place to run our examples, but those add test > scope dependencies re-adding the missing classes, so it is never discovered. > I agree with the original purpose of PR #1683, but also think that the > current state makes for very inconvenient user experience. > I would like to open a discussion on how and when to resolve the issue given > the release of 1.0.0. > 1. Is it a release blocker? > 2. Should the change be reverted or is it sufficient to have proper > documentation around it? Maybe a maven profile for explicitly for developing > Flink without the provided scope? > Note that the issue was originally reported by Gábor Gévay. > [1] https://github.com/apache/flink/pull/1683 > [2] > https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3399) Count with timeout trigger
[ https://issues.apache.org/jira/browse/FLINK-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167413#comment-15167413 ] ASF GitHub Bot commented on FLINK-3399: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1636#issuecomment-188872638 Sorry for the late response, most people are fixing issues for the 1.0 release candidates. This actually looks like a nice addition. +1 for adding a small library/collection of typical triggers and patterns. Let's merge this into master once the 1.0 release is out, to be released with either 1.1.0 or 1.0.1. > Count with timeout trigger > -- > > Key: FLINK-3399 > URL: https://issues.apache.org/jira/browse/FLINK-3399 > Project: Flink > Issue Type: Improvement >Reporter: Shikhar Bhushan >Priority: Minor > > I created an implementation of a trigger that I'd like to contribute, > https://gist.github.com/shikhar/2cb9f1b792be31b7c16e > An example application - if a sink function operates more efficiently if it > is writing in a batched fashion, then the windowing mechanism + this trigger > can be used. Count to have an upper bound on batch size & better control on > memory usage, and timeout to ensure timeliness of the outputs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3399] CountWithTimeoutTrigger
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1636#issuecomment-188872638 Sorry for the late response, most people are fixing issues for the 1.0 release candidates. This actually looks like a nice addition. +1 for adding a small library/collection of typical triggers and patterns. Let's merge this into master once the 1.0 release is out, to be released with either 1.1.0 or 1.0.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3511) Flink library examples not runnable withput adding dependencies
[ https://issues.apache.org/jira/browse/FLINK-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167404#comment-15167404 ] Greg Hogan commented on FLINK-3511: --- Could the Gelly examples be moved to flink-examples-batch? > Flink library examples not runnable withput adding dependencies > --- > > Key: FLINK-3511 > URL: https://issues.apache.org/jira/browse/FLINK-3511 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.0.0 >Reporter: Márton Balassi > > Recent changes to the build [1] where many libraries got their core > dependencies (the ones included in the flink-dist fat jar) moved to the > provided scope. > The reasoning was that when submitting to the Flink cluster the application > already has these dependencies, while when a user writes a program against > these libraries she will include the core dependencies explicitly anyway. > There is one other case of usage however, namely when someone is trying to > run an application defined in these libraries depending on the core jars. To > give an example if you were to run the Gelly ConnectedComponents example [2] > from an IDE after importing Flink (or running with java -jar without > including the flink fat jar in the classpath) you would receive the following > class not found exception as per the current master: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/flink/api/common/ProgramDescription > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122) > Where the class missing in runtime is in flink-core, a transitive dependency > of the jars moved to the provided scope. > Funny thing is we have tests in place to run our examples, but those add test > scope dependencies re-adding the missing classes, so it is never discovered. > I agree with the original purpose of PR #1683, but also think that the > current state makes for very inconvenient user experience. > I would like to open a discussion on how and when to resolve the issue given > the release of 1.0.0. > 1. Is it a release blocker? > 2. Should the change be reverted or is it sufficient to have proper > documentation around it? Maybe a maven profile for explicitly for developing > Flink without the provided scope? > Note that the issue was originally reported by Gábor Gévay. > [1] https://github.com/apache/flink/pull/1683 > [2] > https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3322) MemoryManager creates too much GC pressure with iterative jobs
[ https://issues.apache.org/jira/browse/FLINK-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167402#comment-15167402 ] Stephan Ewen commented on FLINK-3322: - How does the off-heap memory setting affect these stats? Do the numbers look similar, or does pre-allocation not make that much of a difference there? > MemoryManager creates too much GC pressure with iterative jobs > -- > > Key: FLINK-3322 > URL: https://issues.apache.org/jira/browse/FLINK-3322 > Project: Flink > Issue Type: Bug > Components: Distributed Runtime >Affects Versions: 1.0.0 >Reporter: Gabor Gevay >Priority: Critical > Fix For: 1.0.0 > > > When taskmanager.memory.preallocate is false (the default), released memory > segments are not added to a pool, but the GC is expected to take care of > them. This puts too much pressure on the GC with iterative jobs, where the > operators reallocate all memory at every superstep. > See the following discussion on the mailing list: > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Memory-manager-behavior-in-iterative-jobs-tt10066.html > Reproducing the issue: > https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc > The class to start is malom.Solver. If you increase the memory given to the > JVM from 1 to 50 GB, performance gradually degrades by more than 10 times. > (It will generate some lookuptables to /tmp on first run for a few minutes.) > (I think the slowdown might also depend somewhat on > taskmanager.memory.fraction, because more unused non-managed memory results > in rarer GCs.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator
[ https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167393#comment-15167393 ] ASF GitHub Bot commented on FLINK-3513: --- Github user uce commented on the pull request: https://github.com/apache/flink/pull/1712#issuecomment-188862999 `mvn clean verify` was successful on my local machine, so if we need this for a new RC, we can merge it before the Travis run finishes. > Fix interplay of automatic Operator UID and Changing name of WindowOperator > --- > > Key: FLINK-3513 > URL: https://issues.apache.org/jira/browse/FLINK-3513 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.0.0 >Reporter: Aljoscha Krettek >Assignee: Ufuk Celebi >Priority: Blocker > > WindowOperator can have a changing name because it has the TypeSerializer > .toString() output in it's name. For some type serializers that don't > implement toString() this means that the name changes. > This means that savepoint restore does not work for the automatically > generated UID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3513] [FLINK-3512] Fix savepoint issues
Github user uce commented on the pull request: https://github.com/apache/flink/pull/1712#issuecomment-188862999 `mvn clean verify` was successful on my local machine, so if we need this for a new RC, we can merge it before the Travis run finishes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3322) MemoryManager creates too much GC pressure with iterative jobs
[ https://issues.apache.org/jira/browse/FLINK-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167377#comment-15167377 ] Gabor Gevay commented on FLINK-3322: I have constructed a much simpler example to demonstrate the problem: ConnectedComponents on a graph that is an 1000 length path: 1->2, 2->3, 3->4, 4->5, ... 999->1000: https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc-2 The class to run is org.apache.flink.graph.example.ConnectedComponents. Try increasing the memory from eg. 500m to 5000m, and look at the difference when taskmanager.memory.preallocate is true and false (TaskManager.scala:1713). I measured the following times on my laptop: false, 500m: 14s false, 5000m: 115s true, 500m: 8s true, 5000m: 13s (I guess that the difference between the two runs where preallocate is true is due to the time it takes for the JVM to allocate the memory once, but this should also be checked that it isn't for some other unexpected reason.) So the bottom line is that the problem gets worse when there are more iterations. (We have 1001 iterations in the linked example.) > MemoryManager creates too much GC pressure with iterative jobs > -- > > Key: FLINK-3322 > URL: https://issues.apache.org/jira/browse/FLINK-3322 > Project: Flink > Issue Type: Bug > Components: Distributed Runtime >Affects Versions: 1.0.0 >Reporter: Gabor Gevay >Priority: Critical > Fix For: 1.0.0 > > > When taskmanager.memory.preallocate is false (the default), released memory > segments are not added to a pool, but the GC is expected to take care of > them. This puts too much pressure on the GC with iterative jobs, where the > operators reallocate all memory at every superstep. > See the following discussion on the mailing list: > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Memory-manager-behavior-in-iterative-jobs-tt10066.html > Reproducing the issue: > https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc > The class to start is malom.Solver. If you increase the memory given to the > JVM from 1 to 50 GB, performance gradually degrades by more than 10 times. > (It will generate some lookuptables to /tmp on first run for a few minutes.) > (I think the slowdown might also depend somewhat on > taskmanager.memory.fraction, because more unused non-managed memory results > in rarer GCs.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator
[ https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167373#comment-15167373 ] ASF GitHub Bot commented on FLINK-3513: --- Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1712#issuecomment-188858495 :+1: > Fix interplay of automatic Operator UID and Changing name of WindowOperator > --- > > Key: FLINK-3513 > URL: https://issues.apache.org/jira/browse/FLINK-3513 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.0.0 >Reporter: Aljoscha Krettek >Assignee: Ufuk Celebi >Priority: Blocker > > WindowOperator can have a changing name because it has the TypeSerializer > .toString() output in it's name. For some type serializers that don't > implement toString() this means that the name changes. > This means that savepoint restore does not work for the automatically > generated UID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3513] [FLINK-3512] Fix savepoint issues
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1712#issuecomment-188858495 :+1: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3501) flink-runtime-0.10.2-hadoop-1 jar problem
[ https://issues.apache.org/jira/browse/FLINK-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167351#comment-15167351 ] Robert Metzger commented on FLINK-3501: --- I'm not able to reproduce the issue: {code} 1 robert@robert-da /tmp/test % java -version openjdk version "1.8.0_74" OpenJDK Runtime Environment (build 1.8.0_74-b02) OpenJDK 64-Bit Server VM (build 25.74-b02, mixed mode) robert@robert-da /tmp/test % jar xf flink-runtime-0.10.2-hadoop1.jar robert@robert-da /tmp/test % {code} It could be that you are using a different JDK which has that issue. What's your JDK? > flink-runtime-0.10.2-hadoop-1 jar problem > - > > Key: FLINK-3501 > URL: https://issues.apache.org/jira/browse/FLINK-3501 > Project: Flink > Issue Type: Bug > Components: Distributed Runtime >Affects Versions: 0.10.2 >Reporter: Paul Harris > > Unjaring flink-runtime-0.10.2-hadoop-1 results in an error. > Steps to reproduce > {code:none} > jar xf flink-runtime-0.10.2-hadoop1.jar > {code} > Result... > {code:none} > java.io.IOException: META-INF/license : could not create directory > at sun.tools.jar.Main.extractFile(Main.java:934) > at sun.tools.jar.Main.extract(Main.java:877) > at sun.tools.jar.Main.run(Main.java:263) > at sun.tools.jar.Main.main(Main.java:1177) > {code} > This is the result of the jar containing a file 'license' and a directory > license'. The file is unpacked first, preventing the directory being unpacked > to the same name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator
[ https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167350#comment-15167350 ] ASF GitHub Bot commented on FLINK-3513: --- GitHub user uce opened a pull request: https://github.com/apache/flink/pull/1712 [FLINK-3513] [FLINK-3512] Fix savepoint issues @aljoscha spotted the following two issues (so far) while working with savepoints and the rocks DB state backend. - FLINK-3513: The operator name is used in the deterministic hash assignment, which is actually not necessary and leads to problems with the WindowOperator (changing name). - FLINK-3512: The savepoint backend was trying to be too clever and fell back to jobmanager when it found "wrong" config combinations. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uce/flink hash_gen Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1712.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1712 commit dab3c654fafecf8caee6453b7da8965174646771 Author: Ufuk Celebi Date: 2016-02-25T15:55:04Z [FLINK-3513] [runtime] Fix interplay of automatic Operator UID and Changing name of WindowOperator commit 3c3a81acee8de1128a3eee7f59ca6f8803acbd5c Author: Ufuk Celebi Date: 2016-02-25T15:55:38Z [FLINK-3512] [runtime] Savepoint backend should not revert to 'jobmanager' > Fix interplay of automatic Operator UID and Changing name of WindowOperator > --- > > Key: FLINK-3513 > URL: https://issues.apache.org/jira/browse/FLINK-3513 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.0.0 >Reporter: Aljoscha Krettek >Assignee: Ufuk Celebi >Priority: Blocker > > WindowOperator can have a changing name because it has the TypeSerializer > .toString() output in it's name. For some type serializers that don't > implement toString() this means that the name changes. > This means that savepoint restore does not work for the automatically > generated UID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3513] [FLINK-3512] Fix savepoint issues
GitHub user uce opened a pull request: https://github.com/apache/flink/pull/1712 [FLINK-3513] [FLINK-3512] Fix savepoint issues @aljoscha spotted the following two issues (so far) while working with savepoints and the rocks DB state backend. - FLINK-3513: The operator name is used in the deterministic hash assignment, which is actually not necessary and leads to problems with the WindowOperator (changing name). - FLINK-3512: The savepoint backend was trying to be too clever and fell back to jobmanager when it found "wrong" config combinations. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uce/flink hash_gen Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1712.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1712 commit dab3c654fafecf8caee6453b7da8965174646771 Author: Ufuk Celebi Date: 2016-02-25T15:55:04Z [FLINK-3513] [runtime] Fix interplay of automatic Operator UID and Changing name of WindowOperator commit 3c3a81acee8de1128a3eee7f59ca6f8803acbd5c Author: Ufuk Celebi Date: 2016-02-25T15:55:38Z [FLINK-3512] [runtime] Savepoint backend should not revert to 'jobmanager' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-3512) Savepoint backend should not revert to "jobmanager"
Aljoscha Krettek created FLINK-3512: --- Summary: Savepoint backend should not revert to "jobmanager" Key: FLINK-3512 URL: https://issues.apache.org/jira/browse/FLINK-3512 Project: Flink Issue Type: Bug Reporter: Aljoscha Krettek Assignee: Ufuk Celebi Priority: Blocker When not setting a state backend in the config the savepoint coordinator defaults to use "jobmanager" as savepoint backend, ignoring the setting for the savepoint backend in the config. This means that when setting a StateBackend only per job the system always falls back to "jobmanager" even though it could use "filesystem" to make savepoints persistent. See SavepointStoreFactory.java:71 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator
[ https://issues.apache.org/jira/browse/FLINK-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-3513: Assignee: Ufuk Celebi > Fix interplay of automatic Operator UID and Changing name of WindowOperator > --- > > Key: FLINK-3513 > URL: https://issues.apache.org/jira/browse/FLINK-3513 > Project: Flink > Issue Type: Bug > Components: Streaming >Affects Versions: 1.0.0 >Reporter: Aljoscha Krettek >Assignee: Ufuk Celebi >Priority: Blocker > > WindowOperator can have a changing name because it has the TypeSerializer > .toString() output in it's name. For some type serializers that don't > implement toString() this means that the name changes. > This means that savepoint restore does not work for the automatically > generated UID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3513) Fix interplay of automatic Operator UID and Changing name of WindowOperator
Aljoscha Krettek created FLINK-3513: --- Summary: Fix interplay of automatic Operator UID and Changing name of WindowOperator Key: FLINK-3513 URL: https://issues.apache.org/jira/browse/FLINK-3513 Project: Flink Issue Type: Bug Components: Streaming Affects Versions: 1.0.0 Reporter: Aljoscha Krettek Priority: Blocker WindowOperator can have a changing name because it has the TypeSerializer .toString() output in it's name. For some type serializers that don't implement toString() this means that the name changes. This means that savepoint restore does not work for the automatically generated UID. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3463) Implement Calc Support
[ https://issues.apache.org/jira/browse/FLINK-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167330#comment-15167330 ] ASF GitHub Bot commented on FLINK-3463: --- Github user twalthr commented on the pull request: https://github.com/apache/flink/pull/1696#issuecomment-188846370 +1 for merging > Implement Calc Support > -- > > Key: FLINK-3463 > URL: https://issues.apache.org/jira/browse/FLINK-3463 > Project: Flink > Issue Type: Sub-task > Components: Table API >Reporter: Vasia Kalavri >Assignee: Vasia Kalavri > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3463] implement calc translation
Github user twalthr commented on the pull request: https://github.com/apache/flink/pull/1696#issuecomment-188846370 +1 for merging --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3496) Flink-ml tests fail on Windows
[ https://issues.apache.org/jira/browse/FLINK-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167319#comment-15167319 ] ASF GitHub Bot commented on FLINK-3496: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/1711 git push -f[FLINK-3496] Disable RocksDB tests on Windows You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 3496_rocksdb_disable Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1711.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1711 commit a7b965288f4a913bf0f7c37e3f7c88748e3f124a Author: zentol Date: 2016-02-25T13:31:34Z [FLINK-3496] Disable RocksDB tests on Windows > Flink-ml tests fail on Windows > -- > > Key: FLINK-3496 > URL: https://issues.apache.org/jira/browse/FLINK-3496 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.0.0 >Reporter: Chesnay Schepler > > [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ flink-ml_2.10 > --- > [INFO] > [INFO] --- scalatest-maven-plugin:1.0:test (scala-test) @ flink-ml_2.10 --- > The system cannot find the file specified. > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 01:03 min > [INFO] Finished at: 2016-02-24T12:47:23+01:00 > [INFO] Final Memory: 28M/506M -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: git push -f[FLINK-3496] Disable RocksDB tests ...
GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/1711 git push -f[FLINK-3496] Disable RocksDB tests on Windows You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 3496_rocksdb_disable Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1711.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1711 commit a7b965288f4a913bf0f7c37e3f7c88748e3f124a Author: zentol Date: 2016-02-25T13:31:34Z [FLINK-3496] Disable RocksDB tests on Windows --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-3496] Fix ML test discovery on Windows
GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/1710 [FLINK-3496] Fix ML test discovery on Windows This PR adds OS specific arguments in the flink-ml pom that are passed to the scala test runner. The runner uses pattern matching to discover test files that should be run. The original pattern without quotes works on unix but fails on windows, with quotes it works on windows but skip all tests on unix. As such I've added 2 profiles that activate based on the OS and set a pattern property, which is then passed to the runner. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 3496_ml_pattern Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1710.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1710 commit efc62c42879508450544ca72d464420bf3934724 Author: zentol Date: 2016-02-25T15:09:42Z [FLINK-3496] Fix ML test discovery on Windows --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3508) Add more test cases to verify the rules of logical plan optimization
[ https://issues.apache.org/jira/browse/FLINK-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167301#comment-15167301 ] Chengxiang Li commented on FLINK-3508: -- Merged to tableOnCalcite branch at 72686231fd8f9fa6a1c05df48f6f29eaa3ca4f2b. > Add more test cases to verify the rules of logical plan optimization > > > Key: FLINK-3508 > URL: https://issues.apache.org/jira/browse/FLINK-3508 > Project: Flink > Issue Type: Improvement > Components: Table API >Reporter: Chengxiang Li >Assignee: Chengxiang Li >Priority: Minor > > We have enabled many rules in logical plan optimization phase, more > complicated test cases should be added to verify whether these rules actally > work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-3508) Add more test cases to verify the rules of logical plan optimization
[ https://issues.apache.org/jira/browse/FLINK-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li closed FLINK-3508. > Add more test cases to verify the rules of logical plan optimization > > > Key: FLINK-3508 > URL: https://issues.apache.org/jira/browse/FLINK-3508 > Project: Flink > Issue Type: Improvement > Components: Table API >Reporter: Chengxiang Li >Assignee: Chengxiang Li >Priority: Minor > > We have enabled many rules in logical plan optimization phase, more > complicated test cases should be added to verify whether these rules actally > work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-3508) Add more test cases to verify the rules of logical plan optimization
[ https://issues.apache.org/jira/browse/FLINK-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li resolved FLINK-3508. -- Resolution: Fixed > Add more test cases to verify the rules of logical plan optimization > > > Key: FLINK-3508 > URL: https://issues.apache.org/jira/browse/FLINK-3508 > Project: Flink > Issue Type: Improvement > Components: Table API >Reporter: Chengxiang Li >Assignee: Chengxiang Li >Priority: Minor > > We have enabled many rules in logical plan optimization phase, more > complicated test cases should be added to verify whether these rules actally > work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3496) Flink-ml tests fail on Windows
[ https://issues.apache.org/jira/browse/FLINK-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167298#comment-15167298 ] ASF GitHub Bot commented on FLINK-3496: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/1710 [FLINK-3496] Fix ML test discovery on Windows This PR adds OS specific arguments in the flink-ml pom that are passed to the scala test runner. The runner uses pattern matching to discover test files that should be run. The original pattern without quotes works on unix but fails on windows, with quotes it works on windows but skip all tests on unix. As such I've added 2 profiles that activate based on the OS and set a pattern property, which is then passed to the runner. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 3496_ml_pattern Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1710.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1710 commit efc62c42879508450544ca72d464420bf3934724 Author: zentol Date: 2016-02-25T15:09:42Z [FLINK-3496] Fix ML test discovery on Windows > Flink-ml tests fail on Windows > -- > > Key: FLINK-3496 > URL: https://issues.apache.org/jira/browse/FLINK-3496 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.0.0 >Reporter: Chesnay Schepler > > [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ flink-ml_2.10 > --- > [INFO] > [INFO] --- scalatest-maven-plugin:1.0:test (scala-test) @ flink-ml_2.10 --- > The system cannot find the file specified. > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 01:03 min > [INFO] Finished at: 2016-02-24T12:47:23+01:00 > [INFO] Final Memory: 28M/506M -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLINK-3496) Flink-ml tests fail on Windows
[ https://issues.apache.org/jira/browse/FLINK-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167281#comment-15167281 ] Chesnay Schepler edited comment on FLINK-3496 at 2/25/16 2:56 PM: -- using {code}"(? Flink-ml tests fail on Windows > -- > > Key: FLINK-3496 > URL: https://issues.apache.org/jira/browse/FLINK-3496 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.0.0 >Reporter: Chesnay Schepler > > [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ flink-ml_2.10 > --- > [INFO] > [INFO] --- scalatest-maven-plugin:1.0:test (scala-test) @ flink-ml_2.10 --- > The system cannot find the file specified. > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 01:03 min > [INFO] Finished at: 2016-02-24T12:47:23+01:00 > [INFO] Final Memory: 28M/506M -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3496) Flink-ml tests fail on Windows
[ https://issues.apache.org/jira/browse/FLINK-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167281#comment-15167281 ] Chesnay Schepler commented on FLINK-3496: - using {code}"(? Flink-ml tests fail on Windows > -- > > Key: FLINK-3496 > URL: https://issues.apache.org/jira/browse/FLINK-3496 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.0.0 >Reporter: Chesnay Schepler > > [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ flink-ml_2.10 > --- > [INFO] > [INFO] --- scalatest-maven-plugin:1.0:test (scala-test) @ flink-ml_2.10 --- > The system cannot find the file specified. > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 01:03 min > [INFO] Finished at: 2016-02-24T12:47:23+01:00 > [INFO] Final Memory: 28M/506M -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3398) Flink Kafka consumer should support auto-commit opt-outs
[ https://issues.apache.org/jira/browse/FLINK-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167277#comment-15167277 ] ASF GitHub Bot commented on FLINK-3398: --- Github user shikhar commented on the pull request: https://github.com/apache/flink/pull/1690#issuecomment-188818665 Just fixed the `Kafka09ITCase.testCheckpointing()` fail as well, hopefully CI is green with that. > Flink Kafka consumer should support auto-commit opt-outs > > > Key: FLINK-3398 > URL: https://issues.apache.org/jira/browse/FLINK-3398 > Project: Flink > Issue Type: Bug >Reporter: Shikhar Bhushan > > Currently the Kafka source will commit consumer offsets to Zookeeper, either > upon a checkpoint if checkpointing is enabled, otherwise periodically based > on {{auto.commit.interval.ms}} > It should be possible to opt-out of committing consumer offsets to Zookeeper. > Kafka has this config as {{auto.commit.enable}} (0.8) and > {{enable.auto.commit}} (0.9). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: FLINK-3398: Allow for opting-out from Kafka of...
Github user shikhar commented on the pull request: https://github.com/apache/flink/pull/1690#issuecomment-188818665 Just fixed the `Kafka09ITCase.testCheckpointing()` fail as well, hopefully CI is green with that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-3511) Flink library examples not runnable withput adding dependencies
Márton Balassi created FLINK-3511: - Summary: Flink library examples not runnable withput adding dependencies Key: FLINK-3511 URL: https://issues.apache.org/jira/browse/FLINK-3511 Project: Flink Issue Type: Bug Components: Build System Affects Versions: 1.0.0 Reporter: Márton Balassi Recent changes to the build [1] where many libraries got their core dependencies (the ones included in the flink-dist fat jar) moved to the provided scope. The reasoning was that when submitting to the Flink cluster the application already has these dependencies, while when a user writes a program against these libraries she will include the core dependencies explicitly anyway. There is one other case of usage however, namely when someone is trying to run an application defined in these libraries depending on the core jars. To give an example if you were to run the Gelly ConnectedComponents example [2] from an IDE after importing Flink (or running with java -jar without including the flink fat jar in the classpath) you would receive the following class not found exception as per the current master: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/api/common/ProgramDescription at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:760) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:122) Where the class missing in runtime is in flink-core, a transitive dependency of the jars moved to the provided scope. Funny thing is we have tests in place to run our examples, but those add test scope dependencies re-adding the missing classes, so it is never discovered. I agree with the original purpose of PR #1683, but also think that the current state makes for very inconvenient user experience. I would like to open a discussion on how and when to resolve the issue given the release of 1.0.0. 1. Is it a release blocker? 2. Should the change be reverted or is it sufficient to have proper documentation around it? Maybe a maven profile for explicitly for developing Flink without the provided scope? Note that the issue was originally reported by Gábor Gévay. [1] https://github.com/apache/flink/pull/1683 [2] https://github.com/apache/flink/blob/master/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/example/ConnectedComponents.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-3510) Pattern class class-level comment misses type argument
Ufuk Celebi created FLINK-3510: -- Summary: Pattern class class-level comment misses type argument Key: FLINK-3510 URL: https://issues.apache.org/jira/browse/FLINK-3510 Project: Flink Issue Type: Bug Components: CEP Reporter: Ufuk Celebi Assignee: Ufuk Celebi Priority: Trivial Class-level comment code example reads: {code} * {@code * Pattern pattern = Pattern.begin("start") * .next("middle").subtype(F.class) * .followedBy("end").where(new MyFilterFunction()); * } * {code} The 2nd type paramter for {{Pattern}} is missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1159) Case style anonymous functions not supported by Scala API
[ https://issues.apache.org/jira/browse/FLINK-1159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167209#comment-15167209 ] ASF GitHub Bot commented on FLINK-1159: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1704#issuecomment-188790808 @StephanEwen I had to restore some of the context bounds on `ClassTag` to make it compile, apparently the delegated methods use them; I've rebased with the latest changes on the master before putting them back in place. > Case style anonymous functions not supported by Scala API > - > > Key: FLINK-1159 > URL: https://issues.apache.org/jira/browse/FLINK-1159 > Project: Flink > Issue Type: Bug > Components: Scala API >Reporter: Till Rohrmann >Assignee: Stefano Baghino > > In Scala it is very common to define anonymous functions of the following form > {code} > { > case foo: Bar => foobar(foo) > case _ => throw new RuntimeException() > } > {code} > These case style anonymous functions are not supported yet by the Scala API. > Thus, one has to write redundant code to name the function parameter. > What works is the following pattern, but it is not intuitive for someone > coming from Scala: > {code} > dataset.map{ > _ match{ > case foo:Bar => ... > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1159] Case style anonymous functions no...
Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1704#issuecomment-188790808 @StephanEwen I had to restore some of the context bounds on `ClassTag` to make it compile, apparently the delegated methods use them; I've rebased with the latest changes on the master before putting them back in place. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3496) Flink-ml tests fail on Windows
[ https://issues.apache.org/jira/browse/FLINK-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167186#comment-15167186 ] Chesnay Schepler commented on FLINK-3496: - may a fix for this coming up, running on travis now. > Flink-ml tests fail on Windows > -- > > Key: FLINK-3496 > URL: https://issues.apache.org/jira/browse/FLINK-3496 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.0.0 >Reporter: Chesnay Schepler > > [INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ flink-ml_2.10 > --- > [INFO] > [INFO] --- scalatest-maven-plugin:1.0:test (scala-test) @ flink-ml_2.10 --- > The system cannot find the file specified. > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 01:03 min > [INFO] Finished at: 2016-02-24T12:47:23+01:00 > [INFO] Final Memory: 28M/506M -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3332] Cassandra connector
Github user spdrnl commented on a diff in the pull request: https://github.com/apache/flink/pull/1620#discussion_r54090368 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/GenericExactlyOnceSink.java --- @@ -0,0 +1,173 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.flink.streaming.runtime.operators; + +import org.apache.flink.api.common.ExecutionConfig; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.common.typeutils.TypeSerializer; +import org.apache.flink.api.java.tuple.Tuple; +import org.apache.flink.api.java.typeutils.TypeExtractor; +import org.apache.flink.core.memory.DataInputView; +import org.apache.flink.runtime.state.AbstractStateBackend; +import org.apache.flink.runtime.state.StateHandle; +import org.apache.flink.streaming.api.operators.AbstractStreamOperator; +import org.apache.flink.streaming.api.operators.OneInputStreamOperator; +import org.apache.flink.streaming.api.watermark.Watermark; +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord; +import org.apache.flink.streaming.runtime.tasks.StreamTaskState; + +import java.io.IOException; +import java.io.Serializable; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +/** + * Generic Sink that emits its input elements into an arbitrary backend. This sink is integrated with the checkpointing + * mechanism to provide exactly once semantics. + * + * Incoming records are stored within a {@link org.apache.flink.runtime.state.AbstractStateBackend}, and only committed if a + * checkpoint is completed. Should a job fail while the data is being committed, no exactly once guarantee can be made. + * @param + */ +public abstract class GenericExactlyOnceSink extends AbstractStreamOperator implements OneInputStreamOperator { + private AbstractStateBackend.CheckpointStateOutputView out; + private TypeSerializer serializer; + protected TypeInformation typeInfo; + + private ExactlyOnceState state = new ExactlyOnceState(); + + /** +* Saves a handle in the state. +* @param checkpointId +* @throws IOException +*/ + private void saveHandleInState(final long checkpointId) throws IOException { + //only add handle if a new OperatorState was created since the last snapshot/notify + if (out != null) { + out.writeByte(0); //EOF-byte + StateHandle handle = out.closeAndGetHandle(); + if (state.pendingHandles.containsKey(checkpointId)) { + state.pendingHandles.get(checkpointId).add(handle); + } else { + ArrayList> list = new ArrayList<>(); + list.add(handle); + state.pendingHandles.put(checkpointId, list); + } + out = null; + } + } + + @Override + public StreamTaskState snapshotOperatorState(final long checkpointId, final long timestamp) throws Exception { + StreamTaskState taskState = super.snapshotOperatorState(checkpointId, timestamp); + saveHandleInState(checkpointId); + taskState.setFunctionState(state); + return taskState; + } + + @Override + public void restoreState(StreamTaskState state, long recoveryTimestamp) throws Exception { + super.restoreState(state, recoveryTimestamp); + this.state = (ExactlyOnceState) state.getFunctionState(); + out = null; + } + + @Override + public void notifyOfCompletedCheckpoint(long checkpointId) throws Exception { --- End diff -- Apart from Flink specifics it is possible to get Cassandra to behave idempotent wi
[jira] [Commented] (FLINK-3332) Provide an exactly-once Cassandra connector
[ https://issues.apache.org/jira/browse/FLINK-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167166#comment-15167166 ] ASF GitHub Bot commented on FLINK-3332: --- Github user spdrnl commented on a diff in the pull request: https://github.com/apache/flink/pull/1620#discussion_r54090368 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/GenericExactlyOnceSink.java --- @@ -0,0 +1,173 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.flink.streaming.runtime.operators; + +import org.apache.flink.api.common.ExecutionConfig; +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.common.typeutils.TypeSerializer; +import org.apache.flink.api.java.tuple.Tuple; +import org.apache.flink.api.java.typeutils.TypeExtractor; +import org.apache.flink.core.memory.DataInputView; +import org.apache.flink.runtime.state.AbstractStateBackend; +import org.apache.flink.runtime.state.StateHandle; +import org.apache.flink.streaming.api.operators.AbstractStreamOperator; +import org.apache.flink.streaming.api.operators.OneInputStreamOperator; +import org.apache.flink.streaming.api.watermark.Watermark; +import org.apache.flink.streaming.runtime.streamrecord.StreamRecord; +import org.apache.flink.streaming.runtime.tasks.StreamTaskState; + +import java.io.IOException; +import java.io.Serializable; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Set; + +/** + * Generic Sink that emits its input elements into an arbitrary backend. This sink is integrated with the checkpointing + * mechanism to provide exactly once semantics. + * + * Incoming records are stored within a {@link org.apache.flink.runtime.state.AbstractStateBackend}, and only committed if a + * checkpoint is completed. Should a job fail while the data is being committed, no exactly once guarantee can be made. + * @param + */ +public abstract class GenericExactlyOnceSink extends AbstractStreamOperator implements OneInputStreamOperator { + private AbstractStateBackend.CheckpointStateOutputView out; + private TypeSerializer serializer; + protected TypeInformation typeInfo; + + private ExactlyOnceState state = new ExactlyOnceState(); + + /** +* Saves a handle in the state. +* @param checkpointId +* @throws IOException +*/ + private void saveHandleInState(final long checkpointId) throws IOException { + //only add handle if a new OperatorState was created since the last snapshot/notify + if (out != null) { + out.writeByte(0); //EOF-byte + StateHandle handle = out.closeAndGetHandle(); + if (state.pendingHandles.containsKey(checkpointId)) { + state.pendingHandles.get(checkpointId).add(handle); + } else { + ArrayList> list = new ArrayList<>(); + list.add(handle); + state.pendingHandles.put(checkpointId, list); + } + out = null; + } + } + + @Override + public StreamTaskState snapshotOperatorState(final long checkpointId, final long timestamp) throws Exception { + StreamTaskState taskState = super.snapshotOperatorState(checkpointId, timestamp); + saveHandleInState(checkpointId); + taskState.setFunctionState(state); + return taskState; + } + + @Override + public void restoreState(StreamTaskState state, long recoveryTimestamp) throws Exception { + super.restoreState(state, recoveryTimestamp); + this.state = (ExactlyOnceState) state.getFunctionState(); +
[jira] [Created] (FLINK-3509) Update Hadoop versions in release script and on travis to the latest minor version
Robert Metzger created FLINK-3509: - Summary: Update Hadoop versions in release script and on travis to the latest minor version Key: FLINK-3509 URL: https://issues.apache.org/jira/browse/FLINK-3509 Project: Flink Issue Type: Improvement Components: Build System Reporter: Robert Metzger Assignee: Robert Metzger The Hadoop versions in the travis.yml and the create-release-files.sh script are not updated to the latest Hadoop minor releases. Hadoop 2.7.2 and 2.6.4 were released in the meantime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs
[ https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167115#comment-15167115 ] ASF GitHub Bot commented on FLINK-3226: --- Github user twalthr closed the pull request at: https://github.com/apache/flink/pull/1679 > Translate optimized logical Table API plans into physical plans representing > DataSet programs > - > > Key: FLINK-3226 > URL: https://issues.apache.org/jira/browse/FLINK-3226 > Project: Flink > Issue Type: Sub-task > Components: Table API >Reporter: Fabian Hueske >Assignee: Chengxiang Li > > This issue is about translating an (optimized) logical Table API (see > FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 > representation of the DataSet program that will be executed. This means: > - Each Flink RelNode refers to exactly one Flink DataSet or DataStream > operator. > - All (join and grouping) keys of Flink operators are correctly specified. > - The expressions which are to be executed in user-code are identified. > - All fields are referenced with their physical execution-time index. > - Flink type information is available. > - Optional: Add physical execution hints for joins > The translation should be the final part of Calcite's optimization process. > For this task we need to: > - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one > Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all > relevant operator information (keys, user-code expression, strategy hints, > parallelism). > - implement rules to translate optimized Calcite RelNodes into Flink > RelNodes. We start with a straight-forward mapping and later add rules that > merge several relational operators into a single Flink operator, e.g., merge > a join followed by a filter. Timo implemented some rules for the first SQL > implementation which can be used as a starting point. > - Integrate the translation rules into the Calcite optimization process -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-3226] Translation of scalar function su...
Github user twalthr closed the pull request at: https://github.com/apache/flink/pull/1679 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1707) Add an Affinity Propagation Library Method
[ https://issues.apache.org/jira/browse/FLINK-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167061#comment-15167061 ] Josep Rubió commented on FLINK-1707: Sorry, I did the document in sharelatex but I forgot to share it with you. Anyway, I updated the description adding a link to the google doc. Thanks!! > Add an Affinity Propagation Library Method > -- > > Key: FLINK-1707 > URL: https://issues.apache.org/jira/browse/FLINK-1707 > Project: Flink > Issue Type: New Feature > Components: Gelly >Reporter: Vasia Kalavri >Assignee: Josep Rubió >Priority: Minor > Labels: requires-design-doc > Attachments: Binary_Affinity_Propagation_in_Flink_design_doc.pdf > > > This issue proposes adding the an implementation of the Affinity Propagation > algorithm as a Gelly library method and a corresponding example. > The algorithm is described in paper [1] and a description of a vertex-centric > implementation can be found is [2]. > [1]: http://www.psi.toronto.edu/affinitypropagation/FreyDueckScience07.pdf > [2]: http://event.cwi.nl/grades2014/00-ching-slides.pdf > Design doc: > https://docs.google.com/document/d/1QULalzPqMVICi8jRVs3S0n39pell2ZVc7RNemz_SGA4/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332)