[GitHub] flink pull request: [FLINK-2008][FLINK-2296] Fix checkpoint commit...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/895#discussion_r34262145 --- Diff: flink-tests/src/test/resources/log4j-test.properties --- @@ -18,7 +18,7 @@ # Set root logger level to OFF to not flood build logs # set manually to INFO for debugging purposes -log4j.rootLogger=OFF, testlogger +log4j.rootLogger=INFO, testlogger --- End diff -- Yes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [doc] Fix wordcount example in YARN setup
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/897#issuecomment-120279506 +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2008][FLINK-2296] Fix checkpoint commit...
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/895 [FLINK-2008][FLINK-2296] Fix checkpoint committing KafkaITCase [wip] You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink2008 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/895.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #895 commit 0d5773a56acb9f6d7592de9bf5a93f04e6600ca1 Author: Robert Metzger rmetz...@apache.org Date: 2015-06-29T14:52:38Z [FLINK-2008][FLINK-2296] Fix checkpoint committing KafkaITCase --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2337] Multiple SLF4J bindings using Sto...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/903#issuecomment-120702067 +1 looks good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2218] Web client cannot distinguesh bet...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/904#issuecomment-120702048 Are there any other possible options for the options field except for the parallelism? I think users might be confused how the options and arguments differ. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Bumped the docker container to the latest vers...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1366#issuecomment-157504656 thank you for the contribution, the change looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Bumped the docker container to the latest vers...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1366#issuecomment-157506571 Did you test the change? I've tried it but got the following error ``` Removing intermediate container 0b4ba3a6115e Step 3 : RUN mkdir ~/downloads && cd ~/downloads && wget -q -O - http://mirrors.advancedhosters.com/apache/flink/flink-0.10.0/flink-0.10.0-bin-hadoop2.tgz | tar -zxvf - -C /usr/local/ ---> Running in ad41096423ab gzip: stdin: unexpected end of file tar: Child returned status 1 tar: Error is not recoverable: exiting now The command '/bin/sh -c mkdir ~/downloads && cd ~/downloads && wget -q -O - http://mirrors.advancedhosters.com/apache/flink/flink-0.10.0/flink-0.10.0-bin-hadoop2.tgz | tar -zxvf - -C /usr/local/' returned a non-zero code: 2 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2111] Add "stop" signal to cleanly shut...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/750#issuecomment-157833952 There is a readme: https://github.com/apache/flink/tree/master/flink-runtime-web --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2977] Added support for accessing a Ker...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1342#discussion_r44767927
--- Diff: flink-dist/src/main/flink-bin/bin/config.sh ---
@@ -249,7 +249,15 @@ if [ -n "$HADOOP_HOME" ]; then
fi
fi
-INTERNAL_HADOOP_CLASSPATHS="$HADOOP_CLASSPATH:$HADOOP_CONF_DIR:$YARN_CONF_DIR"
+INTERNAL_HADOOP_CLASSPATHS="${HADOOP_CLASSPATH}:${HADOOP_CONF_DIR}:${YARN_CONF_DIR}"
+
+if [ -n "${HBASE_CONF_DIR}" ]; then
+# Setup the HBase classpath.
+INTERNAL_HADOOP_CLASSPATHS="${INTERNAL_HADOOP_CLASSPATHS}:`hbase
classpath`"
--- End diff --
Okay great, as long as it is not stopping the bash script, its fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2977] Added support for accessing a Ker...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1342#issuecomment-156388624 Thank you for addressing my concerns so quickly +1 to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2978][web-dashboard][webclient] Integra...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1338#issuecomment-156450181 here is the jar file: https://transfer.sh/KNAnm/flink0.10-scala2.11-1.0-snapshot.jar it has been build from here: https://github.com/rmetzger/scratch/tree/flink0.10-scala2.11/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2978][web-dashboard][webclient] Integra...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1338#issuecomment-156432901 I think we need to improve a bit on the error messages. This is what I got for submitting a job without arguments on the web interface: ``` Error in invoking the program: org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:929) org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:908) org.apache.flink.client.program.OptimizerPlanEnvironment.execute(OptimizerPlanEnvironment.java:47) org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:627) com.dataartisans.Job$.main(Job.scala:69) com.dataartisans.Job.main(Job.scala) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:497) org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:395) org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:80) org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:218) org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.handleRequest(JarActionHandler.java:117) ``` Starting the same in the IDE will lead to: ``` Exception in thread "main" java.lang.RuntimeException: No data for required key 'input' at org.apache.flink.api.java.utils.ParameterTool.getRequired(ParameterTool.java:236) at com.dataartisans.JobWithoutTrigger$.main(JobWithoutTrigger.scala:36) at com.dataartisans.JobWithoutTrigger.main(JobWithoutTrigger.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) Process finished with exit code 1 ``` It seems you are cutting away the wrong part of the stack trace --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2978][web-dashboard][webclient] Integra...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1338#issuecomment-156433498 Even with all the fields filled, it was not starting: http://i.imgur.com/5WXKw9N.png the log file also contained no helpful information --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2977] Added support for accessing a Ker...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1342#issuecomment-156423600 Amazing, thank you. I'm going to merge you change as soon as Travis gives me green light ;) I've squashed and rebased your commit and will push it probably as https://github.com/rmetzger/flink/commit/55248ec26337797d56bc8bddd5f62c4db0ea195c --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2326] Write yarn properties file to tem...
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/1062 [FLINK-2326] Write yarn properties file to temp directory Flink's YARN client is writing a `.yarn-properties` file to the `conf/` directory containing the JobManager's host and port. It seems that at least two users were installing Flink into a read only location, thus Flink was unable to write the properties file to `conf/`. With this change, the file is written into the `/tmp/` directory with the current user's name. This will allow to use a shared Flink installation with different users at the same time. I also introduced a configuration value to change the location of the properties file. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink2326 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1062.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1062 commit 473940eda06e2874d6b2cec5e50ee07d6054f438 Author: Robert Metzger rmetz...@apache.org Date: 2015-08-26T14:11:30Z [FLINK-2326] Write yarn properties file to temp directory --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2582] [docs] Add documentation how to b...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1070#issuecomment-135501709 Looks good, +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2543] Fix user object deserialization f...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1048#issuecomment-135770247 That sounds very good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2543] Fix user object deserialization f...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1048#issuecomment-135812417 Sending a stringified exception is not an option between jm and client because the user might rely on the exception in the RemoteExecEnv. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2590] fixing DataSetUtils.zipWithUnique...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1075#issuecomment-136129581 Thanks a lot for the contribution. Can you add a test case for the method to make sure the issue is not re-introduced again when somebody else is changing the code? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request:
Github user rmetzger commented on the pull request: https://github.com/apache/flink/commit/554b77bcd9ed66d57d99d4990774a43f35f6a835#commitcomment-12965859 Currently, flink-runtime has a dependency on Hadoop, so I can assume its always available. Even for a binary Flink release without build in Hadoop dependencies, we would assume Hadoop to be present (from the classpath). For a Flink release without any Hadoop, we can either remove this again or use some reflection / fake hadoop class magic (added via maven) if needed. But for now, I would like to have this in the code base because it helps debugging user issues. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2512] [bugfix] Used a concurrent list i...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1058#issuecomment-134980282 Are you sure this is a fix for FLINK-2512 ? I don't see how this change is fixing any issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2555] Properly pass security credential...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1038#issuecomment-134584589 I agree. Lets file a JIRA and do it separately, as this is probably a bigger task. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2555] Properly pass security credential...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1038#issuecomment-134531698 @mxm: I removed the comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2555] Properly pass security credential...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1038#issuecomment-135013779 Thanks alot! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK1919] add HCatOutputFormat
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1064#discussion_r38078423
--- Diff: flink-staging/flink-hcatalog/pom.xml ---
@@ -34,17 +34,64 @@ under the License.
packagingjar/packaging
+ repositories
+ repository
+ idcloudera/id
+
urlhttps://repository.cloudera.com/artifactory/cloudera-repos//url
+ /repository
+ /repositories
+
dependencies
dependency
groupIdorg.apache.flink/groupId
artifactIdflink-java/artifactId
version${project.version}/version
/dependency
-
+ dependency
+ groupIdorg.apache.flink/groupId
+ artifactIdflink-scala/artifactId
+ version${project.version}/version
+ /dependency
+ dependency
+ groupIdorg.apache.hive.hcatalog/groupId
+ artifactIdhive-hcatalog-core/artifactId
+ version1.1.0-cdh5.4.0/version
--- End diff --
+1 to use a vanilla release of the hive / hcatalog project.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK1919] add HCatOutputFormat
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/1064#discussion_r38078384 --- Diff: flink-staging/flink-hcatalog/pom.xml --- @@ -34,17 +34,64 @@ under the License. packagingjar/packaging + repositories + repository + idcloudera/id + urlhttps://repository.cloudera.com/artifactory/cloudera-repos//url + /repository + /repositories + --- End diff -- We should not depend on vendor specific repositories / versions in the normal builds. In the parent pom, there is a profile to enable vendor repositories. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2326] Write yarn properties file to tem...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1062#issuecomment-135372331 The travis build passed btw. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2326] Write yarn properties file to tem...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1062#issuecomment-135371903 Can you check the logs of `application_1440646275804_0005`? They are located in `target/yarn-...`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2326] Write yarn properties file to tem...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1062#issuecomment-135363533 Lets first wait for the builds to pass. The errors are actually not unrelated to my change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2543] Fix user object deserialization f...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1048#issuecomment-135363969 I addressed all PR comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2555] Properly pass security credential...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1038#issuecomment-135343326 I'm manually closing this pull request. It has been merged by @uce. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2555] Properly pass security credential...
Github user rmetzger closed the pull request at: https://github.com/apache/flink/pull/1038 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2386] Add new Kafka Consumer for Flink ...
Github user rmetzger closed the pull request at: https://github.com/apache/flink/pull/1055 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2386] Add new Kafka Consumer for Flink ...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1055#issuecomment-135327605 Thank you for merging it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2584] Check for unshaded classes in fat...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1076#issuecomment-136339437 Make sure it does not break with @tillrohrmann's recent merge ;) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2372] Add new FlinkKafkaProducer
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/1082 [FLINK-2372] Add new FlinkKafkaProducer This pull request is reworking Flink's KafkaSink. I've renamed the class to `FlinkKafkaProducer` to match it with `FlinkKafkaConsumer`. It uses the new Kafka [Producer API](http://kafka.apache.org/documentation.html#producerapi). In the documentation, they recommend using the new Producer API: > As of the 0.8.2 release we encourage all new development to use the new Java producer. This client is production tested and generally both faster and more fully featured than the previous Scala client. I've also noticed a good performance gain while doing some benchmarks with Flink and Kafka. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink2372-second Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1082.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1082 commit f042a6471f300e09894c521678a60aa144790134 Author: Robert Metzger <rmetz...@apache.org> Date: 2015-08-28T12:33:49Z [FLINK-2372] Add new FlinkKafkaProducer --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1984] Integrate Flink with Apache Mesos
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/948#issuecomment-136377793 Thank you for the response. I was asking because I think its a requirement that the JobManager is running as a mesos task in the cluster as well. But as far as I understood your answer (I'm really not a Mesos expert) that is the case. Did you also address the issues I had while deploying Flink on Mesos? Let me know when the PR is ready for a test drive again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2372] Add new FlinkKafkaProducer
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1082#issuecomment-13154 Thank you for the feedback. I will address the concerns. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2372] Add new FlinkKafkaProducer
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1082#issuecomment-136669637 I updated the PR and rebased to master (which is adding some commits from the future (at least from GitHubs perspective) ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2111] Add "stop" signal to cleanly shut...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/750#discussion_r38511039
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
---
@@ -411,6 +411,23 @@ class TaskManager(
log.debug(s"Cannot find task to fail for execution
${executionID})")
}
+// stops a task
+case StopTask(executionID) =>
+ val task = runningTasks.get(executionID)
+ if (task != null) {
+try {
+ task.stopExecution()
--- End diff --
I have to admit that this is the first time I'm looking into this pull
request ;)
Regarding the whole blocking / non-blocking discussion: I think even if we
would write into the javadocs that the cancel()/stop() call has to be
implemented in a non-blocking fashion, there would be still users who do it
wrong! Its just too risky to block the entire actor system by erroneous user
code. (Aren't our windowing threads doing some magic in the close methods as
well?!)
I'm think that the canceling of Tasks in the task manager is also done
using separate cancelling threads?
For the Kafka source: I believe we can move the "fetcher.close()" and
offsetHandler.close() into the close() method as well. (We would probably need
to add cancel() method to the Fetcher interface).
But I would not touch the Kafka consumer and make the stop() mechanism more
robust.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2111] Add "stop" signal to cleanly shut...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/750#discussion_r38511106 --- Diff: docs/apis/cli.md --- @@ -185,6 +189,18 @@ Action "list" lists running and scheduled programs. -s,--scheduledShow only scheduled programs and their JobIDs +Action "stop" stops a running program (streaming jobs only). + + Syntax: stop [OPTIONS] + "stop" action options: + -m,--jobmanagerAddress of the JobManager (master) to which + to connect. Specify 'yarn-cluster' as the + JobManager to deploy a YARN cluster for the + job. Use this flag to connect to a different --- End diff -- what happens when a user enters "stop -m yarn-cluster" ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2525]Add configuration support in Storm...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1046#discussion_r38844802
--- Diff:
flink-contrib/flink-storm-compatibility/flink-storm-compatibility-examples/src/main/java/org/apache/flink/stormcompatibility/excamation/ExclamationWithStormBolt.java
---
@@ -78,6 +86,11 @@ public static void main(final String[] args) throws
Exception {
exclaimed.print();
}
+ // set bolt and map exclamation marks num
+ Configuration conf = new Configuration();
+ conf.setInteger(new String("exclamationNum"), exclamationNum);
--- End diff --
Why are you creating Strings like this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2525]Add configuration support in Storm...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1046#discussion_r38844654
--- Diff:
flink-contrib/flink-storm-compatibility/flink-storm-compatibility-core/src/test/java/org/apache/flink/stormcompatibility/wrappers/StormBoltWrapperTest.java
---
@@ -222,6 +242,64 @@ public void testOpenSink() throws Exception {
@SuppressWarnings("unchecked")
@Test
+ public void testOpenWithStormConf() throws Exception {
+ final IRichBolt bolt = mock(IRichBolt.class);
+ final StormBoltWrapper<Object, Object> wrapper = new
StormBoltWrapper<Object, Object>(bolt);
+
+ Map stormConf = new HashMap();
+ stormConf.put(new String("path"), new
String("/home/user/file.txt"));
--- End diff --
Just use `"path`" here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2525]Add configuration support in Storm...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1046#discussion_r38844529
--- Diff:
flink-contrib/flink-storm-compatibility/flink-storm-compatibility-core/src/main/java/org/apache/flink/stormcompatibility/wrappers/StormWrapperSetupHelper.java
---
@@ -107,4 +112,40 @@ public static TopologyContext
convertToTopologyContext(final StreamingRuntimeCon
return new FlinkTopologyContext(new StormTopology(spoutSpecs,
bolts, null), taskToComponents, taskId);
}
+ /**
+* Get storm configuration from StreamingRuntimeContext.
+*
+* @param ctx
+*The RuntimeContext of operator.
+* @return The storm configuration map.
+* @throws Exception
+*/
+ public static Map getStormConfFromContext(final RuntimeContext ctx)
+ throws Exception {
+ Map stormConf = null;
+ if (ctx instanceof StreamingRuntimeContext) {
+ Configuration jobConfiguration =
((StreamingRuntimeContext) ctx).getJobConfiguration();
+
+ if (jobConfiguration != null) {
+ /* topologies mode */
+ stormConf = (Map)
InstantiationUtil.readObjectFromConfig(jobConfiguration,
StormConfig.STORM_DEFAULT_CONFIG, Map.class.getClassLoader());
--- End diff --
Since the map is untyped, it might happen that users pass arbitrary
objects, containing classes from the user code into the Map.
This would lead to class not found exceptions when running the code on
clusters. Can you use the classloader of `StormWrapperSetupHelper´ ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2111] Add "stop" signal to cleanly shut...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/750#discussion_r38856545 --- Diff: docs/apis/cli.md --- @@ -185,6 +189,18 @@ Action "list" lists running and scheduled programs. -s,--scheduledShow only scheduled programs and their JobIDs +Action "stop" stops a running program (streaming jobs only). + + Syntax: stop [OPTIONS] + "stop" action options: + -m,--jobmanagerAddress of the JobManager (master) to which + to connect. Specify 'yarn-cluster' as the + JobManager to deploy a YARN cluster for the + job. Use this flag to connect to a different --- End diff -- If you put "-m yarn-cluster", it will deploy a YARN cluster from the client. The "-m yarn cluster" option should only be available to the "run" command. For stop, users should usually not specify the JobManager address, because - in standalone mode, the JM address is in the conf - in YARN Mode, the JM address is in a special `.yarn-properties` file. Only if that is not working, users can use the "-m ". --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2111] Add "stop" signal to cleanly shut...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/750#discussion_r38863585 --- Diff: docs/apis/cli.md --- @@ -185,6 +189,18 @@ Action "list" lists running and scheduled programs. -s,--scheduledShow only scheduled programs and their JobIDs +Action "stop" stops a running program (streaming jobs only). + + Syntax: stop [OPTIONS] + "stop" action options: + -m,--jobmanagerAddress of the JobManager (master) to which + to connect. Specify 'yarn-cluster' as the + JobManager to deploy a YARN cluster for the + job. Use this flag to connect to a different --- End diff -- I'll write it down on my TODO list and fix it with the next YARN-related pull request (I'm planning to clean up the whole code a bit anyways) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2003] [docs] Building on some encrypted...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1100#issuecomment-138313755 Thank you for opening a new pull request. +1 to merge (@ merging committer: please rename the commit to include the JIRA id ;) ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2591]Add configuration parameter for de...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1107#issuecomment-138600722 Thanks a lot for opening the pull request. Can you change the behavior of this a little bit to make the `-n` argument a required argument again? (if the config param is not set) I think this behavior is the best: if `yarn.numberOfTaskManagers` == -1 (default value): users have to supply the `-n` argument. If `yarn.numberOfTaskManagers` is set to `>= 1`, the value will be used by default. If `yarn.numberOfTaskManagers` is set to `>= 1` and the user specified `-n`, the value of `-n` will have priority. Can you rename the configuration value to `yarn.defaultNumberOfTaskManagers`? Can you add a test into the `flink-yarn-tests` which is using a configuration file with `yarn.defaultNumberOfTaskManagers` set to `1` WITHOUT a `-n` argument to see if everything is working as expected? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2583] Add Stream Sink For Rolling HDFS ...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1084#issuecomment-137390414 I'm currently trying out the module. Some comments: - Why do we name the module `flink-connector-hdfs`. I think a name such as `flink-connector-filesystems` or `flink-connector-hfs` would be more appropriate because its is implemented against Hadoops `FileSystems` classes, and not HDFS classes. So users should be able to use the connector with other FS such as Tachyon, NFS, S3 etc. - Is there a way of re-using existing InputFormats with the rolling file sink? I guess users will start asking about CSV, Avro, Parquet ... - I think there is already some code to monitor a file system directory to ingest it into a data stream. Maybe it would make sense to move that code out of the core into this module? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2583] Add Stream Sink For Rolling HDFS ...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1084#discussion_r38627730
--- Diff: docs/apis/streaming_guide.md ---
@@ -1836,6 +1837,110 @@ More about information about Elasticsearch can be
found [here](https://elastic.c
[Back to top](#top)
+### HDFS
+
+This connector provides a Sink that writes rolling files to HDFS. To use
this connector, add the
+following dependency to your project:
+
+{% highlight xml %}
+
+ org.apache.flink
+ flink-connector-hdfs
+ {{site.version}}
+
+{% endhighlight %}
+
+Note that the streaming connectors are currently not part of the binary
+distribution. See
+[here](cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution)
+for information about how to package the program with the libraries for
+cluster execution.
+
+ HDFS Rolling File Sink
+
+The rolling behaviour as well as the writing can be configured but we will
get to that later.
+This is how you can create a default rolling sink:
+
+
+
+{% highlight java %}
+DataStream input = ...;
+
+input.addSink(new RollingHDFSSink("/base/path"));
+
+{% endhighlight %}
+
+
+{% highlight scala %}
+val input: DataStream[String] = ...
+
+input.addSink(new RollingHDFSSink("/base/path"))
+
+{% endhighlight %}
+
+
+
+The only required parameter is the base path in HDFS where the rolling
files (buckets) will be
+stored. The sink can be configured by specifying a custom bucketer, HDFS
writer and batch size.
+
+By default the rolling sink will use the pattern `"-MM-dd--HH"` to
name the rolling buckets.
--- End diff --
Can you make it a bit more explicit that a new directory is created when
the pattern changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2583] Add Stream Sink For Rolling HDFS ...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1084#discussion_r38435351
--- Diff:
flink-staging/flink-streaming/flink-streaming-connectors/flink-connector-hdfs/pom.xml
---
@@ -0,0 +1,107 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-streaming-connectors-parent
+ 0.10-SNAPSHOT
+ ..
+
+
+ flink-connector-hdfs
+ flink-connector-hdfs
+
+ jar
+
+
+
+
+
+
+
+ org.apache.flink
+ flink-streaming-core
+ ${project.version}
+
+
+
+
+ org.apache.flink
+ flink-streaming-core
+ ${project.version}
+ test
+ test-jar
+
+
+
+ org.apache.flink
+ flink-tests
+ ${project.version}
+ test
+
+
+
+ org.apache.flink
+ flink-test-utils
+ ${project.version}
+ test
+
+
+
+
+ org.apache.flink
+ flink-runtime
+ test
+ test-jar
+ ${project.version}
+
+
+
--- End diff --
why are you not using our shaded hadoop dependency?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2607][quickstart] ignore signature file...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1085#issuecomment-137095491 +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1674] Add test with nested avro type
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1091#issuecomment-137679740 Thank you for the contribution. We'll soon review your pull request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2386] Add new Kafka Consumer for Flink ...
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/1055 [FLINK-2386] Add new Kafka Consumer for Flink 0.9.1 This is PR https://github.com/apache/flink/pull/1039 against the **release-0.9** branch! You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink091-kafka Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1055.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1055 commit 3fffaab3710344a415835d4e264f0fbeac044350 Author: Robert Metzger rmetz...@apache.org Date: 2015-08-24T11:58:22Z [FLINK-2386] Add new Kafka Consumer for Flink --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2386] Add new KafkaConsumer, based on K...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1039#issuecomment-134584442 I addressed @hsaputra's comments rebased to the latest master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1984] Integrate Flink with Apache Mesos
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/948#discussion_r38957167
--- Diff:
flink-mesos/src/main/scala/org/apache/flink/mesos/scheduler/SchedulerUtils.scala
---
@@ -0,0 +1,358 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.mesos.scheduler
+
+import java.util.{List => JList}
+
+import scala.collection.JavaConversions._
+import scala.util.{Failure, Success, Try}
+import scala.util.control.NonFatal
+
+import com.google.common.base.Splitter
+import com.google.protobuf.{ByteString, GeneratedMessage}
+import org.apache.flink.configuration.{ConfigConstants, Configuration,
GlobalConfiguration}
+import org.apache.flink.configuration.ConfigConstants._
+import org.apache.flink.mesos._
+import org.apache.flink.mesos.scheduler.FlinkScheduler._
+import org.apache.flink.runtime.StreamingMode
+import org.apache.flink.runtime.jobmanager.{JobManager, JobManagerMode}
+import org.apache.flink.runtime.util.EnvironmentInformation
+import org.apache.mesos.{MesosSchedulerDriver, Scheduler, SchedulerDriver}
+import org.apache.mesos.Protos._
+import org.apache.mesos.Protos.CommandInfo.URI
+import org.apache.mesos.Protos.Value.Ranges
+import org.apache.mesos.Protos.Value.Type._
+
+/**
+ * This code is borrowed and inspired from Apache Spark Project:
+ *
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
+ */
+trait SchedulerUtils {
+
+ /**
+ * Converts the attributes from the resource offer into a Map of name ->
Attribute Value
+ * The attribute values are the mesos attribute types and they are
+ * @param offerAttributes List of attributes sent with an offer
+ * @return
+ */
+ def toAttributeMap(offerAttributes: JList[Attribute]): Map[String,
GeneratedMessage] = {
+offerAttributes.map(attr => {
+ val attrValue = attr.getType match {
+case SCALAR => attr.getScalar
+case Value.Type.RANGES => attr.getRanges
+case Value.Type.SET => attr.getSet
+case Value.Type.TEXT => attr.getText
+ }
+ (attr.getName, attrValue)
+}).toMap
+ }
+
+ def createJavaExecCommand(jvmArgs: String = "", classPath: String =
"flink-*.jar",
+classToExecute: String, args: String = ""):
String = {
+s"env; java $jvmArgs -cp $classPath $classToExecute $args"
+ }
+
+ def createExecutorInfo(id: String, role: String, artifactURIs:
Set[String], command: String,
+ nativeLibPath: String): ExecutorInfo = {
+val uris = artifactURIs.map(uri =>
URI.newBuilder().setValue(uri).build())
+ExecutorInfo.newBuilder()
+ .setExecutorId(ExecutorID
+.newBuilder()
+.setValue(s"executor_$id"))
+ .setName(s"Apache Flink Mesos Executor - $id")
+ .setCommand(CommandInfo.newBuilder()
+.setValue(s"env; $command")
+.addAllUris(uris)
+.setEnvironment(Environment.newBuilder()
+ .addVariables(Environment.Variable.newBuilder()
+.setName("MESOS_NATIVE_JAVA_LIBRARY").setValue(nativeLibPath)))
+.setValue(command))
+ .build()
+ }
+
+ def createTaskInfo(taskName: String, taskId: TaskID, slaveID: SlaveID,
role: String, mem: Double,
+ cpus: Double, disk: Double, ports: Set[Int],
executorInfo: ExecutorInfo,
+ conf: Configuration): TaskInfo = {
+
+val portRanges = Ranges.newBuilder().addAllRange(
+ ports.map(port =>
Value.Range.newBuilder().setBegin(port).setEnd(port).build())).build()
+
+val taskConf = conf.clone()
+val portsSeq = ports.toSeq
+// set task manager ports
+taskConf.setInteger(Confi
[GitHub] flink pull request: [FLINK-1984] Integrate Flink with Apache Mesos
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/948#issuecomment-138640533 I'm trying out the code again in GCE ;) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2097] Implement job session management
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/858#discussion_r38951273
--- Diff:
flink-tests/src/test/scala/org/apache/flink/api/scala/runtime/jobmanager/JobManagerFailsITCase.scala
---
@@ -53,41 +54,41 @@ class JobManagerFailsITCase(_system: ActorSystem)
}
"A TaskManager" should {
-"detect a lost connection to the JobManager and try to reconnect to
it" in {
-
- val num_slots = 13
- val cluster = startDeathwatchCluster(num_slots, 1)
-
- val tm = cluster.getTaskManagers(0)
- val jmGateway =
cluster.getLeaderGateway(TestingUtils.TESTING_DURATION)
-
- // disable disconnect message to test death watch
- tm ! DisableDisconnect
-
- try {
-within(TestingUtils.TESTING_DURATION) {
- jmGateway.tell(RequestNumberRegisteredTaskManager, self)
- expectMsg(1)
-
- tm ! NotifyWhenJobManagerTerminated(jmGateway.actor)
-
- jmGateway.tell(PoisonPill, self)
-
- expectMsgClass(classOf[JobManagerTerminated])
-
- cluster.restartLeadingJobManager()
-
- cluster.waitForTaskManagersToBeRegistered()
-
- cluster.getLeaderGateway(TestingUtils.TESTING_DURATION)
-.tell(RequestNumberRegisteredTaskManager, self)
-
- expectMsg(1)
-}
- } finally {
-cluster.stop()
- }
-}
+//"detect a lost connection to the JobManager and try to reconnect to
it" in {
--- End diff --
Re-enable the test?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1984] Integrate Flink with Apache Mesos
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/948#discussion_r38958292
--- Diff:
flink-mesos/src/main/scala/org/apache/flink/mesos/scheduler/SchedulerUtils.scala
---
@@ -0,0 +1,358 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.mesos.scheduler
+
+import java.util.{List => JList}
+
+import scala.collection.JavaConversions._
+import scala.util.{Failure, Success, Try}
+import scala.util.control.NonFatal
+
+import com.google.common.base.Splitter
+import com.google.protobuf.{ByteString, GeneratedMessage}
+import org.apache.flink.configuration.{ConfigConstants, Configuration,
GlobalConfiguration}
+import org.apache.flink.configuration.ConfigConstants._
+import org.apache.flink.mesos._
+import org.apache.flink.mesos.scheduler.FlinkScheduler._
+import org.apache.flink.runtime.StreamingMode
+import org.apache.flink.runtime.jobmanager.{JobManager, JobManagerMode}
+import org.apache.flink.runtime.util.EnvironmentInformation
+import org.apache.mesos.{MesosSchedulerDriver, Scheduler, SchedulerDriver}
+import org.apache.mesos.Protos._
+import org.apache.mesos.Protos.CommandInfo.URI
+import org.apache.mesos.Protos.Value.Ranges
+import org.apache.mesos.Protos.Value.Type._
+
+/**
+ * This code is borrowed and inspired from Apache Spark Project:
+ *
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
+ */
+trait SchedulerUtils {
+
+ /**
+ * Converts the attributes from the resource offer into a Map of name ->
Attribute Value
+ * The attribute values are the mesos attribute types and they are
+ * @param offerAttributes List of attributes sent with an offer
+ * @return
+ */
+ def toAttributeMap(offerAttributes: JList[Attribute]): Map[String,
GeneratedMessage] = {
+offerAttributes.map(attr => {
+ val attrValue = attr.getType match {
+case SCALAR => attr.getScalar
+case Value.Type.RANGES => attr.getRanges
+case Value.Type.SET => attr.getSet
+case Value.Type.TEXT => attr.getText
+ }
+ (attr.getName, attrValue)
+}).toMap
+ }
+
+ def createJavaExecCommand(jvmArgs: String = "", classPath: String =
"flink-*.jar",
+classToExecute: String, args: String = ""):
String = {
+s"env; java $jvmArgs -cp $classPath $classToExecute $args"
+ }
+
+ def createExecutorInfo(id: String, role: String, artifactURIs:
Set[String], command: String,
+ nativeLibPath: String): ExecutorInfo = {
+val uris = artifactURIs.map(uri =>
URI.newBuilder().setValue(uri).build())
+ExecutorInfo.newBuilder()
+ .setExecutorId(ExecutorID
+.newBuilder()
+.setValue(s"executor_$id"))
+ .setName(s"Apache Flink Mesos Executor - $id")
+ .setCommand(CommandInfo.newBuilder()
+.setValue(s"env; $command")
+.addAllUris(uris)
+.setEnvironment(Environment.newBuilder()
+ .addVariables(Environment.Variable.newBuilder()
+.setName("MESOS_NATIVE_JAVA_LIBRARY").setValue(nativeLibPath)))
+.setValue(command))
+ .build()
+ }
+
+ def createTaskInfo(taskName: String, taskId: TaskID, slaveID: SlaveID,
role: String, mem: Double,
+ cpus: Double, disk: Double, ports: Set[Int],
executorInfo: ExecutorInfo,
+ conf: Configuration): TaskInfo = {
+
+val portRanges = Ranges.newBuilder().addAllRange(
+ ports.map(port =>
Value.Range.newBuilder().setBegin(port).setEnd(port).build())).build()
+
+val taskConf = conf.clone()
+val portsSeq = ports.toSeq
+// set task manager ports
+taskConf.setInteger(Confi
[GitHub] flink pull request: [FLINK-1984] Integrate Flink with Apache Mesos
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/948#issuecomment-138649089 it seems that the taskmanagers are failing and the scheduler keeps scheduling new ones: http://i.imgur.com/iZiYa4u.png --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2408] Define all maven properties outsi...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/941#issuecomment-139186364 I will rebase and merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2639] Add repository for hdp specific j...
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/1113 [FLINK-2639] Add repository for hdp specific jetty to 'vendor-repos' profile This adds a second hortonworks repository to the `vendor-repos` profile. Certain HDP-specific Hadoop releases depend on ``` [INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26.hwx:compile [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26.hwx:compile ``` Which is only available in a special Hortonworks repository. The dependency comes from `org.apache.hadoop:hadoop-common:jar:2.6.0.2.2.6.0-2800:compile`. Usually, we exclude jetty and other libraries which are only needed for Hadoop's web interfaces. In this case, the dependency comes from the `flink-shaded-include-yarn-tests`: The module builds a full shaded Hadoop package. For the `flink-yarn-tests` we need to also have jetty in the dependencies, because the YarnMiniCluster is also starting web servers (and we actually use these to test that Flink's web interface proxy is properly working). This change does not affect the regular builds, because the additional repository is "hidden" by the maven build profile. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink2639 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1113.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1113 commit 5aa5294e91a5476ac0f2df606ceaa03d30a532b4 Author: Robert Metzger <rmetz...@apache.org> Date: 2015-09-08T17:42:05Z [FLINK-2639] Add repository for hdp specific jetty to 'vendor-repos' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2111] Add "stop" signal to cleanly shut...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/750#issuecomment-139194317 > The cancel only sets an internal variable. That is probably true for all sources that Flink provides. But the interface for sources is a public API, and trusting that users are implementing this properly is dangerous. It can basically lock the entire Flink cluster. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2631] [streaming] Fixes the StreamFold ...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1101#issuecomment-139178919 +1 to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: FLINK-1737: Kronecker product
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1078#issuecomment-138825531 Thanks a lot for the pull request. Sorry that nobody looked at it yet. It seems that all committers are currently very busy. I'm sure somebody will give you soon feedback. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: FLINK-1737: Kronecker product
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1078#issuecomment-138826497 Your build is failing due to scalastyle checks ``` [INFO] [INFO] --- maven-failsafe-plugin:2.17:verify (default) @ flink-ml --- [INFO] Failsafe report directory: /home/travis/build/apache/flink/flink-staging/flink-ml/target/failsafe-reports [INFO] [INFO] --- scalastyle-maven-plugin:0.5.0:check (default) @ flink-ml --- error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/DenseVectorSuite.scala message=File line length exceeds 100 characters line=101 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/DenseVectorSuite.scala message=File line length exceeds 100 characters line=108 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/DenseVectorSuite.scala message=File line length exceeds 100 characters line=111 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/DenseVectorSuite.scala message=File line length exceeds 100 characters line=118 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/DenseVectorSuite.scala message=File line length exceeds 100 characters line=125 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/DenseVectorSuite.scala message=File line length exceeds 100 characters line=132 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/DenseVectorSuite.scala message=File line length exceeds 100 characters line=136 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/DenseVectorSuite.scala message=File line length exceeds 100 characters line=139 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/DenseVectorSuite.scala message=File line length exceeds 100 characters line=146 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/SparseVectorSuite.scala message=File line length exceeds 100 characters line=151 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/SparseVectorSuite.scala message=File line length exceeds 100 characters line=158 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/SparseVectorSuite.scala message=File line length exceeds 100 characters line=162 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/SparseVectorSuite.scala message=File line length exceeds 100 characters line=169 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/SparseVectorSuite.scala message=File line length exceeds 100 characters line=176 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/SparseVectorSuite.scala message=File line length exceeds 100 characters line=180 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/SparseVectorSuite.scala message=File line length exceeds 100 characters line=183 error file=/home/travis/build/apache/flink/flink-staging/flink-ml/src/test/scala/org/apache/flink/ml/math/SparseVectorSuite.scala message=File line length exceeds 100 characters line=190 Saving to outputFile=/home/travis/build/apache/flink/flink-staging/flink-ml/scalastyle-output.xml Processed 64 file(s) Found 17 errors ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2600] Enable test rerun for Elasticsear...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1108#issuecomment-138820891 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2632] Web Client does not respect the c...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1114#issuecomment-139223104 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2591]Add configuration parameter for de...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1107#issuecomment-138885088 I agree with your proposal! If `yarn.defaultNumberOfTaskManagers` is set to an invalid value, we should make `-n` required. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2654][flink-java] Add JavaDoc to Parame...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1116#discussion_r39173067
--- Diff:
flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java ---
@@ -136,15 +157,35 @@ public static ParameterTool fromPropertiesFile(String
path) throws IOException {
return fromMap((Map)props);
}
+/**
+ * Returns {@link ParameterTool} for the given map
+*
+ * @param map A map of arguments. Both Key and Value have to be Strings
+ * @return A {@link ParameterTool}
+ */
public static ParameterTool fromMap(Map<String, String> map) {
Preconditions.checkNotNull(map, "Unable to initialize from
empty map");
return new ParameterTool(map);
}
+/**
+ * Returns {@link ParameterTool} from the system properties
--- End diff --
Maybe it would make sense to add that people can pass system properties to
the jvm using `-Dkey=value`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2654][flink-java] Add JavaDoc to Parame...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1116#issuecomment-139280931 Thanks a lot for adding the Javadocs to the class. Good work! I have some comments in the diff. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2654][flink-java] Add JavaDoc to Parame...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1116#discussion_r39173366
--- Diff:
flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java ---
@@ -234,6 +295,10 @@ public long getLong(String key) {
return Long.valueOf(value);
}
+ /**
+* Returns the Long value for the given key. If the key does not exists
it will return the default value given.
+* The method fails if the value is not an Long.
--- End diff --
an long.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2654][flink-java] Add JavaDoc to Parame...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1116#discussion_r39173008
--- Diff:
flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java ---
@@ -124,6 +138,13 @@ else if(arg.startsWith("-")) {
return fromMap(map);
}
+/**
+ * Returns {@link ParameterTool} for the given properties file
--- End diff --
Maybe it would make sense to link to the documentation of Java Properties
(so that people know how the format looks like)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2654][flink-java] Add JavaDoc to Parame...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1116#discussion_r39173169
--- Diff:
flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java ---
@@ -124,6 +138,13 @@ else if(arg.startsWith("-")) {
return fromMap(map);
}
+/**
--- End diff --
The comments above are indented using tabs, here, you are switching to
spaces.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2654][flink-java] Add JavaDoc to Parame...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1116#discussion_r39172874
--- Diff:
flink-java/src/main/java/org/apache/flink/api/java/utils/ParameterTool.java ---
@@ -124,6 +138,13 @@ else if(arg.startsWith("-")) {
return fromMap(map);
}
+/**
+ * Returns {@link ParameterTool} for the given properties file
+*
--- End diff --
the `*` is off here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2617] Added static mutexes for configur...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/#issuecomment-138928137 +1 Looks good to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2656] Fix behavior of FlinkKafkaConsume...
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/1117 [FLINK-2656] Fix behavior of FlinkKafkaConsumer for out of range offsets You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink2656 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1117.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1117 commit 582d265e7ac068ff74a107e2bdcfbce87828c810 Author: Robert Metzger <rmetz...@apache.org> Date: 2015-09-10T16:47:45Z [FLINK-2656] Fix behavior of FlinkKafkaConsumer for out of range offsets --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2665] [api-breaking] [runtime] Makes Ex...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1128#issuecomment-140153585 +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2637] [api-breaking] [scala, types] Add...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1134#issuecomment-140663091 Big change, but I didn't spot anything suspicious. +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2566] FlinkTopologyContext not populate...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1135#issuecomment-140730955 There is a test failure ``` StormWrapperSetupHelperTest.testCreateTopologyContext:190 û ClassCast class su... ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2583] Add Stream Sink For Rolling HDFS ...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1084#issuecomment-140732373 I think the pull request has grown quite a lot. I think we should merge it now and then improve it from there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2640][yarn] integrate off-heap configur...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1132#issuecomment-140748694 +1 to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2583] Add Stream Sink For Rolling HDFS ...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/1084#discussion_r39624668 --- Diff: .travis.yml --- @@ -19,9 +19,9 @@ matrix: - jdk: "oraclejdk7" # this will also deploy a uberjar to s3 at some point env: PROFILE="-Dhadoop.profile=1" - jdk: "openjdk7" - env: PROFILE="-P!include-yarn -Dhadoop.version=2.0.0-alpha" -- jdk: "oraclejdk7" - env: PROFILE="-Dhadoop.version=2.2.0" + env: PROFILE="-Dhadoop.version=2.4.0" --- End diff -- As per mailing list discussion, we agreed to support hadoop 2.3.0. But it seems that the yarn tests are not working with 2.3.0, so if you want we can set the version to 2.3.0 when the tests are fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2200] Add Flink with Scala 2.11 in Mave...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/885#issuecomment-140731019 I agree. We need to use the script. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2566] FlinkTopologyContext not populate...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1135#discussion_r39624182
--- Diff:
flink-contrib/flink-storm-compatibility/flink-storm-compatibility-core/src/main/java/org/apache/flink/stormcompatibility/api/FlinkTopology.java
---
@@ -28,17 +28,14 @@
* cannot be executed directly, but must be handed over to a {@link
FlinkLocalCluster}, {@link FlinkSubmitter}, or
* {@link FlinkClient}.
*/
-class FlinkTopology extends StreamExecutionEnvironment {
+public class FlinkTopology extends StreamExecutionEnvironment {
--- End diff --
While trying out the Storm compat, I stumbled across the fact that the
FlinkTopology was only package visible
--> Good fix ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2583] Add Stream Sink For Rolling HDFS ...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/1084#discussion_r39640862 --- Diff: .travis.yml --- @@ -19,9 +19,9 @@ matrix: - jdk: "oraclejdk7" # this will also deploy a uberjar to s3 at some point env: PROFILE="-Dhadoop.profile=1" - jdk: "openjdk7" - env: PROFILE="-P!include-yarn -Dhadoop.version=2.0.0-alpha" -- jdk: "oraclejdk7" - env: PROFILE="-Dhadoop.version=2.2.0" + env: PROFILE="-Dhadoop.version=2.4.0" --- End diff -- I am okay, but this is a pretty important thing. Maybe we should wait for at least one more committer agreeing with us. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2656] Fix behavior of FlinkKafkaConsume...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1117#discussion_r39251890
--- Diff:
flink-staging/flink-streaming/flink-streaming-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/internals/LegacyFetcher.java
---
@@ -356,29 +357,24 @@ public void run() {
// make sure that all partitions have some
offsets to start with
// those partitions that do not have an offset
from a checkpoint need to get
// their start offset from ZooKeeper
-
- List partitionsToGetOffsetsFor
= new ArrayList<>();
+ {
--- End diff --
I'm opening a new scope for the `partitionsToGetOffsetsFor` variable.
I'm using another list with the same name later in the code. WIth a new
scope for the operation here, I can use the same variable name later again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2656] Fix behavior of FlinkKafkaConsume...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1117#issuecomment-139491178 Sorry Henry. I've updated the description. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2656] Fix behavior of FlinkKafkaConsume...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1117#issuecomment-139529893 Thank you for the review. I'll merge it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2591] Add configuration parameter for d...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1121#issuecomment-141016403 Sorry for the late response, I didn't see this new pull request. The failed test is okay. The test is known for being unstable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2591] Add configuration parameter for d...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1121#issuecomment-141019208 For others who follow this PR, there was already some discussion on this change here: https://github.com/apache/flink/pull/1107 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2357] Add the new web dashboard and mon...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1139#issuecomment-141035558 I agree with this approach, so +1 to merge ;) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2357] Add the new web dashboard and mon...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1139#issuecomment-141050563 Another follow-up todo is adding support for yarn: ``` 11:31:01,955 ERROR org.apache.flink.runtime.jobmanager.JobManager - WebServer could not be created org.apache.flink.configuration.IllegalConfigurationException: The path to the static contents (/yarn/nm/usercache/robert/appcache/application_1441703985068_0007/container_e07_1441703985068_0007_01_01/resources/web-runtime-monitor) is not a readable directory. at org.apache.flink.runtime.webmonitor.WebRuntimeMonitor.(WebRuntimeMonitor.java:137) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.flink.runtime.jobmanager.JobManager$.startWebRuntimeMonitor(JobManager.scala:1741) at org.apache.flink.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:134) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:356) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1651) at org.apache.flink.yarn.ApplicationMaster$.main(ApplicationMaster.scala:69) at org.apache.flink.yarn.ApplicationMaster.main(ApplicationMaster.scala) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2591] Add configuration parameter for d...
Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/1121#discussion_r39725184 --- Diff: flink-dist/src/main/resources/flink-conf.yaml --- @@ -130,6 +130,15 @@ state.backend: jobmanager #== +# YARN +#== + +# Default number of YARN container to allocate (=Number of Task Managers) + +yarn.defaultNumberOfTaskManagers: 1 --- End diff -- Can you remove the configuration value from the default configuration again? I would like to force new users to specify the number of yarn containers when they start Flink on YARN. The configuration value is meant for production users which want to control everything using configuration values. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2357] Add the new web dashboard and mon...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1139#issuecomment-141018736 Very nice. I tried the new interface locally, and it seems to work. I suspect these values are still test values? http://i.imgur.com/d9ZRR7g.png There is certainly more work to do until the web interface has the same features as the old interface (task manager overview / monitoring) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2591] Add configuration parameter for d...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1121#discussion_r39725238
--- Diff:
flink-yarn-tests/src/main/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java
---
@@ -111,6 +111,20 @@ public void testClientStartup() {
}
/**
+* Test configuration parameter for default number of yarn containers
+*/
+ @Test
+ public void testDefaultNumberOfTaskManagers() {
+ LOG.info("Starting testDefaultNumberOfTaskManagers()");
--- End diff --
I would generate a flink-conf.yaml just for this test to test the behavior
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2689] [runtime] Fix reuse of null objec...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1136#issuecomment-141168778 Hi @srowen, thank you for notifying us. We were trying out https://github.com/databricks/spark-pr-dashboard on Google App Engine (because we are also getting more and more flooded by pull requests ;) ). It seems that we didn't properly configure and test the tool before starting it. After you told us about the issue, we immediately stopped the tool. You can also see from Max (@mxm)'s JIRA activity (https://issues.apache.org/jira/secure/ViewProfile.jspa?name=mxm) that he stopped changing Spark JIRAs. I'm really sorry that this has happened. We'll manually clean up the Spark JIRAs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2637] [api-breaking] [scala, types] Add...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1134#issuecomment-141167625 Hey @rxin, I'm really sorry that this happened. Max told me that he stopped the application on AppEngine as soon as he learned that the tool is not working as expected. As far as I can see the JIRA activity of Max's user account, the tool stopped changing Spark issues 3 hours ago. Please let me know if more Spark JIRAs are affected. I'll ask @mxm to remove the automated comments again and change the links back to their old value. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1125#issuecomment-140029908 The pull request should also update the documentation. In particular these configuration values affect the documentation in many places. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1789] [core] [runtime] [java-api] Allow...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/593#discussion_r40493024
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/classloading/ClassLoaderITCase.java
---
@@ -51,12 +50,25 @@ public void testJobWithCustomInputFormat() {
PackagedProgram inputSplitTestProg = new
PackagedProgram(new File(INPUT_SPLITS_PROG_JAR_FILE),
new String[] {
INPUT_SPLITS_PROG_JAR_FILE,
+
"", // classpath
"localhost",
String.valueOf(port),
"4" // parallelism
} );
inputSplitTestProg.invokeInteractiveModeForExecution();
+
+
+ String classpath = new
File(INPUT_SPLITS_PROG_JAR_FILE).toURI().toURL().toString();
+ PackagedProgram inputSplitTestProg2 = new
PackagedProgram(new File(INPUT_SPLITS_PROG_JAR_FILE),
+ new String[] { "",
+
classpath, // classpath
+
"localhost",
+
String.valueOf(port),
+
"4" // parallelism
+ } );
+
inputSplitTestProg2.invokeInteractiveModeForExecution();
+
--- End diff --
Looks like I wrote this comment before testing the functionality ;) It
seems that the test does not detect the issues I found with manual testing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
