Perfect after downloading it into the plugin, it is working well. I am wondering why these jars have been removed from opt/ folder, earlier I was able to copy from opt/ to lib/ folder for 1.10. For now I just downloaded from Maven for 1.11 and copied in plugin/.
Regards, Vijay On Tue, Oct 27, 2020 at 11:18 AM Chesnay Schepler <ches...@apache.org> wrote: > So the plugins directory is completely empty? > > In that case, please download the flink-metrics-graphite jar > <https://mvnrepository.com/artifact/org.apache.flink/flink-metrics-graphite/1.11.0> > and also copy it into the plugins directory. > > On 10/27/2020 7:04 PM, Vijayendra Yadav wrote: > > Also, you are right that the plugin did not have anything by default when > we created EMR 5.31 with Flink 1.11. > > In opt/ I see: > > [hadoop@ip-10-223-71-70 flink]$ pwd > /usr/lib/flink > [hadoop@ip-10-223-71-70 flink]$ ll opt/ > total 172860 > -rw-r--r-- 1 root root 24029243 Sep 19 03:08 > flink-azure-fs-hadoop-1.11.0.jar > -rw-r--r-- 1 root root 185395 Sep 19 03:11 flink-cep_2.11-1.11.0.jar > -rw-r--r-- 1 root root 53473 Sep 19 03:17 > flink-cep-scala_2.11-1.11.0.jar > -rw-r--r-- 1 root root 640604 Sep 19 03:16 flink-gelly_2.11-1.11.0.jar > -rw-r--r-- 1 root root 764049 Sep 19 03:16 > flink-gelly-scala_2.11-1.11.0.jar > -rw-r--r-- 1 root root 268951 Sep 19 03:17 flink-ml_2.11-1.11.0.jar > -rw-r--r-- 1 root root 22316430 Sep 19 03:08 flink-oss-fs-hadoop-1.11.0.jar > -rw-r--r-- 1 root root 37228704 Sep 19 03:17 flink-python_2.11-1.11.0.jar > -rw-r--r-- 1 root root 22155 Sep 19 03:16 > flink-queryable-state-runtime_2.11-1.11.0.jar > -rw-r--r-- 1 root root 19985454 Sep 19 03:08 flink-s3-fs-hadoop-1.11.0.jar > -rw-r--r-- 1 root root 36173428 Sep 19 03:08 flink-s3-fs-presto-1.11.0.jar > -rw-r--r-- 1 root root 194834 Aug 28 16:51 > flink-shaded-netty-tcnative-dynamic-2.0.25.Final-11.0.jar > -rw-r--r-- 1 root root 8028165 Aug 28 17:04 > flink-shaded-zookeeper-3.5.6.jar > -rw-r--r-- 1 root root 544183 Sep 19 03:17 > flink-sql-client_2.11-1.11.0.jar > -rw-r--r-- 1 root root 103766 Sep 19 03:17 > flink-state-processor-api_2.11-1.11.0.jar > -rw-r--r-- 1 root root 26428976 Sep 19 03:08 > flink-swift-fs-hadoop-1.11.0.jar > drwxr-xr-x 2 root root 134 Oct 13 18:01 python > > in lib/ I see: > > [hadoop@ip-10-223-71-70 flink]$ ll lib/ > total 190304 > -rw-r--r-- 1 root root 90784 Sep 19 03:14 flink-csv-1.11.0.jar > -rw-r--r-- 1 root root 114256876 Sep 19 03:17 flink-dist_2.11-1.11.0.jar > -rw-r--r-- 1 root root 94866 Sep 19 03:14 flink-json-1.11.0.jar > -rw-r--r-- 1 root root 7712156 Aug 28 16:51 > flink-shaded-zookeeper-3.4.14.jar > -rw-r--r-- 1 root root 33325748 Sep 19 03:17 flink-table_2.11-1.11.0.jar > -rw-r--r-- 1 root root 37330514 Sep 19 03:17 > flink-table-blink_2.11-1.11.0.jar > -rw-r--r-- 1 root root 67114 Aug 28 16:50 log4j-1.2-api-2.12.1.jar > -rw-r--r-- 1 root root 276771 Aug 28 16:50 log4j-api-2.12.1.jar > -rw-r--r-- 1 root root 1674433 Aug 28 16:50 log4j-core-2.12.1.jar > -rw-r--r-- 1 root root 23518 Aug 28 16:50 log4j-slf4j-impl-2.12.1.jar > > Regards, > Vijay > > On Tue, Oct 27, 2020 at 10:57 AM Vijayendra Yadav <contact....@gmail.com> > wrote: > >> Hi Chesnay, >> >> Steps to upgrade are as follows: >> >> 1) Created EMR 5.31 Cluster which comes with Flink 1.11 >> 2) Copied flink-s3-fs-hadoop-1.11.0.jar to plugin folder for application. >> >> cd /usr/lib/flink/ >> >> mkdir -p ./plugins/s3-fs-hadoop >> >> cp ./opt/flink-s3-fs-hadoop-1.11.0.jar ./plugins/s3-fs-hadoop/ >> >> 3) Recompiled Application with Flink 1.11 dependency. >> 4) Updated Graphite plugin class in config >> >> That is all I did. >> >> Regards, >> Vijay >> >> >> On Tue, Oct 27, 2020 at 10:00 AM Chesnay Schepler <ches...@apache.org> >> wrote: >> >>> How exactly did you do the upgrade? Did you copy some files from 1.11 >>> into an existing 1.10 distribution? >>> >>> The configuration is correct, but it appears as if the entire plugins >>> directory is either a) empty or b) not shipped. >>> >>> On 10/27/2020 5:22 PM, Vijayendra Yadav wrote: >>> >>> Hi Robert and Chesnay, >>> >>> Only thing changed is I upgraded from Flink 1.10 to 1.11 and to support >>> that updated conf yaml with factory class. >>> >>> Here is attached Full Log with classpath etc. (log.txt) >>> >>> Regards, >>> Vijay >>> >>> >>> >>> On Tue, Oct 27, 2020 at 9:31 AM Chesnay Schepler <ches...@apache.org> >>> wrote: >>> >>>> Are you writing a test? (otherwise the ReporterSetupTest reporters >>>> wouldn't be around) >>>> Do you have a dependency on the graphite reporter? >>>> >>>> On 10/27/2020 8:27 AM, Robert Metzger wrote: >>>> >>>> Hi Vijayendra, >>>> can you post or upload the entire logs, so that we can see the >>>> Classpath logged on startup, as well as the effective configuration >>>> parameters? >>>> >>>> On Tue, Oct 27, 2020 at 12:49 AM Vijayendra Yadav < >>>> contact....@gmail.com> wrote: >>>> >>>>> Hi Chesnay, >>>>> >>>>> Another log message: >>>>> >>>>> 2020-10-26 23:33:08,516 WARN >>>>> org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory >>>>> (org.apache.flink.metrics.graphite.GraphiteReporterFactory) could not be >>>>> found for reporter grph. Available factories: >>>>> [org.apache.flink.runtime.metrics.ReporterSetupTest$ConfigExposingReporterFactory, >>>>> org.apache.flink.runtime.metrics.ReporterSetupTest$TestReporterFactory, >>>>> org.apache.flink.runtime.metrics.ReporterSetupTest$InstantiationTypeTrackingTestReporterFactory, >>>>> org.apache.flink.runtime.metrics.ReporterSetupTest$FailingFactory]. >>>>> 2020-10-26 23:33:08,517 INFO >>>>> org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter >>>>> configured, no metrics will be exposed/reported. >>>>> Regards, >>>>> Vijay >>>>> >>>>> On Mon, Oct 26, 2020 at 2:34 PM Vijayendra Yadav < >>>>> contact....@gmail.com> wrote: >>>>> >>>>>> Hi Chesnay, >>>>>> >>>>>> I have the same, and I am exporting the flinkconf like below, where i >>>>>> have flink-conf.yaml with configuration you have given.What else can I >>>>>> try ? >>>>>> >>>>>> export FLINK_CONF_DIR=${app_install_path}/flinkconf/ >>>>>> >>>>>> regards, >>>>>> Vijay >>>>>> >>>>>> On Sun, Oct 25, 2020 at 8:03 AM Chesnay Schepler <ches...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> Ah wait, in 1.11 it should not longer be necessary to explicitly >>>>>>> copy the reporter jar. >>>>>>> >>>>>>> Please update your reporter configuration to this: >>>>>>> >>>>>>> metrics.reporter.grph.factory.class: >>>>>>> org.apache.flink.metrics.graphite.GraphiteReporterFactory >>>>>>> >>>>>>> On 10/25/2020 4:00 PM, Chesnay Schepler wrote: >>>>>>> >>>>>>> Have you followed the documentation, specifically this bit? >>>>>>> >>>>>>> > In order to use this reporter you must copy >>>>>>> /opt/flink-metrics-influxdb-1.11.2.jar into the plugins/influxdb >>>>>>> folder of your Flink distribution. >>>>>>> >>>>>>> On 10/24/2020 12:17 AM, Vijayendra Yadav wrote: >>>>>>> >>>>>>> Hi Team, >>>>>>> >>>>>>> for Flink 1.11 Graphite Metrics. I see the following Error in the >>>>>>> log. >>>>>>> Any suggestions? >>>>>>> >>>>>>> 020-10-23 21:55:14,652 ERROR >>>>>>> org.apache.flink.runtime.metrics.ReporterSetup - Could >>>>>>> not instantiate metrics reporter grph. Metrics might not be >>>>>>> exposed/reported. >>>>>>> java.lang.ClassNotFoundException: >>>>>>> org.apache.flink.metrics.graphite.GraphiteReporter >>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:382) >>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:418) >>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) >>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:351) >>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>> at java.lang.Class.forName(Class.java:264) >>>>>>> at >>>>>>> org.apache.flink.runtime.metrics.ReporterSetup.loadViaReflection(ReporterSetup.java:313) >>>>>>> at >>>>>>> org.apache.flink.runtime.metrics.ReporterSetup.loadReporter(ReporterSetup.java:274) >>>>>>> at >>>>>>> org.apache.flink.runtime.metrics.ReporterSetup.setupReporters(ReporterSetup.java:235) >>>>>>> at >>>>>>> org.apache.flink.runtime.metrics.ReporterSetup.fromConfiguration(ReporterSetup.java:148) >>>>>>> at >>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createMetricRegistry(ClusterEntrypoint.java:316) >>>>>>> at >>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:270) >>>>>>> at >>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:208) >>>>>>> at >>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:169) >>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422) >>>>>>> at >>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) >>>>>>> at >>>>>>> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) >>>>>>> at >>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:168) >>>>>>> at >>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:517) >>>>>>> at >>>>>>> org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:89) >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Vijay >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>> >>> >
