So the plugins directory is completely empty?
In that case, please download the flink-metrics-graphite jar
<https://mvnrepository.com/artifact/org.apache.flink/flink-metrics-graphite/1.11.0>
and also copy it into the plugins directory.
On 10/27/2020 7:04 PM, Vijayendra Yadav wrote:
Also, you are right that the plugin did not have anything by default
when we created EMR 5.31 with Flink 1.11.
In opt/ I see:
[hadoop@ip-10-223-71-70 flink]$ pwd
/usr/lib/flink
[hadoop@ip-10-223-71-70 flink]$ ll opt/
total 172860
-rw-r--r-- 1 root root 24029243 Sep 19 03:08
flink-azure-fs-hadoop-1.11.0.jar
-rw-r--r-- 1 root root 185395 Sep 19 03:11 flink-cep_2.11-1.11.0.jar
-rw-r--r-- 1 root root 53473 Sep 19 03:17
flink-cep-scala_2.11-1.11.0.jar
-rw-r--r-- 1 root root 640604 Sep 19 03:16 flink-gelly_2.11-1.11.0.jar
-rw-r--r-- 1 root root 764049 Sep 19 03:16
flink-gelly-scala_2.11-1.11.0.jar
-rw-r--r-- 1 root root 268951 Sep 19 03:17 flink-ml_2.11-1.11.0.jar
-rw-r--r-- 1 root root 22316430 Sep 19 03:08
flink-oss-fs-hadoop-1.11.0.jar
-rw-r--r-- 1 root root 37228704 Sep 19 03:17 flink-python_2.11-1.11.0.jar
-rw-r--r-- 1 root root 22155 Sep 19 03:16
flink-queryable-state-runtime_2.11-1.11.0.jar
-rw-r--r-- 1 root root 19985454 Sep 19 03:08 flink-s3-fs-hadoop-1.11.0.jar
-rw-r--r-- 1 root root 36173428 Sep 19 03:08 flink-s3-fs-presto-1.11.0.jar
-rw-r--r-- 1 root root 194834 Aug 28 16:51
flink-shaded-netty-tcnative-dynamic-2.0.25.Final-11.0.jar
-rw-r--r-- 1 root root 8028165 Aug 28 17:04
flink-shaded-zookeeper-3.5.6.jar
-rw-r--r-- 1 root root 544183 Sep 19 03:17
flink-sql-client_2.11-1.11.0.jar
-rw-r--r-- 1 root root 103766 Sep 19 03:17
flink-state-processor-api_2.11-1.11.0.jar
-rw-r--r-- 1 root root 26428976 Sep 19 03:08
flink-swift-fs-hadoop-1.11.0.jar
drwxr-xr-x 2 root root 134 Oct 13 18:01 python
in lib/ I see:
[hadoop@ip-10-223-71-70 flink]$ ll lib/
total 190304
-rw-r--r-- 1 root root 90784 Sep 19 03:14 flink-csv-1.11.0.jar
-rw-r--r-- 1 root root 114256876 Sep 19 03:17 flink-dist_2.11-1.11.0.jar
-rw-r--r-- 1 root root 94866 Sep 19 03:14 flink-json-1.11.0.jar
-rw-r--r-- 1 root root 7712156 Aug 28 16:51
flink-shaded-zookeeper-3.4.14.jar
-rw-r--r-- 1 root root 33325748 Sep 19 03:17 flink-table_2.11-1.11.0.jar
-rw-r--r-- 1 root root 37330514 Sep 19 03:17
flink-table-blink_2.11-1.11.0.jar
-rw-r--r-- 1 root root 67114 Aug 28 16:50 log4j-1.2-api-2.12.1.jar
-rw-r--r-- 1 root root 276771 Aug 28 16:50 log4j-api-2.12.1.jar
-rw-r--r-- 1 root root 1674433 Aug 28 16:50 log4j-core-2.12.1.jar
-rw-r--r-- 1 root root 23518 Aug 28 16:50 log4j-slf4j-impl-2.12.1.jar
Regards,
Vijay
On Tue, Oct 27, 2020 at 10:57 AM Vijayendra Yadav
<contact....@gmail.com <mailto:contact....@gmail.com>> wrote:
Hi Chesnay,
Steps to upgrade are as follows:
1) Created EMR 5.31 Cluster which comes with Flink 1.11
2) Copied flink-s3-fs-hadoop-1.11.0.jar to plugin folder for
application.
cd /usr/lib/flink/
mkdir -p ./plugins/s3-fs-hadoop
cp ./opt/flink-s3-fs-hadoop-1.11.0.jar ./plugins/s3-fs-hadoop/
3) Recompiled Application with Flink 1.11 dependency.
4) Updated Graphite plugin class in config
That is all I did.
Regards,
Vijay
On Tue, Oct 27, 2020 at 10:00 AM Chesnay Schepler
<ches...@apache.org <mailto:ches...@apache.org>> wrote:
How exactly did you do the upgrade? Did you copy some files
from 1.11 into an existing 1.10 distribution?
The configuration is correct, but it appears as if the entire
plugins directory is either a) empty or b) not shipped.
On 10/27/2020 5:22 PM, Vijayendra Yadav wrote:
Hi Robert and Chesnay,
Only thing changed is I upgraded from Flink 1.10 to 1.11 and
to support that updated conf yaml with factory class.
Here is attached Full Log with classpath etc. (log.txt)
Regards,
Vijay
On Tue, Oct 27, 2020 at 9:31 AM Chesnay Schepler
<ches...@apache.org <mailto:ches...@apache.org>> wrote:
Are you writing a test? (otherwise the ReporterSetupTest
reporters wouldn't be around)
Do you have a dependency on the graphite reporter?
On 10/27/2020 8:27 AM, Robert Metzger wrote:
Hi Vijayendra,
can you post or upload the entire logs, so that we can
see the Classpath logged on startup, as well as the
effective configuration parameters?
On Tue, Oct 27, 2020 at 12:49 AM Vijayendra Yadav
<contact....@gmail.com <mailto:contact....@gmail.com>>
wrote:
Hi Chesnay,
Another log message:
2020-10-26 23:33:08,516 WARN
org.apache.flink.runtime.metrics.ReporterSetup - The
reporter factory
(org.apache.flink.metrics.graphite.GraphiteReporterFactory)
could not be found for reporter grph. Available
factories:
[org.apache.flink.runtime.metrics.ReporterSetupTest$ConfigExposingReporterFactory,
org.apache.flink.runtime.metrics.ReporterSetupTest$TestReporterFactory,
org.apache.flink.runtime.metrics.ReporterSetupTest$InstantiationTypeTrackingTestReporterFactory,
org.apache.flink.runtime.metrics.ReporterSetupTest$FailingFactory].
2020-10-26 23:33:08,517 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl
- No metrics reporter configured, no metrics will be
exposed/reported.
Regards,
Vijay
On Mon, Oct 26, 2020 at 2:34 PM Vijayendra Yadav
<contact....@gmail.com
<mailto:contact....@gmail.com>> wrote:
Hi Chesnay,
I have the same, and I am exporting the
flinkconf like below, where i have
flink-conf.yaml with configuration you have
given.What else can I try ?
export FLINK_CONF_DIR=${app_install_path}/flinkconf/
regards,
Vijay
On Sun, Oct 25, 2020 at 8:03 AM Chesnay Schepler
<ches...@apache.org <mailto:ches...@apache.org>>
wrote:
Ah wait, in 1.11 it should not longer be
necessary to explicitly copy the reporter jar.
Please update your reporter configuration to
this:
|metrics.reporter.grph.factory.class:
org.apache.flink.metrics.graphite.GraphiteReporterFactory|
On 10/25/2020 4:00 PM, Chesnay Schepler wrote:
Have you followed the documentation,
specifically this bit?
> In order to use this reporter you must
copy
|/opt/flink-metrics-influxdb-1.11.2.jar|
into the |plugins/influxdb| folder of your
Flink distribution.
On 10/24/2020 12:17 AM, Vijayendra Yadav wrote:
Hi Team,
for Flink 1.11 Graphite Metrics. I see the
following Error in the log.
Any suggestions?
020-10-23 21:55:14,652 ERROR
org.apache.flink.runtime.metrics.ReporterSetup - Could not
instantiate metrics reporter grph. Metrics might not be exposed/reported.
java.lang.ClassNotFoundException:
org.apache.flink.metrics.graphite.GraphiteReporter
at
java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at
java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at
java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
org.apache.flink.runtime.metrics.ReporterSetup.loadViaReflection(ReporterSetup.java:313)
at
org.apache.flink.runtime.metrics.ReporterSetup.loadReporter(ReporterSetup.java:274)
at
org.apache.flink.runtime.metrics.ReporterSetup.setupReporters(ReporterSetup.java:235)
at
org.apache.flink.runtime.metrics.ReporterSetup.fromConfiguration(ReporterSetup.java:148)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createMetricRegistry(ClusterEntrypoint.java:316)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:270)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:208)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:169)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:168)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:517)
at
org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:89)
Regards,
Vijay
