In the normal Flink distribution these jars were moved from opt/ to
plugins/ so that they are available by default without having to mess
around with any jars.
I don't think anyone was aware that the plugin directory is not
populated on EMR.
On 10/27/2020 9:53 PM, Vijayendra Yadav wrote:
Perfect after downloading it into the plugin, it is working well. I am
wondering why these jars have been removed from opt/ folder, earlier I
was able to copy from opt/ to lib/ folder for 1.10.
For now I just downloaded from Maven for 1.11 and copied in plugin/.
Regards,
Vijay
On Tue, Oct 27, 2020 at 11:18 AM Chesnay Schepler <ches...@apache.org
<mailto:ches...@apache.org>> wrote:
So the plugins directory is completely empty?
In that case, please download the flink-metrics-graphite jar
<https://mvnrepository.com/artifact/org.apache.flink/flink-metrics-graphite/1.11.0>
and also copy it into the plugins directory.
On 10/27/2020 7:04 PM, Vijayendra Yadav wrote:
Also, you are right that the plugin did not have anything by
default when we created EMR 5.31 with Flink 1.11.
In opt/ I see:
[hadoop@ip-10-223-71-70 flink]$ pwd
/usr/lib/flink
[hadoop@ip-10-223-71-70 flink]$ ll opt/
total 172860
-rw-r--r-- 1 root root 24029243 Sep 19 03:08
flink-azure-fs-hadoop-1.11.0.jar
-rw-r--r-- 1 root root 185395 Sep 19 03:11
flink-cep_2.11-1.11.0.jar
-rw-r--r-- 1 root root 53473 Sep 19 03:17
flink-cep-scala_2.11-1.11.0.jar
-rw-r--r-- 1 root root 640604 Sep 19 03:16
flink-gelly_2.11-1.11.0.jar
-rw-r--r-- 1 root root 764049 Sep 19 03:16
flink-gelly-scala_2.11-1.11.0.jar
-rw-r--r-- 1 root root 268951 Sep 19 03:17 flink-ml_2.11-1.11.0.jar
-rw-r--r-- 1 root root 22316430 Sep 19 03:08
flink-oss-fs-hadoop-1.11.0.jar
-rw-r--r-- 1 root root 37228704 Sep 19 03:17
flink-python_2.11-1.11.0.jar
-rw-r--r-- 1 root root 22155 Sep 19 03:16
flink-queryable-state-runtime_2.11-1.11.0.jar
-rw-r--r-- 1 root root 19985454 Sep 19 03:08
flink-s3-fs-hadoop-1.11.0.jar
-rw-r--r-- 1 root root 36173428 Sep 19 03:08
flink-s3-fs-presto-1.11.0.jar
-rw-r--r-- 1 root root 194834 Aug 28 16:51
flink-shaded-netty-tcnative-dynamic-2.0.25.Final-11.0.jar
-rw-r--r-- 1 root root 8028165 Aug 28 17:04
flink-shaded-zookeeper-3.5.6.jar
-rw-r--r-- 1 root root 544183 Sep 19 03:17
flink-sql-client_2.11-1.11.0.jar
-rw-r--r-- 1 root root 103766 Sep 19 03:17
flink-state-processor-api_2.11-1.11.0.jar
-rw-r--r-- 1 root root 26428976 Sep 19 03:08
flink-swift-fs-hadoop-1.11.0.jar
drwxr-xr-x 2 root root 134 Oct 13 18:01 python
in lib/ I see:
[hadoop@ip-10-223-71-70 flink]$ ll lib/
total 190304
-rw-r--r-- 1 root root 90784 Sep 19 03:14 flink-csv-1.11.0.jar
-rw-r--r-- 1 root root 114256876 Sep 19 03:17
flink-dist_2.11-1.11.0.jar
-rw-r--r-- 1 root root 94866 Sep 19 03:14 flink-json-1.11.0.jar
-rw-r--r-- 1 root root 7712156 Aug 28 16:51
flink-shaded-zookeeper-3.4.14.jar
-rw-r--r-- 1 root root 33325748 Sep 19 03:17
flink-table_2.11-1.11.0.jar
-rw-r--r-- 1 root root 37330514 Sep 19 03:17
flink-table-blink_2.11-1.11.0.jar
-rw-r--r-- 1 root root 67114 Aug 28 16:50
log4j-1.2-api-2.12.1.jar
-rw-r--r-- 1 root root 276771 Aug 28 16:50 log4j-api-2.12.1.jar
-rw-r--r-- 1 root root 1674433 Aug 28 16:50 log4j-core-2.12.1.jar
-rw-r--r-- 1 root root 23518 Aug 28 16:50
log4j-slf4j-impl-2.12.1.jar
Regards,
Vijay
On Tue, Oct 27, 2020 at 10:57 AM Vijayendra Yadav
<contact....@gmail.com <mailto:contact....@gmail.com>> wrote:
Hi Chesnay,
Steps to upgrade are as follows:
1) Created EMR 5.31 Cluster which comes with Flink 1.11
2) Copied flink-s3-fs-hadoop-1.11.0.jar to plugin folder for
application.
cd /usr/lib/flink/
mkdir -p ./plugins/s3-fs-hadoop
cp ./opt/flink-s3-fs-hadoop-1.11.0.jar ./plugins/s3-fs-hadoop/
3) Recompiled Application with Flink 1.11 dependency.
4) Updated Graphite plugin class in config
That is all I did.
Regards,
Vijay
On Tue, Oct 27, 2020 at 10:00 AM Chesnay Schepler
<ches...@apache.org <mailto:ches...@apache.org>> wrote:
How exactly did you do the upgrade? Did you copy some
files from 1.11 into an existing 1.10 distribution?
The configuration is correct, but it appears as if the
entire plugins directory is either a) empty or b) not
shipped.
On 10/27/2020 5:22 PM, Vijayendra Yadav wrote:
Hi Robert and Chesnay,
Only thing changed is I upgraded from Flink 1.10 to
1.11 and to support that updated conf yaml with factory
class.
Here is attached Full Log with classpath etc. (log.txt)
Regards,
Vijay
On Tue, Oct 27, 2020 at 9:31 AM Chesnay Schepler
<ches...@apache.org <mailto:ches...@apache.org>> wrote:
Are you writing a test? (otherwise the
ReporterSetupTest reporters wouldn't be around)
Do you have a dependency on the graphite reporter?
On 10/27/2020 8:27 AM, Robert Metzger wrote:
Hi Vijayendra,
can you post or upload the entire logs, so that we
can see the Classpath logged on startup, as well as
the effective configuration parameters?
On Tue, Oct 27, 2020 at 12:49 AM Vijayendra Yadav
<contact....@gmail.com
<mailto:contact....@gmail.com>> wrote:
Hi Chesnay,
Another log message:
2020-10-26 23:33:08,516 WARN
org.apache.flink.runtime.metrics.ReporterSetup
- The reporter factory
(org.apache.flink.metrics.graphite.GraphiteReporterFactory)
could not be found for reporter grph. Available
factories:
[org.apache.flink.runtime.metrics.ReporterSetupTest$ConfigExposingReporterFactory,
org.apache.flink.runtime.metrics.ReporterSetupTest$TestReporterFactory,
org.apache.flink.runtime.metrics.ReporterSetupTest$InstantiationTypeTrackingTestReporterFactory,
org.apache.flink.runtime.metrics.ReporterSetupTest$FailingFactory].
2020-10-26 23:33:08,517 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl
- No metrics reporter configured, no metrics
will be exposed/reported.
Regards,
Vijay
On Mon, Oct 26, 2020 at 2:34 PM Vijayendra
Yadav <contact....@gmail.com
<mailto:contact....@gmail.com>> wrote:
Hi Chesnay,
I have the same, and I am exporting the
flinkconf like below, where i have
flink-conf.yaml with configuration you have
given.What else can I try ?
export
FLINK_CONF_DIR=${app_install_path}/flinkconf/
regards,
Vijay
On Sun, Oct 25, 2020 at 8:03 AM Chesnay
Schepler <ches...@apache.org
<mailto:ches...@apache.org>> wrote:
Ah wait, in 1.11 it should not longer
be necessary to explicitly copy the
reporter jar.
Please update your reporter
configuration to this:
|metrics.reporter.grph.factory.class:
org.apache.flink.metrics.graphite.GraphiteReporterFactory|
On 10/25/2020 4:00 PM, Chesnay Schepler
wrote:
Have you followed the documentation,
specifically this bit?
> In order to use this reporter you
must copy
|/opt/flink-metrics-influxdb-1.11.2.jar|
into the |plugins/influxdb| folder of
your Flink distribution.
On 10/24/2020 12:17 AM, Vijayendra
Yadav wrote:
Hi Team,
for Flink 1.11 Graphite Metrics. I
see the following Error in the log.
Any suggestions?
020-10-23 21:55:14,652 ERROR
org.apache.flink.runtime.metrics.ReporterSetup - Could not
instantiate metrics reporter grph. Metrics might not be exposed/reported.
java.lang.ClassNotFoundException:
org.apache.flink.metrics.graphite.GraphiteReporter
at
java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at
java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at
java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
org.apache.flink.runtime.metrics.ReporterSetup.loadViaReflection(ReporterSetup.java:313)
at
org.apache.flink.runtime.metrics.ReporterSetup.loadReporter(ReporterSetup.java:274)
at
org.apache.flink.runtime.metrics.ReporterSetup.setupReporters(ReporterSetup.java:235)
at
org.apache.flink.runtime.metrics.ReporterSetup.fromConfiguration(ReporterSetup.java:148)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createMetricRegistry(ClusterEntrypoint.java:316)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:270)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:208)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:169)
at
java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:168)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:517)
at
org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:89)
Regards,
Vijay
