[jira] [Created] (FLINK-20404) ZooKeeper quorum fails to start due to missing log4j library
Pedro Miguel Rainho Chaves created FLINK-20404:
--
Summary: ZooKeeper quorum fails to start due to missing log4j
library
Key: FLINK-20404
URL: https://issues.apache.org/jira/browse/FLINK-20404
Project: Flink
Issue Type: Bug
Affects Versions: 1.11.2
Reporter: Pedro Miguel Rainho Chaves
Upon starting a zookeeper quorum using flink's bootstrapped zookeeper, it
throws the following exception.
{code:java}
2020-11-27 13:13:38,371 ERROR
org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer [] - Error running
ZooKeeper quorum peer: org/apache/log4j/jmx/HierarchyDynamicMBean
java.lang.NoClassDefFoundError: org/apache/log4j/jmx/HierarchyDynamicMBean
at
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:51)
~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
at
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:125)
~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
at
org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.runFlinkZkQuorumPeer(FlinkZooKeeperQuorumPeer.java:123)
~[flink-dist_2.11-1.11.2.jar:1.11.2]
at
org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.main(FlinkZooKeeperQuorumPeer.java:79)
[flink-dist_2.11-1.11.2.jar:1.11.2]
Caused by: java.lang.ClassNotFoundException:
org.apache.log4j.jmx.HierarchyDynamicMBean
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
~[?:1.8.0_262]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_262]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
~[?:1.8.0_262]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_262]
... 4 more
{code}
This happens because the new flink version is missing a log4j library. This can
be solved by adding log4j-1.2.17.jar to the classpath, nonetheless the
bootstrapped zookeepeer version should be compatible with the log4j2 libraries
that come with flink's default installation.
*Steps to reproduce:*
# Fresh install of flink version 1.11.2
# Change the zookeeper config to start as a quorum
{code:java}
server.1=:2888:3888
server.2=:2888:3888{code}
# Start zookeeper
# /bin/zookeeper.sh start-foreground 1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-10135) The JobManager doesn't report the cluster-level metrics
[
https://issues.apache.org/jira/browse/FLINK-10135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609650#comment-16609650
]
Pedro Miguel Rainho Chaves commented on FLINK-10135:
I'm currently on version1.5.3 and am experiencing the same issue. I've listed
all of the available metrics via jmxterm and I get the following output.
{code:java}
$>domains
#following domains are available
JMImplementation
com.sun.management
java.lang
java.nio
java.util.logging
org.apache.flink.jobmanager.Status.JVM.CPU.Load
org.apache.flink.jobmanager.Status.JVM.CPU.Time
org.apache.flink.jobmanager.Status.JVM.ClassLoader.ClassesLoaded
org.apache.flink.jobmanager.Status.JVM.ClassLoader.ClassesUnloaded
org.apache.flink.jobmanager.Status.JVM.GarbageCollector.PS_MarkSweep.Count
org.apache.flink.jobmanager.Status.JVM.GarbageCollector.PS_MarkSweep.Time
org.apache.flink.jobmanager.Status.JVM.GarbageCollector.PS_Scavenge.Count
org.apache.flink.jobmanager.Status.JVM.GarbageCollector.PS_Scavenge.Time
org.apache.flink.jobmanager.Status.JVM.Memory.Direct.Count
org.apache.flink.jobmanager.Status.JVM.Memory.Direct.MemoryUsed
org.apache.flink.jobmanager.Status.JVM.Memory.Direct.TotalCapacity
org.apache.flink.jobmanager.Status.JVM.Memory.Heap.Committed
org.apache.flink.jobmanager.Status.JVM.Memory.Heap.Max
org.apache.flink.jobmanager.Status.JVM.Memory.Heap.Used
org.apache.flink.jobmanager.Status.JVM.Memory.Mapped.Count
org.apache.flink.jobmanager.Status.JVM.Memory.Mapped.MemoryUsed
org.apache.flink.jobmanager.Status.JVM.Memory.Mapped.TotalCapacity
org.apache.flink.jobmanager.Status.JVM.Memory.NonHeap.Committed
org.apache.flink.jobmanager.Status.JVM.Memory.NonHeap.Max
org.apache.flink.jobmanager.Status.JVM.Memory.NonHeap.Used
org.apache.flink.jobmanager.Status.JVM.Threads.Count
org.apache.flink.jobmanager.job.downtime
org.apache.flink.jobmanager.job.fullRestarts
org.apache.flink.jobmanager.job.lastCheckpointAlignmentBuffered
org.apache.flink.jobmanager.job.lastCheckpointDuration
org.apache.flink.jobmanager.job.lastCheckpointExternalPath
org.apache.flink.jobmanager.job.lastCheckpointRestoreTimestamp
org.apache.flink.jobmanager.job.lastCheckpointSize
org.apache.flink.jobmanager.job.numberOfCompletedCheckpoints
org.apache.flink.jobmanager.job.numberOfFailedCheckpoints
org.apache.flink.jobmanager.job.numberOfInProgressCheckpoints
org.apache.flink.jobmanager.job.restartingTime
org.apache.flink.jobmanager.job.totalNumberOfCheckpoints
org.apache.flink.jobmanager.job.uptime
{code}
> The JobManager doesn't report the cluster-level metrics
> ---
>
> Key: FLINK-10135
> URL: https://issues.apache.org/jira/browse/FLINK-10135
> Project: Flink
> Issue Type: Bug
> Components: JobManager, Metrics
>Affects Versions: 1.5.0, 1.6.0, 1.7.0
>Reporter: Joey Echeverria
>Assignee: vinoyang
>Priority: Critical
>
> In [the documentation for
> metrics|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/metrics.html#cluster]
> in the Flink 1.5.0 release, it says that the following metrics are reported
> by the JobManager:
> {noformat}
> numRegisteredTaskManagers
> numRunningJobs
> taskSlotsAvailable
> taskSlotsTotal
> {noformat}
> In the job manager REST endpoint
> ({{http://:8081/jobmanager/metrics}}), those metrics don't
> appear.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-7157) Provide support for asynchronous operations over connected streams
Pedro Miguel Rainho Chaves created FLINK-7157: - Summary: Provide support for asynchronous operations over connected streams Key: FLINK-7157 URL: https://issues.apache.org/jira/browse/FLINK-7157 Project: Flink Issue Type: New Feature Components: DataStream API Reporter: Pedro Miguel Rainho Chaves The asynchronous function provides a great way to process events in parallel, which can greatly increase the throughput. However, a user may want to process events in parallel based on a stored state created by another stream. For instance, one could have a stream of scripts (e.g. groovy scripts) and a stream of events. Processing the events by the scripts in a CoFlatMap greatly reduces the performance if those take a long time to execute, specially in cases with a high number of scripts. One could argue that increasing the parallelism of the CoFlatMap would solve the problem, however it would come at a very high resource cost. It would be nice to have a *CoAsyncFunction* which would allow one of the streams in a *connectedStream* to be processed in an asynchronous fashion. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
