Hi Li Shao, Currently Flink will create a user classloader in JobManager for each job which can only be released by FullGC, I think this is why JVM metaspace is increasing, you can check it. Are you using session mode? I have a small question: Is your job SQL only without UDF or DataStream? Thanks
Best, Shammon FY On Tue, Jun 6, 2023 at 4:27 AM Li Shao <lsgreat12...@gmail.com> wrote: > Hi, > > Recently I noticed my job manager JVM metaspace is keeping increasing for > running batch flink jobs. I found similar stackoverflow post: > https://stackoverflow.com/questions/73184042/apache-flink-job-manager-node-runs-out-of-jvm-metaspace-quickly, > but there is no solution on this. I am wondering if flink can clean up the > job manager JVM metaspace periodically or it does not. Please suggest. > > Thanks, > Li > > Version: 1.14.4 Flink HA mode > JVM Metaspace: 1.88 GB / 2.00 GB > > JVM (Heap/Non-Heap) Memory > TypeCommittedUsedMaximum > Heap 6.00 GB 3.79 GB 6.00 GB > Non-Heap 2.34 GB 2.25 GB 3.23 GB > Outside JVM Memory > TypeCountUsedCapacity > Direct 927 86.9 MB 87.0 MB > Mapped 0 0 B 0 B > Garbage Collection > CollectorCountTime > G1_Young_Generation 1355 57139 > G1_Old_Generation 1 1325 > > On Mon, Jun 5, 2023 at 1:21 PM <user-h...@flink.apache.org> wrote: > >> Hi! This is the ezmlm program. I'm managing the >> user@flink.apache.org mailing list. >> >> Acknowledgment: I have added the address >> >> lsgreat12...@gmail.com >> >> to the user mailing list. >> >> Welcome to user@flink.apache.org! >> >> Please save this message so that you know the address you are >> subscribed under, in case you later want to unsubscribe or change your >> subscription address. >> >> >> --- Administrative commands for the user list --- >> >> I can handle administrative requests automatically. Please >> do not send them to the list address! Instead, send >> your message to the correct command address: >> >> To subscribe to the list, send a message to: >> <user-subscr...@flink.apache.org> >> >> To remove your address from the list, send a message to: >> <user-unsubscr...@flink.apache.org> >> >> Send mail to the following for info and FAQ for this list: >> <user-i...@flink.apache.org> >> <user-...@flink.apache.org> >> >> Similar addresses exist for the digest list: >> <user-digest-subscr...@flink.apache.org> >> <user-digest-unsubscr...@flink.apache.org> >> >> To get messages 123 through 145 (a maximum of 100 per request), mail: >> <user-get.123_...@flink.apache.org> >> >> To get an index with subject and author for messages 123-456 , mail: >> <user-index.123_...@flink.apache.org> >> >> They are always returned as sets of 100, max 2000 per request, >> so you'll actually get 100-499. >> >> To receive all messages with the same subject as message 12345, >> send a short message to: >> <user-thread.12...@flink.apache.org> >> >> The messages should contain one line or word of text to avoid being >> treated as sp@m, but I will ignore their content. >> Only the ADDRESS you send to is important. >> >> You can start a subscription for an alternate address, >> for example "john@host.domain", just add a hyphen and your >> address (with '=' instead of '@') after the command word: >> <user-subscribe-john=host.dom...@flink.apache.org> >> >> To stop subscription for this address, mail: >> <user-unsubscribe-john=host.dom...@flink.apache.org> >> >> In both cases, I'll send a confirmation message to that address. When >> you receive it, simply reply to it to complete your subscription. >> >> If despite following these instructions, you do not get the >> desired results, please contact my owner at >> user-ow...@flink.apache.org. Please be patient, my owner is a >> lot slower than I am ;-) >> >> --- Enclosed is a copy of the request I received. >> >> Return-Path: <lsgreat12...@gmail.com> >> Received: (qmail 1410866 invoked by uid 116); 5 Jun 2023 20:20:57 -0000 >> Received: from spamproc1-he-de.apache.org (HELO >> spamproc1-he-de.apache.org) (116.203.196.100) >> by apache.org (qpsmtpd/0.94) with ESMTP; Mon, 05 Jun 2023 20:20:57 +0000 >> Authentication-Results: apache.org; auth=none >> Received: from localhost (localhost [127.0.0.1]) >> by spamproc1-he-de.apache.org (ASF Mail Server at >> spamproc1-he-de.apache.org) with ESMTP id 5CD4B1FF748 >> for <user-sc.1685996278.imlfpciahckpkkgnmmod-lsgreat12800= >> gmail....@flink.apache.org>; Mon, 5 Jun 2023 20:20:57 +0000 (UTC) >> X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org >> X-Spam-Flag: NO >> X-Spam-Score: 0.24 >> X-Spam-Level: >> X-Spam-Status: No, score=0.24 tagged_above=-999 required=6.31 >> tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, >> DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, >> HTML_MESSAGE=0.2, >> RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H2=-0.001, >> SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001] >> autolearn=disabled >> Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); >> dkim=pass (2048-bit key) header.d=gmail.com >> Received: from mx1-ec2-va.apache.org ([116.203.227.195]) >> by localhost (spamproc1-he-de.apache.org [116.203.196.100]) >> (amavisd-new, port 10024) >> with ESMTP id 7fWybnrBQhFr >> for <user-sc.1685996278.imlfpciahckpkkgnmmod-lsgreat12800= >> gmail....@flink.apache.org>; >> Mon, 5 Jun 2023 20:20:56 +0000 (UTC) >> Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.217.51; >> helo=mail-vs1-f51.google.com; envelope-from=lsgreat12...@gmail.com; >> receiver=<UNKNOWN> >> Received: from mail-vs1-f51.google.com (mail-vs1-f51.google.com >> [209.85.217.51]) >> by mx1-ec2-va.apache.org (ASF Mail Server at >> mx1-ec2-va.apache.org) with ESMTPS id CD21BBE717 >> for <user-sc.1685996278.imlfpciahckpkkgnmmod-lsgreat12800= >> gmail....@flink.apache.org>; Mon, 5 Jun 2023 20:20:55 +0000 (UTC) >> Received: by mail-vs1-f51.google.com with SMTP id >> ada2fe7eead31-439494cbfedso1268755137.3 >> for <user-sc.1685996278.imlfpciahckpkkgnmmod-lsgreat12800= >> gmail....@flink.apache.org>; Mon, 05 Jun 2023 13:20:55 -0700 (PDT) >> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; >> d=gmail.com; s=20221208; t=1685996455; x=1688588455; >> >> h=to:subject:message-id:date:from:in-reply-to:references:mime-version >> :from:to:cc:subject:date:message-id:reply-to; >> bh=BOoYAwNr7SDyHoaXPVhbv/HOz6DsmbwaDHSeUUi3RNE=; >> >> b=gRIWONJ7vQ5t55/MBE8dEazMMIlpeceF7xjxzxE5ZAIBSkwjCKcCvLb3wHfSunk8R7 >> >> xKIck1zp2/OngGTRx0S0snAKKzkt0fxHD+y67mtcwA/A652AwXBZ+1+d4nBn0xy8xKiH >> >> VC+9NXNa3OKz7AsmxveA/OorOtC0CkoBcDLdjM+7TYOIh/LF+1p2DFuXD3kZDJv4V4Rl >> >> Wmft3mlraOE3k5O+1NnLOIVqlNKS8cBxFEuAQ/n/OiPJjhBxiDMSaMfo3DaGxgE9RPTh >> >> QTtu79ZoOjuUCKpIs2dbRHaxqC+GM75W40iSbl5NRbckjIOyiVypsmzcjLy/Y7JT7GXq >> Z1Kg== >> X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; >> d=1e100.net; s=20221208; t=1685996455; x=1688588455; >> >> h=to:subject:message-id:date:from:in-reply-to:references:mime-version >> :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; >> bh=BOoYAwNr7SDyHoaXPVhbv/HOz6DsmbwaDHSeUUi3RNE=; >> >> b=dxWDgXBWtSil8bzaDE4fZ2NCJ4NTCQYbgGJ5SI/WFCvAxZ8+az/cAUBUGb1wB0NbiQ >> >> 3Z/pbc6XcOQlOYiOb2sV43Ft3Nd8eQzMwok1GgakE7IiFOPwHNjSQrab+lwd1tcMSFzn >> >> z/DCNoLgIzFyUt9nFQkbCX3BK/9fXJd4dsz17+lKUCNrA5cM/YHQlO99jVc8LpaGMXJz >> >> k/WO4jonGgVg/Bvh85uwzUcQ3p9NsEXwztA2noW+vbKL0ZnNvRvc43YWHGzVPmgjLyUe >> >> QazCeSaHIQzIv+EPKXYuew4R4zVsslk2ZfVWfoziPQqde3rQzYcY18T2n9qVw2ZQiS1k >> 1FAA== >> X-Gm-Message-State: >> AC+VfDyt4DvFW/idna3uvRQt/DUgY3EyGfjKtrIXhcQmaDFJJnRYC+E2 >> 1+dbbePF34KaTlXtulVHI8De7Es3xi49/SOkv6zhrW8A >> X-Google-Smtp-Source: >> ACHHUZ4TA9HL20BJkplv+aKF4JIgFzMTstEbk8wBibKC6A+pgMtL2xR1vncOySB2OWiQde0+VMHTfK5z4o/RK9pM1+4= >> X-Received: by 2002:a67:ee11:0:b0:434:66e3:74b4 with SMTP id >> f17-20020a67ee11000000b0043466e374b4mr237678vsp.8.1685996454892; Mon, 05 >> Jun >> 2023 13:20:54 -0700 (PDT) >> MIME-Version: 1.0 >> References: <1685996278.1401487.ez...@flink.apache.org> >> In-Reply-To: <1685996278.1401487.ez...@flink.apache.org> >> From: Li Shao <lsgreat12...@gmail.com> >> Date: Mon, 5 Jun 2023 13:20:43 -0700 >> Message-ID: < >> cap8bux+wbk4kqs0nbobx9zx6m2u4j4_qcdfbeirvotj3mcm...@mail.gmail.com> >> Subject: Re: confirm subscribe to user@flink.apache.org >> To: user-sc.1685996278.imlfpciahckpkkgnmmod-lsgreat12800= >> gmail....@flink.apache.org >> Content-Type: multipart/alternative; >> boundary="000000000000c9587405fd67a37b" >> >>
