Hi, Thanks Li Shao, I got it. You can analyze the specific memory usage of metaspace. In fact, we have also encountered the same problem of running batch jobs in the session cluster, which resulted in metaspace growth due to the classloader. I have created a PR [1] for FLIP-32265 [2] for this and try to fix this issue.
[1] https://github.com/apache/flink/pull/22718 [2] https://issues.apache.org/jira/browse/FLINK-32265 Best, Shammon FY On Tue, Jun 6, 2023 at 8:58 AM Li Shao <lsgreat12...@gmail.com> wrote: > Hi Shammon, > > Thank you for your reply. My flink job is using batch mode. For streaming > mode I never see the increasing metaspace. > > > On Mon, Jun 5, 2023 at 5:55 PM Shammon FY <zjur...@gmail.com> wrote: > >> Hi Li Shao, >> >> Currently Flink will create a user classloader in JobManager for each job >> which can only be released by FullGC, I think this is why JVM metaspace is >> increasing, you can check it. >> Are you using session mode? I have a small question: Is your job SQL only >> without UDF or DataStream? Thanks >> >> Best, >> Shammon FY >> >> On Tue, Jun 6, 2023 at 4:27 AM Li Shao <lsgreat12...@gmail.com> wrote: >> >>> Hi, >>> >>> Recently I noticed my job manager JVM metaspace is keeping increasing >>> for running batch flink jobs. I found similar stackoverflow post: >>> https://stackoverflow.com/questions/73184042/apache-flink-job-manager-node-runs-out-of-jvm-metaspace-quickly, >>> but there is no solution on this. I am wondering if flink can clean up the >>> job manager JVM metaspace periodically or it does not. Please suggest. >>> >>> Thanks, >>> Li >>> >>> Version: 1.14.4 Flink HA mode >>> JVM Metaspace: 1.88 GB / 2.00 GB >>> >>> JVM (Heap/Non-Heap) Memory >>> TypeCommittedUsedMaximum >>> Heap 6.00 GB 3.79 GB 6.00 GB >>> Non-Heap 2.34 GB 2.25 GB 3.23 GB >>> Outside JVM Memory >>> TypeCountUsedCapacity >>> Direct 927 86.9 MB 87.0 MB >>> Mapped 0 0 B 0 B >>> Garbage Collection >>> CollectorCountTime >>> G1_Young_Generation 1355 57139 >>> G1_Old_Generation 1 1325 >>> >>> On Mon, Jun 5, 2023 at 1:21 PM <user-h...@flink.apache.org> wrote: >>> >>>> Hi! This is the ezmlm program. I'm managing the >>>> user@flink.apache.org mailing list. >>>> >>>> Acknowledgment: I have added the address >>>> >>>> lsgreat12...@gmail.com >>>> >>>> to the user mailing list. >>>> >>>> Welcome to user@flink.apache.org! >>>> >>>> Please save this message so that you know the address you are >>>> subscribed under, in case you later want to unsubscribe or change your >>>> subscription address. >>>> >>>> >>>> --- Administrative commands for the user list --- >>>> >>>> I can handle administrative requests automatically. Please >>>> do not send them to the list address! Instead, send >>>> your message to the correct command address: >>>> >>>> To subscribe to the list, send a message to: >>>> <user-subscr...@flink.apache.org> >>>> >>>> To remove your address from the list, send a message to: >>>> <user-unsubscr...@flink.apache.org> >>>> >>>> Send mail to the following for info and FAQ for this list: >>>> <user-i...@flink.apache.org> >>>> <user-...@flink.apache.org> >>>> >>>> Similar addresses exist for the digest list: >>>> <user-digest-subscr...@flink.apache.org> >>>> <user-digest-unsubscr...@flink.apache.org> >>>> >>>> To get messages 123 through 145 (a maximum of 100 per request), mail: >>>> <user-get.123_...@flink.apache.org> >>>> >>>> To get an index with subject and author for messages 123-456 , mail: >>>> <user-index.123_...@flink.apache.org> >>>> >>>> They are always returned as sets of 100, max 2000 per request, >>>> so you'll actually get 100-499. >>>> >>>> To receive all messages with the same subject as message 12345, >>>> send a short message to: >>>> <user-thread.12...@flink.apache.org> >>>> >>>> The messages should contain one line or word of text to avoid being >>>> treated as sp@m, but I will ignore their content. >>>> Only the ADDRESS you send to is important. >>>> >>>> You can start a subscription for an alternate address, >>>> for example "john@host.domain", just add a hyphen and your >>>> address (with '=' instead of '@') after the command word: >>>> <user-subscribe-john=host.dom...@flink.apache.org> >>>> >>>> To stop subscription for this address, mail: >>>> <user-unsubscribe-john=host.dom...@flink.apache.org> >>>> >>>> In both cases, I'll send a confirmation message to that address. When >>>> you receive it, simply reply to it to complete your subscription. >>>> >>>> If despite following these instructions, you do not get the >>>> desired results, please contact my owner at >>>> user-ow...@flink.apache.org. Please be patient, my owner is a >>>> lot slower than I am ;-) >>>> >>>> --- Enclosed is a copy of the request I received. >>>> >>>> Return-Path: <lsgreat12...@gmail.com> >>>> Received: (qmail 1410866 invoked by uid 116); 5 Jun 2023 20:20:57 -0000 >>>> Received: from spamproc1-he-de.apache.org (HELO >>>> spamproc1-he-de.apache.org) (116.203.196.100) >>>> by apache.org (qpsmtpd/0.94) with ESMTP; Mon, 05 Jun 2023 20:20:57 >>>> +0000 >>>> Authentication-Results: apache.org; auth=none >>>> Received: from localhost (localhost [127.0.0.1]) >>>> by spamproc1-he-de.apache.org (ASF Mail Server at >>>> spamproc1-he-de.apache.org) with ESMTP id 5CD4B1FF748 >>>> for <user-sc.1685996278.imlfpciahckpkkgnmmod-lsgreat12800= >>>> gmail....@flink.apache.org>; Mon, 5 Jun 2023 20:20:57 +0000 (UTC) >>>> X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org >>>> X-Spam-Flag: NO >>>> X-Spam-Score: 0.24 >>>> X-Spam-Level: >>>> X-Spam-Status: No, score=0.24 tagged_above=-999 required=6.31 >>>> tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, >>>> DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, >>>> HTML_MESSAGE=0.2, >>>> RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H2=-0.001, >>>> SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, >>>> URIBL_BLOCKED=0.001] >>>> autolearn=disabled >>>> Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); >>>> dkim=pass (2048-bit key) header.d=gmail.com >>>> Received: from mx1-ec2-va.apache.org ([116.203.227.195]) >>>> by localhost (spamproc1-he-de.apache.org [116.203.196.100]) >>>> (amavisd-new, port 10024) >>>> with ESMTP id 7fWybnrBQhFr >>>> for <user-sc.1685996278.imlfpciahckpkkgnmmod-lsgreat12800= >>>> gmail....@flink.apache.org>; >>>> Mon, 5 Jun 2023 20:20:56 +0000 (UTC) >>>> Received-SPF: Pass (mailfrom) identity=mailfrom; >>>> client-ip=209.85.217.51; helo=mail-vs1-f51.google.com; envelope-from= >>>> lsgreat12...@gmail.com; receiver=<UNKNOWN> >>>> Received: from mail-vs1-f51.google.com (mail-vs1-f51.google.com >>>> [209.85.217.51]) >>>> by mx1-ec2-va.apache.org (ASF Mail Server at >>>> mx1-ec2-va.apache.org) with ESMTPS id CD21BBE717 >>>> for <user-sc.1685996278.imlfpciahckpkkgnmmod-lsgreat12800= >>>> gmail....@flink.apache.org>; Mon, 5 Jun 2023 20:20:55 +0000 (UTC) >>>> Received: by mail-vs1-f51.google.com with SMTP id >>>> ada2fe7eead31-439494cbfedso1268755137.3 >>>> for <user-sc.1685996278.imlfpciahckpkkgnmmod-lsgreat12800= >>>> gmail....@flink.apache.org>; Mon, 05 Jun 2023 13:20:55 -0700 (PDT) >>>> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; >>>> d=gmail.com; s=20221208; t=1685996455; x=1688588455; >>>> >>>> h=to:subject:message-id:date:from:in-reply-to:references:mime-version >>>> :from:to:cc:subject:date:message-id:reply-to; >>>> bh=BOoYAwNr7SDyHoaXPVhbv/HOz6DsmbwaDHSeUUi3RNE=; >>>> >>>> b=gRIWONJ7vQ5t55/MBE8dEazMMIlpeceF7xjxzxE5ZAIBSkwjCKcCvLb3wHfSunk8R7 >>>> >>>> xKIck1zp2/OngGTRx0S0snAKKzkt0fxHD+y67mtcwA/A652AwXBZ+1+d4nBn0xy8xKiH >>>> >>>> VC+9NXNa3OKz7AsmxveA/OorOtC0CkoBcDLdjM+7TYOIh/LF+1p2DFuXD3kZDJv4V4Rl >>>> >>>> Wmft3mlraOE3k5O+1NnLOIVqlNKS8cBxFEuAQ/n/OiPJjhBxiDMSaMfo3DaGxgE9RPTh >>>> >>>> QTtu79ZoOjuUCKpIs2dbRHaxqC+GM75W40iSbl5NRbckjIOyiVypsmzcjLy/Y7JT7GXq >>>> Z1Kg== >>>> X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; >>>> d=1e100.net; s=20221208; t=1685996455; x=1688588455; >>>> >>>> h=to:subject:message-id:date:from:in-reply-to:references:mime-version >>>> >>>> :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; >>>> bh=BOoYAwNr7SDyHoaXPVhbv/HOz6DsmbwaDHSeUUi3RNE=; >>>> >>>> b=dxWDgXBWtSil8bzaDE4fZ2NCJ4NTCQYbgGJ5SI/WFCvAxZ8+az/cAUBUGb1wB0NbiQ >>>> >>>> 3Z/pbc6XcOQlOYiOb2sV43Ft3Nd8eQzMwok1GgakE7IiFOPwHNjSQrab+lwd1tcMSFzn >>>> >>>> z/DCNoLgIzFyUt9nFQkbCX3BK/9fXJd4dsz17+lKUCNrA5cM/YHQlO99jVc8LpaGMXJz >>>> >>>> k/WO4jonGgVg/Bvh85uwzUcQ3p9NsEXwztA2noW+vbKL0ZnNvRvc43YWHGzVPmgjLyUe >>>> >>>> QazCeSaHIQzIv+EPKXYuew4R4zVsslk2ZfVWfoziPQqde3rQzYcY18T2n9qVw2ZQiS1k >>>> 1FAA== >>>> X-Gm-Message-State: >>>> AC+VfDyt4DvFW/idna3uvRQt/DUgY3EyGfjKtrIXhcQmaDFJJnRYC+E2 >>>> 1+dbbePF34KaTlXtulVHI8De7Es3xi49/SOkv6zhrW8A >>>> X-Google-Smtp-Source: >>>> ACHHUZ4TA9HL20BJkplv+aKF4JIgFzMTstEbk8wBibKC6A+pgMtL2xR1vncOySB2OWiQde0+VMHTfK5z4o/RK9pM1+4= >>>> X-Received: by 2002:a67:ee11:0:b0:434:66e3:74b4 with SMTP id >>>> f17-20020a67ee11000000b0043466e374b4mr237678vsp.8.1685996454892; Mon, >>>> 05 Jun >>>> 2023 13:20:54 -0700 (PDT) >>>> MIME-Version: 1.0 >>>> References: <1685996278.1401487.ez...@flink.apache.org> >>>> In-Reply-To: <1685996278.1401487.ez...@flink.apache.org> >>>> From: Li Shao <lsgreat12...@gmail.com> >>>> Date: Mon, 5 Jun 2023 13:20:43 -0700 >>>> Message-ID: < >>>> cap8bux+wbk4kqs0nbobx9zx6m2u4j4_qcdfbeirvotj3mcm...@mail.gmail.com> >>>> Subject: Re: confirm subscribe to user@flink.apache.org >>>> To: user-sc.1685996278.imlfpciahckpkkgnmmod-lsgreat12800= >>>> gmail....@flink.apache.org >>>> Content-Type: multipart/alternative; >>>> boundary="000000000000c9587405fd67a37b" >>>> >>>>
