Re: WELCOME to user@flink.apache.org
Hi, Thanks Li Shao, I got it. You can analyze the specific memory usage of metaspace. In fact, we have also encountered the same problem of running batch jobs in the session cluster, which resulted in metaspace growth due to the classloader. I have created a PR [1] for FLIP-32265 [2] for this and try to fix this issue. [1] https://github.com/apache/flink/pull/22718 [2] https://issues.apache.org/jira/browse/FLINK-32265 Best, Shammon FY On Tue, Jun 6, 2023 at 8:58 AM Li Shao wrote: > Hi Shammon, > > Thank you for your reply. My flink job is using batch mode. For streaming > mode I never see the increasing metaspace. > > > On Mon, Jun 5, 2023 at 5:55 PM Shammon FY wrote: > >> Hi Li Shao, >> >> Currently Flink will create a user classloader in JobManager for each job >> which can only be released by FullGC, I think this is why JVM metaspace is >> increasing, you can check it. >> Are you using session mode? I have a small question: Is your job SQL only >> without UDF or DataStream? Thanks >> >> Best, >> Shammon FY >> >> On Tue, Jun 6, 2023 at 4:27 AM Li Shao wrote: >> >>> Hi, >>> >>> Recently I noticed my job manager JVM metaspace is keeping increasing >>> for running batch flink jobs. I found similar stackoverflow post: >>> https://stackoverflow.com/questions/73184042/apache-flink-job-manager-node-runs-out-of-jvm-metaspace-quickly, >>> but there is no solution on this. I am wondering if flink can clean up the >>> job manager JVM metaspace periodically or it does not. Please suggest. >>> >>> Thanks, >>> Li >>> >>> Version: 1.14.4 Flink HA mode >>> JVM Metaspace: 1.88 GB / 2.00 GB >>> >>> JVM (Heap/Non-Heap) Memory >>> TypeCommittedUsedMaximum >>> Heap 6.00 GB 3.79 GB 6.00 GB >>> Non-Heap 2.34 GB 2.25 GB 3.23 GB >>> Outside JVM Memory >>> TypeCountUsedCapacity >>> Direct 927 86.9 MB 87.0 MB >>> Mapped 0 0 B 0 B >>> Garbage Collection >>> CollectorCountTime >>> G1_Young_Generation 1355 57139 >>> G1_Old_Generation 1 1325 >>> >>> On Mon, Jun 5, 2023 at 1:21 PM wrote: >>> Hi! This is the ezmlm program. I'm managing the user@flink.apache.org mailing list. Acknowledgment: I have added the address lsgreat12...@gmail.com to the user mailing list. Welcome to user@flink.apache.org! Please save this message so that you know the address you are subscribed under, in case you later want to unsubscribe or change your subscription address. --- Administrative commands for the user list --- I can handle administrative requests automatically. Please do not send them to the list address! Instead, send your message to the correct command address: To subscribe to the list, send a message to: To remove your address from the list, send a message to: Send mail to the following for info and FAQ for this list: Similar addresses exist for the digest list: To get messages 123 through 145 (a maximum of 100 per request), mail: To get an index with subject and author for messages 123-456 , mail: They are always returned as sets of 100, max 2000 per request, so you'll actually get 100-499. To receive all messages with the same subject as message 12345, send a short message to: The messages should contain one line or word of text to avoid being treated as sp@m, but I will ignore their content. Only the ADDRESS you send to is important. You can start a subscription for an alternate address, for example "john@host.domain", just add a hyphen and your address (with '=' instead of '@') after the command word: To stop subscription for this address, mail: In both cases, I'll send a confirmation message to that address. When you receive it, simply reply to it to complete your subscription. If despite following these instructions, you do not get the desired results, please contact my owner at user-ow...@flink.apache.org. Please be patient, my owner is a lot slower than I am ;-) --- Enclosed is a copy of the request I received. Return-Path: Received: (qmail 1410866 invoked by uid 116); 5 Jun 2023 20:20:57 - Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.94) with ESMTP; Mon, 05 Jun 2023 20:20:57 + Authentication-Results: apache.org; auth=none Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id 5CD4B1FF748 for >>> gmail@flink.apache.org>; Mon, 5 Jun 2023 20:20:57 + (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: 0.24 X-Spam-Level:
Re: WELCOME to user@flink.apache.org
Hi Li Shao, Currently Flink will create a user classloader in JobManager for each job which can only be released by FullGC, I think this is why JVM metaspace is increasing, you can check it. Are you using session mode? I have a small question: Is your job SQL only without UDF or DataStream? Thanks Best, Shammon FY On Tue, Jun 6, 2023 at 4:27 AM Li Shao wrote: > Hi, > > Recently I noticed my job manager JVM metaspace is keeping increasing for > running batch flink jobs. I found similar stackoverflow post: > https://stackoverflow.com/questions/73184042/apache-flink-job-manager-node-runs-out-of-jvm-metaspace-quickly, > but there is no solution on this. I am wondering if flink can clean up the > job manager JVM metaspace periodically or it does not. Please suggest. > > Thanks, > Li > > Version: 1.14.4 Flink HA mode > JVM Metaspace: 1.88 GB / 2.00 GB > > JVM (Heap/Non-Heap) Memory > TypeCommittedUsedMaximum > Heap 6.00 GB 3.79 GB 6.00 GB > Non-Heap 2.34 GB 2.25 GB 3.23 GB > Outside JVM Memory > TypeCountUsedCapacity > Direct 927 86.9 MB 87.0 MB > Mapped 0 0 B 0 B > Garbage Collection > CollectorCountTime > G1_Young_Generation 1355 57139 > G1_Old_Generation 1 1325 > > On Mon, Jun 5, 2023 at 1:21 PM wrote: > >> Hi! This is the ezmlm program. I'm managing the >> user@flink.apache.org mailing list. >> >> Acknowledgment: I have added the address >> >>lsgreat12...@gmail.com >> >> to the user mailing list. >> >> Welcome to user@flink.apache.org! >> >> Please save this message so that you know the address you are >> subscribed under, in case you later want to unsubscribe or change your >> subscription address. >> >> >> --- Administrative commands for the user list --- >> >> I can handle administrative requests automatically. Please >> do not send them to the list address! Instead, send >> your message to the correct command address: >> >> To subscribe to the list, send a message to: >> >> >> To remove your address from the list, send a message to: >> >> >> Send mail to the following for info and FAQ for this list: >> >> >> >> Similar addresses exist for the digest list: >> >> >> >> To get messages 123 through 145 (a maximum of 100 per request), mail: >> >> >> To get an index with subject and author for messages 123-456 , mail: >> >> >> They are always returned as sets of 100, max 2000 per request, >> so you'll actually get 100-499. >> >> To receive all messages with the same subject as message 12345, >> send a short message to: >> >> >> The messages should contain one line or word of text to avoid being >> treated as sp@m, but I will ignore their content. >> Only the ADDRESS you send to is important. >> >> You can start a subscription for an alternate address, >> for example "john@host.domain", just add a hyphen and your >> address (with '=' instead of '@') after the command word: >> >> >> To stop subscription for this address, mail: >> >> >> In both cases, I'll send a confirmation message to that address. When >> you receive it, simply reply to it to complete your subscription. >> >> If despite following these instructions, you do not get the >> desired results, please contact my owner at >> user-ow...@flink.apache.org. Please be patient, my owner is a >> lot slower than I am ;-) >> >> --- Enclosed is a copy of the request I received. >> >> Return-Path: >> Received: (qmail 1410866 invoked by uid 116); 5 Jun 2023 20:20:57 - >> Received: from spamproc1-he-de.apache.org (HELO >> spamproc1-he-de.apache.org) (116.203.196.100) >> by apache.org (qpsmtpd/0.94) with ESMTP; Mon, 05 Jun 2023 20:20:57 + >> Authentication-Results: apache.org; auth=none >> Received: from localhost (localhost [127.0.0.1]) >> by spamproc1-he-de.apache.org (ASF Mail Server at >> spamproc1-he-de.apache.org) with ESMTP id 5CD4B1FF748 >> for > gmail@flink.apache.org>; Mon, 5 Jun 2023 20:20:57 + (UTC) >> X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org >> X-Spam-Flag: NO >> X-Spam-Score: 0.24 >> X-Spam-Level: >> X-Spam-Status: No, score=0.24 tagged_above=-999 required=6.31 >> tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, >> DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, >> HTML_MESSAGE=0.2, >> RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H2=-0.001, >> SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001] >> autolearn=disabled >> Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); >> dkim=pass (2048-bit key) header.d=gmail.com >> Received: from mx1-ec2-va.apache.org ([116.203.227.195]) >> by localhost (spamproc1-he-de.apache.org [116.203.196.100]) >> (amavisd-new, port 10024) >> with ESMTP id 7fWybnrBQhFr >> for > gmail@flink.apache.org>; >> Mon, 5 Jun 2023 20:20:56 + (UTC) >> Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.217.51; >> helo=mail-vs1-f51.google.com; envelope-from=lsgreat12...@gmail.com; >>
Re: WELCOME to user@flink.apache.org
Hi, Recently I noticed my job manager JVM metaspace is keeping increasing for running batch flink jobs. I found similar stackoverflow post: https://stackoverflow.com/questions/73184042/apache-flink-job-manager-node-runs-out-of-jvm-metaspace-quickly, but there is no solution on this. I am wondering if flink can clean up the job manager JVM metaspace periodically or it does not. Please suggest. Thanks, Li Version: 1.14.4 Flink HA mode JVM Metaspace: 1.88 GB / 2.00 GB JVM (Heap/Non-Heap) Memory TypeCommittedUsedMaximum Heap 6.00 GB 3.79 GB 6.00 GB Non-Heap 2.34 GB 2.25 GB 3.23 GB Outside JVM Memory TypeCountUsedCapacity Direct 927 86.9 MB 87.0 MB Mapped 0 0 B 0 B Garbage Collection CollectorCountTime G1_Young_Generation 1355 57139 G1_Old_Generation 1 1325 On Mon, Jun 5, 2023 at 1:21 PM wrote: > Hi! This is the ezmlm program. I'm managing the > user@flink.apache.org mailing list. > > Acknowledgment: I have added the address > >lsgreat12...@gmail.com > > to the user mailing list. > > Welcome to user@flink.apache.org! > > Please save this message so that you know the address you are > subscribed under, in case you later want to unsubscribe or change your > subscription address. > > > --- Administrative commands for the user list --- > > I can handle administrative requests automatically. Please > do not send them to the list address! Instead, send > your message to the correct command address: > > To subscribe to the list, send a message to: > > > To remove your address from the list, send a message to: > > > Send mail to the following for info and FAQ for this list: > > > > Similar addresses exist for the digest list: > > > > To get messages 123 through 145 (a maximum of 100 per request), mail: > > > To get an index with subject and author for messages 123-456 , mail: > > > They are always returned as sets of 100, max 2000 per request, > so you'll actually get 100-499. > > To receive all messages with the same subject as message 12345, > send a short message to: > > > The messages should contain one line or word of text to avoid being > treated as sp@m, but I will ignore their content. > Only the ADDRESS you send to is important. > > You can start a subscription for an alternate address, > for example "john@host.domain", just add a hyphen and your > address (with '=' instead of '@') after the command word: > > > To stop subscription for this address, mail: > > > In both cases, I'll send a confirmation message to that address. When > you receive it, simply reply to it to complete your subscription. > > If despite following these instructions, you do not get the > desired results, please contact my owner at > user-ow...@flink.apache.org. Please be patient, my owner is a > lot slower than I am ;-) > > --- Enclosed is a copy of the request I received. > > Return-Path: > Received: (qmail 1410866 invoked by uid 116); 5 Jun 2023 20:20:57 - > Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) > (116.203.196.100) > by apache.org (qpsmtpd/0.94) with ESMTP; Mon, 05 Jun 2023 20:20:57 + > Authentication-Results: apache.org; auth=none > Received: from localhost (localhost [127.0.0.1]) > by spamproc1-he-de.apache.org (ASF Mail Server at > spamproc1-he-de.apache.org) with ESMTP id 5CD4B1FF748 > for gmail@flink.apache.org>; Mon, 5 Jun 2023 20:20:57 + (UTC) > X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org > X-Spam-Flag: NO > X-Spam-Score: 0.24 > X-Spam-Level: > X-Spam-Status: No, score=0.24 tagged_above=-999 required=6.31 > tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, > DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, > HTML_MESSAGE=0.2, > RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H2=-0.001, > SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001] > autolearn=disabled > Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); > dkim=pass (2048-bit key) header.d=gmail.com > Received: from mx1-ec2-va.apache.org ([116.203.227.195]) > by localhost (spamproc1-he-de.apache.org [116.203.196.100]) > (amavisd-new, port 10024) > with ESMTP id 7fWybnrBQhFr > for gmail@flink.apache.org>; > Mon, 5 Jun 2023 20:20:56 + (UTC) > Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.217.51; > helo=mail-vs1-f51.google.com; envelope-from=lsgreat12...@gmail.com; > receiver= > Received: from mail-vs1-f51.google.com (mail-vs1-f51.google.com > [209.85.217.51]) > by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) > with ESMTPS id CD21BBE717 > for gmail@flink.apache.org>; Mon, 5 Jun 2023 20:20:55 + (UTC) > Received: by mail-vs1-f51.google.com with SMTP id > ada2fe7eead31-439494cbfedso1268755137.3 > for gmail@flink.apache.org>; Mon, 05 Jun 2023 13:20:55 -0700 (PDT) > DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; >
Re: WELCOME to user@flink.apache.org
Hi Penny, When you complete step 1 and step 2, it means that you have subscribed to the User mailing list so you can post the email that you want to send to the User mailing list by performing step 3. I can see why the email can be confusing though. Best regards, Martijn On Sat, Mar 11, 2023 at 11:42 AM Penny Rastogi wrote: > Hi There, > As per the guidelines: > > If you’d like to post to the mailing list, you need to > >1. subscribe to the mailing list by sending an email to >user-subscr...@flink.apache.org, >2. confirm the subscription by replying to the confirmation email, and >3. send your email to user@flink.apache.org > > > I am sending the confirmation mail to user@flink.apache.org > > Please acknowledge. > > Regards, > Vallari > > -- Forwarded message - > From: > Date: Sat, Mar 11, 2023 at 4:08 PM > Subject: WELCOME to user@flink.apache.org > To: > > > Hi! This is the ezmlm program. I'm managing the > user@flink.apache.org mailing list. > > Acknowledgment: I have added the address > >walls.fl...@gmail.com > > to the user mailing list. > > Welcome to user@flink.apache.org! > > Please save this message so that you know the address you are > subscribed under, in case you later want to unsubscribe or change your > subscription address. > > > --- Administrative commands for the user list --- > > I can handle administrative requests automatically. Please > do not send them to the list address! Instead, send > your message to the correct command address: > > To subscribe to the list, send a message to: > > > To remove your address from the list, send a message to: > > > Send mail to the following for info and FAQ for this list: > > > > Similar addresses exist for the digest list: > > > > To get messages 123 through 145 (a maximum of 100 per request), mail: > > > To get an index with subject and author for messages 123-456 , mail: > > > They are always returned as sets of 100, max 2000 per request, > so you'll actually get 100-499. > > To receive all messages with the same subject as message 12345, > send a short message to: > > > The messages should contain one line or word of text to avoid being > treated as sp@m, but I will ignore their content. > Only the ADDRESS you send to is important. > > You can start a subscription for an alternate address, > for example "john@host.domain", just add a hyphen and your > address (with '=' instead of '@') after the command word: > > > To stop subscription for this address, mail: > > > In both cases, I'll send a confirmation message to that address. When > you receive it, simply reply to it to complete your subscription. > > If despite following these instructions, you do not get the > desired results, please contact my owner at > user-ow...@flink.apache.org. Please be patient, my owner is a > lot slower than I am ;-) > > --- Enclosed is a copy of the request I received. > > Return-Path: > Received: (qmail 3560623 invoked by uid 116); 11 Mar 2023 10:38:15 - > Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) > (95.217.134.168) > by apache.org (qpsmtpd/0.94) with ESMTP; Sat, 11 Mar 2023 10:38:15 + > Authentication-Results: apache.org; auth=none > Received: from localhost (localhost [127.0.0.1]) > by spamproc1-he-fi.apache.org (ASF Mail Server at > spamproc1-he-fi.apache.org) with ESMTP id C5D06C06DB > for gmail@flink.apache.org>; Sat, 11 Mar 2023 10:38:14 + (UTC) > X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org > X-Spam-Flag: NO > X-Spam-Score: 0 > X-Spam-Level: > X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 > tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, > DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, SPF_PASS=-0.001, > URIBL_BLOCKED=0.001] autolearn=disabled > Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); > dkim=pass (2048-bit key) header.d=gmail.com > Received: from mx1-he-de.apache.org ([116.203.227.195]) > by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) > (amavisd-new, port 10024) > with ESMTP id Ef2eCQzSFakt > for gmail@flink.apache.org>; > Sat, 11 Mar 2023 10:38:13 + (UTC) > Received-SPF: Pass (mailfrom) identity=mailfrom; > client-ip=2607:f8b0:4864:20::929; helo=mail-ua1-x929.google.com; > envelope-from=walls.fl...@gmail.com; receiver= > Received: from mail-ua1-x929.google.com (mail-ua1-x929.google.com > [IPv6:2607:f8b0:4864:20::929]) > by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) > with ESMTPS id 820C27D788 > for gmail@flink.apache.org>; Sat, 11 Mar 2023 10:38:13 + (UTC) > Received: by mail-ua1-x929.google.com with SMTP id v48so5207308uad.6 > for gmail@flink.apache.org>; Sat, 11 Mar 2023 02:38:13 -0800 (PST) > DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; > d=gmail.com; s=20210112; t=1678531092; > >
Re: WELCOME to user@flink.apache.org
Hey Karthikeyan,
Welcome to Flink.
Make sure that
1. IAM role has enough permissions to the required buckets. ( and
/*)
2.
3. Once you define the IAM role, you need to annotate the SA with the role
ARN and attach the SA to Flink pods. ("kubernetes.service-account" under
flink-conf.yaml
i.e
annotations = {
"eks.amazonaws.com/role-arn" =
}
4. Add aws-java-sdk-sts dependency(1.12.+) to job's jar. (to read the SA
properly).
From: Karthikeyan Muthusamy (karmuthu)
Sent: Tuesday, August 2, 2022 5:25 PM
To: user-h...@flink.apache.org ;
user@flink.apache.org
Cc: Mohan S G (mosg)
Subject: Re: WELCOME to user@flink.apache.org
EXTERNAL EMAIL
Hi Team,
We have deployed our Flink Cluster on AWS EKS using Flink Operator. We have
created required service accounts with IAM OIDC integration, however flink-main
container seems to bypass this service account role and directly tries to
create and delete objects in s3 using Node role. As the flink application fails
to use service account, the access is getting denied.
**Please note that we have enabled HA and our Storage Directory is s3 and if we
log into the pod and check, it does have the required permission at the
container level and are able to write and read from s3, however application is
not using the service account.
Error:
Caused by: java.nio.file.AccessDeniedException:
s3a://preint-us-east-1-flink/flink-peak-trunk-utilization/flink-deploy/submittedJobGraph50e30a08e280:
delete on
s3a://preint-us-east-1-flink/flink-peak-trunk-utilization/flink-deploy/submittedJobGraph50e30a08e280:
com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service:
Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID:
MCRDGVQN7X6EVJ69; S3 Extended Request ID:
+MwyWabe4upNDsWmEJeEOxtvRYCJDa840uk5AtLam0c3O4vnZMD3k4tlI+VU+o/o2JgO9GpMDYY=;
Proxy: null), S3 Extended Request ID:
+MwyWabe4upNDsWmEJeEOxtvRYCJDa840uk5AtLam0c3O4vnZMD3k4tlI+VU+o/o2JgO9GpMDYY=:AccessDenied
-- Karthikeyan
Technical Leader Engineering
Cisco Systems
Ph: 9019431391
From: user-h...@flink.apache.org
Date: Tuesday, 2 August 2022 at 7:46 PM
To: Karthikeyan Muthusamy (karmuthu)
Subject: WELCOME to user@flink.apache.org
Hi! This is the ezmlm program. I'm managing the
user@flink.apache.org mailing list.
Acknowledgment: I have added the address
karmu...@cisco.com
to the user mailing list.
Welcome to user@flink.apache.org!
Please save this message so that you know the address you are
subscribed under, in case you later want to unsubscribe or change your
subscription address.
--- Administrative commands for the user list ---
I can handle administrative requests automatically. Please
do not send them to the list address! Instead, send
your message to the correct command address:
To subscribe to the list, send a message to:
To remove your address from the list, send a message to:
Send mail to the following for info and FAQ for this list:
Similar addresses exist for the digest list:
To get messages 123 through 145 (a maximum of 100 per request), mail:
To get an index with subject and author for messages 123-456 , mail:
They are always returned as sets of 100, max 2000 per request,
so you'll actually get 100-499.
To receive all messages with the same subject as message 12345,
send a short message to:
The messages should contain one line or word of text to avoid being
treated as sp@m, but I will ignore their content.
Only the ADDRESS you send to is important.
You can start a subscription for an alternate address,
for example "john@host.domain", just add a hyphen and your
address (with '=' instead of '@') after the command word:
To stop subscription for this address, mail:
In both cases, I'll send a confirmation message to that address. When
you receive it, simply reply to it to complete your subscription.
If despite following these instructions, you do not get the
desired results, please contact my owner at
user-ow...@flink.apache.org. Please be patient, my owner is a
lot slower than I am ;-)
--- Enclosed is a copy of the request I received.
Return-Path:
Received: (qmail 3559598 invoked by uid 116); 2 Aug 2022 14:16:38 -
Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org)
(116.203.196.100)
by apache.org (qpsmtpd/0.94) with ESMTP; Tue, 02 Aug 2022 14:16:38 +
Authentication-Results: apache.org; auth=none
Received: from localhost (localhost [127.0.0.1])
by spamproc1-he-de.apache.org (ASF Mail Server at
spamproc1-he-de.apache.org) with ESMTP id 7573D1FF613
for
;
Tue, 2 Aug 2022 14:16:38 + (UTC)
X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org
X-Spam-Flag: NO
X-Spam-Score: -7.51
X-Spam-Level:
X-Spam-Status: No, score=-7.51 tagged_above=-999 required=6.31
tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,
DK
Re: WELCOME to user@flink.apache.org
Hi Team, We have deployed our Flink Cluster on AWS EKS using Flink Operator. We have created required service accounts with IAM OIDC integration, however flink-main container seems to bypass this service account role and directly tries to create and delete objects in s3 using Node role. As the flink application fails to use service account, the access is getting denied. **Please note that we have enabled HA and our Storage Directory is s3 and if we log into the pod and check, it does have the required permission at the container level and are able to write and read from s3, however application is not using the service account. Error: Caused by: java.nio.file.AccessDeniedException: s3a://preint-us-east-1-flink/flink-peak-trunk-utilization/flink-deploy/submittedJobGraph50e30a08e280: delete on s3a://preint-us-east-1-flink/flink-peak-trunk-utilization/flink-deploy/submittedJobGraph50e30a08e280: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: MCRDGVQN7X6EVJ69; S3 Extended Request ID: +MwyWabe4upNDsWmEJeEOxtvRYCJDa840uk5AtLam0c3O4vnZMD3k4tlI+VU+o/o2JgO9GpMDYY=; Proxy: null), S3 Extended Request ID: +MwyWabe4upNDsWmEJeEOxtvRYCJDa840uk5AtLam0c3O4vnZMD3k4tlI+VU+o/o2JgO9GpMDYY=:AccessDenied -- Karthikeyan Technical Leader Engineering Cisco Systems Ph: 9019431391 From: user-h...@flink.apache.org Date: Tuesday, 2 August 2022 at 7:46 PM To: Karthikeyan Muthusamy (karmuthu) Subject: WELCOME to user@flink.apache.org Hi! This is the ezmlm program. I'm managing the user@flink.apache.org mailing list. Acknowledgment: I have added the address karmu...@cisco.com to the user mailing list. Welcome to user@flink.apache.org! Please save this message so that you know the address you are subscribed under, in case you later want to unsubscribe or change your subscription address. --- Administrative commands for the user list --- I can handle administrative requests automatically. Please do not send them to the list address! Instead, send your message to the correct command address: To subscribe to the list, send a message to: To remove your address from the list, send a message to: Send mail to the following for info and FAQ for this list: Similar addresses exist for the digest list: To get messages 123 through 145 (a maximum of 100 per request), mail: To get an index with subject and author for messages 123-456 , mail: They are always returned as sets of 100, max 2000 per request, so you'll actually get 100-499. To receive all messages with the same subject as message 12345, send a short message to: The messages should contain one line or word of text to avoid being treated as sp@m, but I will ignore their content. Only the ADDRESS you send to is important. You can start a subscription for an alternate address, for example "john@host.domain", just add a hyphen and your address (with '=' instead of '@') after the command word: To stop subscription for this address, mail: In both cases, I'll send a confirmation message to that address. When you receive it, simply reply to it to complete your subscription. If despite following these instructions, you do not get the desired results, please contact my owner at user-ow...@flink.apache.org. Please be patient, my owner is a lot slower than I am ;-) --- Enclosed is a copy of the request I received. Return-Path: Received: (qmail 3559598 invoked by uid 116); 2 Aug 2022 14:16:38 - Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.94) with ESMTP; Tue, 02 Aug 2022 14:16:38 + Authentication-Results: apache.org; auth=none Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id 7573D1FF613 for ; Tue, 2 Aug 2022 14:16:38 + (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: -7.51 X-Spam-Level: X-Spam-Status: No, score=-7.51 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=cisco.com header.b=BtLzGoZh; dkim=pass (1024-bit key) header.d=cisco.onmicrosoft.com header.b=QVxJ3PPM Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id oXPDzA0B_LlH for ; Tue, 2 Aug 2022 14:16:36 + (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=173.37.142.91; helo=alln-iport-4.cisco.com;
Re: WELCOME to user@flink.apache.org
Hi Wanghui, unfortunately, this is not supported to my knowledge. See also this similar question on Stackoverflow: https://stackoverflow.com/questions/60950594/flink-encryption-parameters-in-flink-conf-yaml Best regards, Nico On Mon, Jul 5, 2021 at 3:45 PM Wanghui (HiCampus) wrote: > Hello, I find that security.ssl.rest.enabled: true is configured in flink, > and the Java keystore password is stored in plaintext in the configuration > file. Can the keystore password be encrypted for storage? > > >
