RE: SecurityManager in Flink

Hi Gabor, The issue is that, read permission is not getting checked when Flink FileSource is listing the files under given source directory. This is happening as Security Manager is coming as null. public String[] list() { SecurityManager security = System.getSecurityManager(); -> Here

RE: SecurityManager in Flink

Hi Yanfei, I am facing this issue on jdk1.8/11. Thanks for giving pointer, I will try to set Security manager and check the behaviour. Regards, Kirti Dhar -Original Message- From: Yanfei Lei Sent: Wednesday, March 6, 2024 4:37 PM To: Kirti Dhar Upadhyay K Cc: User@flink.apache.org

RE: SecurityManager in Flink

Hi Hang, You got it right. The problem is exactly at the same line where you pointed [1]. I have used below solution as of now. ``` If(!Files.isReadable(Paths.get(fileStatus.getPath().getPath( { throw new FlinkRuntimeException("Cannot list files under " + fileStatus.getPath()); }

Re:Reading Iceberg Tables in Batch mode performance

Hi, can you provide more details about this Flink batch job? For instance, through a flame graph, the threads are found spending most of their time on some certain tasks. -- Best！ Xuyang At 2024-03-07 08:40:32, "Charles Tan" wrote: Hi all, I have been looking into using

Re:Running Flink SQL in production

Hi, IMO, both the SQL Client and the Restful API can provide connections to the SQL Gateway service for submitting jobs. A slight difference is that the SQL Client also offers a command-line visual interface for users to view results. In your production scenes, placing the SQL to be submitted

Reading Iceberg Tables in Batch mode performance

Hi all, I have been looking into using Flink in batch mode to process Iceberg tables. I noticed that the performance for queries in Flink's batch mode is quite slow, especially when compared to Spark. I'm wondering if there are any configurations that I'm missing to get better performance out of

Running Flink SQL in production

I'm reading the deployment guide[1] and wanted to check my understanding. For deploying a SQL job into production, would the pattern be to write the SQL in a file that's under source control, and pass that file as an argument to SQL Client with -f argument (as in this docs example[2])? Or script a

Re: SecurityManager in Flink

Hi Kirti, Not sure what is the exact issue here but I'm not convinced that having FlinkSecurityManager is going to solve it. Here is the condition however: * cluster.intercept-user-system-exit != DISABLED (this must be changed) * cluster.processes.halt-on-fatal-error == false (this is good by

Re: SecurityManager in Flink

Hi, Kirti. Could you please provide the stack trace of this NPE? I check the code and I think maybe the problem lies in LocalFileSystem#listStatus. The code in line 161[1] may return null, which will let LocalFileSystem#listStatus return null. Then the `containedFiles` is null and the NPE occurs.

Re: SecurityManager in Flink

Hi Kirti Dhar, What is your java version? I guess this problem may be related to FLINK-33309[1]. Maybe you can try adding "-Djava.security.manager" to the java options. [1] https://issues.apache.org/jira/browse/FLINK-33309 Kirti Dhar Upadhyay K via user 于2024年3月6日周三 18:10写道： > > Hi Team, > > >

Re: Question about time-based operators with RocksDB backend

Hi Gabriele, use (or extend) the window operator provided by Flink is a better idea. A window operator in Flink manages two types of state: - Window state: accumlate data for windows, and provide data to window function when a window comes to its end time. - Timer state: store the end

SecurityManager in Flink

Hi Team, I am using Flink File Source with Local File System. I am facing an issue, if source directory does not has read permission, it is returning the list of files as null instead of throwing permission exception (refer the highlighted line below), resulting in NPE. final FileStatus[]