Re: Choosing compression codec when using parquet file sink

Hi, Tim. If you look at the doc here https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/filesystem/#format-types-1 , you just need to write a cu

Re: Flink K8s operator pod section of CRD

Yep! Simple oversight, it was :/ Cheers, Matyas On Thu, Feb 23, 2023 at 10:54 PM Gyula Fóra wrote: > Hey! > You are right, these fields could have been of the PodTemplate / > PodTemplateSpec type (probably PodTemplateSpec is actually better). > > I think the reason why we used it is two fold:

Choosing compression codec when using parquet file sink

I'm writing a Flink processor that will read a bunch of JSON records from Kafka and then write them to S3 in parquet format using the FileSink. I've got most things in place, the only thing I haven't been able to figure out is how to change the compression codec used by the writer. Is there any rec

Re: 我使用 start-cluster.sh 运行的flink，我的jar运行一段时间后被关闭了，这个时间是不固定的

我修改为单线程，仍然报相同的结果 发自我的iPhone -- 原始邮件 -- 发件人: Shammon FY

RE: Re: Inconsistent data format of flink-training-repo and learn-flink doc

Hi Yuxia, I made a new clean pull request [1] to resolve it, as the previous commit was a little messy with extra markdown formatting changes. [1] https://github.com/apache/flink/pull/22016 On 2023/02/22 02:40:23 yuxia wrote: > Seems in this pr[1], startTime and endTIme have been replaced wi

Re: RE: Re: Re: Should we always mark ValueState as "transient" forRichFunctions

稍等，我看一下您的反馈 发自我的iPhone -- Original -- From: Zhongpu Chen https://github.com/apache/flink-training/blob/master/rides-and-fares/src/solution/java/org/apache/flink/training/solutions/ridesandfares/RidesAndFaresSolution.java > > > > > > > > [2] > > > > > > > >

RE: Re: Re: Should we always mark ValueState as "transient" for RichFunctions

Hi Shammon, Sorry for the inaccurate description of my last reply. Let me restate my question again: Fact 1: we know that ValueState here should not serialized/de-serialized, so it is a good practice to mark it with "transient". Fact 2: on the other hand, if we don't mark it with "transien

Re: Re: Should we always mark ValueState as "transient" for RichFunctions

Hi Sorry that I don't quite understand your question. I think the above functions will only be deserialized when the job is submitted, do you want to test the impact of this on submission throughput? Best, Shammon On Fri, Feb 24, 2023 at 3:04 PM Zhongpu Chen wrote: > Hi Gen, > > Thanks for yo

Re: 我使用 start-cluster.sh 运行的flink，我的jar运行一段时间后被关闭了，这个时间是不固定的

Hi According to the exception Caused by: java.util.ConcurrentModificationException at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043) at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMu

Re: 我使用 start-cluster.sh 运行的flink，我的jar运行一段时间后被关闭了，这个时间是不固定的

Hi, The cause of the error is clearly shown in the log. You need to check the logic in 'com.flink.batch.lbs.running.hbase.udf. BatchRunningTimesHBaseUdfRichSinkFunction#flush' Best, Weihua On Fri, Feb 24, 2023 at 4:26 PM simple <1028108...@qq.com> wrote: > > 2023-02-24 16:05:26,104 INFO > org

?????? start-cluster.sh ??????flink??????jar??????????????????????????????????????????

2023-02-24 16:05:26,104 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: Custom Source (1/1) (9644f5bc58ed204c6d31bb133156fa95_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from RUNNING to CANCELING. 2023-02-24 16:05:26,108 INFO  org.apache.flink.runtime.executi

?????? start-cluster.sh ??????flink??????jar??????????????????????????????????????????

2023-02-24 16:05:26,104 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: Custom Source (1/1) (9644f5bc58ed204c6d31bb133156fa95_bc764cd8ddf7a0cff126f51c16239658_0_0) switched from RUNNING to CANCELING. 2023-02-24 16:05:26,108 INFO  org.apache.flink.runtime.executi