are stored at
google cloud storage. Does anyone encounter the same problem? Any comment
is appreciated.
Caused by: org.apache.hadoop.fs.FileAlreadyExistsException:
'gs://bucket/flink-app//chk-5812/_metadata'
already exists
Best wishes,
Chen-Che Huang
of iterationstream. Without iteration, the
savepoints/checkpoints can be created successfully. Does anyone encounter
similar issues or know how to fix this? Any comment is appreciated.
Best wishes,
Chen-Che Huang
{
"threadName": "OutputFlusher for xxx-operator",
&q
he/flink/streaming/examples/iteration/IterateExample.java
>
> Best,
> Guowei
>
>
> On Mon, Mar 21, 2022 at 2:27 PM Chen-Che Huang wrote:
>
>> Hi all,
>>
>> We have an application where the operations on some keys depend on the
>> results of related keys.
n't be
sent to the input stream. We wonder whether it's doable for our scenario
with Flink? If so, how should we modify our code to
achieve the goal? Many thanks for any comments.
Best regards,
Chen-Che Huang
Hi all,
To minimize the recovery time from failure, we employ incremental, retained
checkpoint with `state.checkpoints.num-retained
as 10` in our Flink apps. With this setting, Flink automatically creates
new checkpoints regularly and keeps only the latest 10
checkpoints. Besides, for app
job from Savepoint
Best wishes,
Chen-Che Huang
ternative and has been incorrectly written to the release
> note. You can refer to the [1] instead.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#state-backends-latency-tracking-options
>
> Best,
> Yangze Guo
>
> On F
for metrics with a marginal impact. However, I couldn't see. Thanks.
Best wishes,
Chen-Che Huang
putFormat.
>
>
> On Fri, Apr 16, 2021 at 5:37 AM Chen-Che Huang wrote:
>
> > Hi Robert,
> >
> > Thanks for your code. It's really helpful!
> >
> > However, with the readKeyedState api of state processor, we get dataset
> > for our data instead of da
Hi,
I have a job that uses the state processor to load data from checkpoints on
google cloud storage to do some processing and then write the result to google
cloud storage. The total data size is about 30-50 GB and the job may take more
than 2 hours to finish. From the flame graph generated
Hi Robert,
Due to some concerns, we planned to use state processor to achieve our goal.
Now we will consider to reevaluate using datastream to do the job while
exploring the possibility of implementing a custom FileOutputFormat. Thanks for
your comments!
Best wishes,
Chen-Che Huang
On 2021
the dataset to a datastream. I'm not sure it's doable based on
https://www.alibabacloud.com/blog/deep-insights-into-flink-sql-flink-advanced-tutorials_596628.
If it's doable, then I'll be able to solve our problem with applying
streamfilesink to the transformed dataset.
Best wishes,
Chen-Che Huang
solution is usually more
efficient and reliable than doing it by myself. Many thanks for any comment.
key1.txt
key1 value11
key1 value21
key1 value31
key2.txt
key2 value21
key2 value22
key2 value23
Best wishes,
Chen-Che Huang
ocs-master/docs/deployment/resource-providers/native_kubernetes/#pod-template
>
> Best,
> Yang
>
> Chen-Che Huang 于2021年3月16日周二 下午1:26写道:
>
> > Hi,
> >
> > We use the per-job deployment mode to deploy our Flink services on
> > Kubernetes. We're considering
create an issue for this feature. Thanks.
Best regards,
Chen-Che Huang
15 matches
Mail list logo