Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data

Can you try with latest master. We have fixed this recently. On Mon, 3 Apr 2023 at 19:49, lee wrote: > > Should we stop SparkContext? > 李杰 > leedd1...@163.com > >

Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data

Should we stop SparkContext? | | 李杰 | | leedd1...@163.com | Replied Message | From | lee | | Date | 4/3/2023 11:09 | | To | Sivabalan | | Cc | dev@hudi.apache.org | | Subject | Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of

Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data

I tried using the 'org.apache.hudi.utilities.deltastreamer.NoNewDataTerminationStrategy' to stop the task, but it didn't seem to meet my expectations. I think that after it stops ExecutorService, the subsequent SparkContext will also stop, but now SparkContext will always be started and no

Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data

We do have Graceful termination possibility w/ deltastreamer continuous mode. Please check here for post write termination strategy. You can implement your own termination strategy. Hope that helps. On Thu, 30

Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data

I believe there is no control today. You could hack a precommit validator and call System.exit if you want ;) (ugly, I know) But maybe we could introduce some abstraction to do a check between loops? or allow users to plugin some logic to decide whether to continue or exit? Love to understand

When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data

When I use the HoodieDeltaStreamer, the "-- continuous" parameter: "Delta Streamer runs in continuous mode running source match ->Transform ->Hudi Write in loop". So I would like to ask if there are any corresponding parameters that can control the number of cycles, such as stopping accessing