I tried using the 'org.apache.hudi.utilities.deltastreamer.NoNewDataTerminationStrategy' to stop the task, but it didn't seem to meet my expectations. I think that after it stops ExecutorService, the subsequent SparkContext will also stop, but now SparkContext will always be started and no subsequent logs will be visible.
| | 李杰 | | leedd1...@163.com | ---- Replied Message ---- | From | Sivabalan<n.siv...@gmail.com> | | Date | 4/1/2023 01:07 | | To | <dev@hudi.apache.org> | | Subject | Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data | We do have Graceful termination possibility w/ deltastreamer continuous mode. Please check here <https://hudi.apache.org/releases/release-0.12.2#deltastreamer-termination-strategy> for post write termination strategy. You can implement your own termination strategy. Hope that helps. On Thu, 30 Mar 2023 at 20:16, Vinoth Chandar <vin...@apache.org> wrote: I believe there is no control today. You could hack a precommit validator and call System.exit if you want ;) (ugly, I know) But maybe we could introduce some abstraction to do a check between loops? or allow users to plugin some logic to decide whether to continue or exit? Love to understand the use-case more here. On Wed, Mar 29, 2023 at 7:32 AM lee <leedd1...@163.com> wrote: When I use the HoodieDeltaStreamer, the "-- continuous" parameter: "Delta Streamer runs in continuous mode running source match ->Transform ->Hudi Write in loop". So I would like to ask if there are any corresponding parameters that can control the number of cycles, such as stopping accessing data when I cycle 5 times. 李杰 leedd1...@163.com < https://dashi.163.com/projects/signature-manager/detail/index.html?ftlId=1&name=%E6%9D%8E%E6%9D%B0&uid=leedd1912%40163.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsmc4215b668fdb6b5ca355a1c3319c4a0e.jpg&items=%5B%22leedd1912%40163.com%22%5D -- Regards, -Sivabalan
