Can you try with latest master. We have fixed <https://github.com/apache/hudi/pull/8173> this recently.
On Mon, 3 Apr 2023 at 19:49, lee <leedd1...@163.com> wrote: > > Should we stop SparkContext? > 李杰 > leedd1...@163.com > > <https://dashi.163.com/projects/signature-manager/detail/index.html?ftlId=1&name=%E6%9D%8E%E6%9D%B0&uid=leedd1912%40163.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsmc4215b668fdb6b5ca355a1c3319c4a0e.jpg&items=%5B%22leedd1912%40163.com%22%5D> > ---- Replied Message ---- > From lee<leedd1...@163.com> <leedd1...@163.com> > Date 4/3/2023 11:09 > To Sivabalan<n.siv...@gmail.com> <n.siv...@gmail.com> > Cc dev@hudi.apache.org<dev@hudi.apache.org> <dev@hudi.apache.org> > Subject Re: When using the HoodieDeltaStreamer, is there a corresponding > parameter that can control the number of cycles? For example, if I cycle 5 > times, I stop accessing data > I tried using the > 'org.apache.hudi.utilities.deltastreamer.NoNewDataTerminationStrategy' to > stop the task, but it didn't seem to meet my expectations. I think that > after it stops ExecutorService, the subsequent SparkContext will also stop, > but now SparkContext will always be started and no subsequent logs will be > visible. > > > > > 李杰 > leedd1...@163.com > > <https://dashi.163.com/projects/signature-manager/detail/index.html?ftlId=1&name=%E6%9D%8E%E6%9D%B0&uid=leedd1912%40163.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsmc4215b668fdb6b5ca355a1c3319c4a0e.jpg&items=%5B%22leedd1912%40163.com%22%5D> > ---- Replied Message ---- > From Sivabalan<n.siv...@gmail.com> <n.siv...@gmail.com> > Date 4/1/2023 01:07 > To <dev@hudi.apache.org> <dev@hudi.apache.org> > Subject Re: When using the HoodieDeltaStreamer, is there a corresponding > parameter that can control the number of cycles? For example, if I cycle 5 > times, I stop accessing data > We do have Graceful termination possibility w/ deltastreamer > continuous mode. Please check here > < > https://hudi.apache.org/releases/release-0.12.2#deltastreamer-termination-strategy > > > for post write termination strategy. You can implement your own termination > strategy. Hope that helps. > > On Thu, 30 Mar 2023 at 20:16, Vinoth Chandar <vin...@apache.org> wrote: > > I believe there is no control today. You could hack a precommit validator > and call System.exit if you want ;) (ugly, I know) > > But maybe we could introduce some abstraction to do a check between loops? > or allow users to plugin some logic to decide whether to continue or exit? > > Love to understand the use-case more here. > > On Wed, Mar 29, 2023 at 7:32 AM lee <leedd1...@163.com> wrote: > > When I use the HoodieDeltaStreamer, the "-- continuous" parameter: "Delta > Streamer runs in continuous mode running source match ->Transform ->Hudi > Write in loop". So I would like to ask if there are any corresponding > parameters that can control the number of cycles, such as stopping > accessing data when I cycle 5 times. > > > > 李杰 > leedd1...@163.com > > < > > > https://dashi.163.com/projects/signature-manager/detail/index.html?ftlId=1&name=%E6%9D%8E%E6%9D%B0&uid=leedd1912%40163.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsmc4215b668fdb6b5ca355a1c3319c4a0e.jpg&items=%5B%22leedd1912%40163.com%22%5D > > > > > > > -- > Regards, > -Sivabalan > -- Regards, -Sivabalan
