Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data
Should we stop SparkContext? | | 李杰 | | leedd1...@163.com | Replied Message | From | lee | | Date | 4/3/2023 11:09 | | To | Sivabalan | | Cc | dev@hudi.apache.org | | Subject | Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data | I tried using the 'org.apache.hudi.utilities.deltastreamer.NoNewDataTerminationStrategy' to stop the task, but it didn't seem to meet my expectations. I think that after it stops ExecutorService, the subsequent SparkContext will also stop, but now SparkContext will always be started and no subsequent logs will be visible. | | 李杰 | | leedd1...@163.com | Replied Message | From | Sivabalan | | Date | 4/1/2023 01:07 | | To | | | Subject | Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data | We do have Graceful termination possibility w/ deltastreamer continuous mode. Please check here <https://hudi.apache.org/releases/release-0.12.2#deltastreamer-termination-strategy> for post write termination strategy. You can implement your own termination strategy. Hope that helps. On Thu, 30 Mar 2023 at 20:16, Vinoth Chandar wrote: I believe there is no control today. You could hack a precommit validator and call System.exit if you want ;) (ugly, I know) But maybe we could introduce some abstraction to do a check between loops? or allow users to plugin some logic to decide whether to continue or exit? Love to understand the use-case more here. On Wed, Mar 29, 2023 at 7:32 AM lee wrote: When I use the HoodieDeltaStreamer, the "-- continuous" parameter: "Delta Streamer runs in continuous mode running source match ->Transform ->Hudi Write in loop". So I would like to ask if there are any corresponding parameters that can control the number of cycles, such as stopping accessing data when I cycle 5 times. 李杰 leedd1...@163.com < https://dashi.163.com/projects/signature-manager/detail/index.html?ftlId=1&name=%E6%9D%8E%E6%9D%B0&uid=leedd1912%40163.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsmc4215b668fdb6b5ca355a1c3319c4a0e.jpg&items=%5B%22leedd1912%40163.com%22%5D -- Regards, -Sivabalan
Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data
I tried using the 'org.apache.hudi.utilities.deltastreamer.NoNewDataTerminationStrategy' to stop the task, but it didn't seem to meet my expectations. I think that after it stops ExecutorService, the subsequent SparkContext will also stop, but now SparkContext will always be started and no subsequent logs will be visible. | | 李杰 | | leedd1...@163.com | Replied Message | From | Sivabalan | | Date | 4/1/2023 01:07 | | To | | | Subject | Re: When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data | We do have Graceful termination possibility w/ deltastreamer continuous mode. Please check here <https://hudi.apache.org/releases/release-0.12.2#deltastreamer-termination-strategy> for post write termination strategy. You can implement your own termination strategy. Hope that helps. On Thu, 30 Mar 2023 at 20:16, Vinoth Chandar wrote: I believe there is no control today. You could hack a precommit validator and call System.exit if you want ;) (ugly, I know) But maybe we could introduce some abstraction to do a check between loops? or allow users to plugin some logic to decide whether to continue or exit? Love to understand the use-case more here. On Wed, Mar 29, 2023 at 7:32 AM lee wrote: When I use the HoodieDeltaStreamer, the "-- continuous" parameter: "Delta Streamer runs in continuous mode running source match ->Transform ->Hudi Write in loop". So I would like to ask if there are any corresponding parameters that can control the number of cycles, such as stopping accessing data when I cycle 5 times. 李杰 leedd1...@163.com < https://dashi.163.com/projects/signature-manager/detail/index.html?ftlId=1&name=%E6%9D%8E%E6%9D%B0&uid=leedd1912%40163.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fsmc4215b668fdb6b5ca355a1c3319c4a0e.jpg&items=%5B%22leedd1912%40163.com%22%5D -- Regards, -Sivabalan
When using the HoodieDeltaStreamer, is there a corresponding parameter that can control the number of cycles? For example, if I cycle 5 times, I stop accessing data
When I use the HoodieDeltaStreamer, the "-- continuous" parameter: "Delta Streamer runs in continuous mode running source match ->Transform ->Hudi Write in loop". So I would like to ask if there are any corresponding parameters that can control the number of cycles, such as stopping accessing data when I cycle 5 times. | | 李杰 | | leedd1...@163.com |
Want to be added as a contributor to the project
Hi, I want to be added as a contributor to the project. My username is noninertialframe. Thanks so much :)
Re: contributor permission
sorry, my mistake. My apache id is CodeCooker17. Please add permission for me, thank you. | | ChuangLee | | codecooker_h...@163.com | 签名由网易邮箱大师定制 On 04/18/2022 10:18，Sivabalan wrote： Is HUDI-3898 is your apache id? we might need your apache id to add you as a contributor. On Sun, 17 Apr 2022 at 05:16, Chuang Lee wrote: Hi, I want to contribute to Apache Hudi. Would you please give me the contributor permission? My JIRA ID is HUDI-3898. Thank you. | | Chuang Lee | | codecooker_h...@163.com | 签名由网易邮箱大师定制 -- Regards, -Sivabalan
contributor permission
Hi, I want to contribute to Apache Hudi. Would you please give me the contributor permission? My JIRA ID is HUDI-3898. Thank you. | | Chuang Lee | | codecooker_h...@163.com | 签名由网易邮箱大师定制
