ninsmiracle commented on PR #1961: URL: https://github.com/apache/incubator-pegasus/pull/1961#issuecomment-2061094185
> > > > > It's OK to me to remove the periodic manual compaction function, it can be replaced by thirdparty tools, feel free to remove it. > > > > > > > > > > > > I think it's reasonable to keep this periodic manual compaction function. Cause many user using spark to set manual compaction time now. If we remove this function in new version, we have to change the change user habits. Actully, I think we lack a unified management platform to control every things. > > > > > > > > > What is "using spark to set manual compaction time" ? Is it a Pegasus binding method? I don't think it's a block to remove the periodic manual compaction. This is not the core tasks of Pegasus, it make the code a bit of smelly. > > > At least, you need to make this patch to resolve these issues mentioned above. > > > > > > In high version of [PEGASUS-SPARK](https://github.com/pegasus-kv/pegasus-spark) , a compaction request will be send to cluster after user call `startBulkLoad` . And I will update PEGASUS-SPARK this week. I do not think we should remove perodic manual compaction code now . Cause I think all its bugs about manual compaction have been fixed, and they are all minor issues. > > Thanks to update the pegasus-spark project! It's not well maintenance for about 4 years. > > Do you think it's confusing that the manual compact status depends on `last_manual_compact_used_time`? Is it possible `manual_compact_last_time_used_ms` == 0 but `manual_compact_last_finish_time_ms` > 0 ? > > If the paramaters are set in a previous round, and havn't complete compaction and set the vaiables in current round before restarting the server, will it be considered as finished? > > Could you add some tests to verify the bug has been fixed? Firstly , I update the newest version of [PEGASUS-SPARK](https://github.com/pegasus-kv/pegasus-spark) and commit a merge request. And I think `manual_compact_last_time_used_ms == 0 but manual_compact_last_finish_time_ms > 0` will happened before this pr . However , now , in `pegasus_server_impl::start` replica will reload the parameter from disk . If `last_finish_time_ms` be set , `last_time_used_ms` will be set also.The results of the staging environment test verified this point. In this week we will try to put this manual compaction fix version to online environment , and we will verify the effectiveness of bug fixes in large data environments . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pegasus.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pegasus.apache.org For additional commands, e-mail: dev-h...@pegasus.apache.org
