[389-devel] Re: Should stopping the server interrupt an import job?
> On 16 Oct 2019, at 00:27, Mark Reynolds wrote: > > > On 10/14/19 11:13 PM, William Brown wrote: >> >>> On 15 Oct 2019, at 15:51, Mark Reynolds wrote: >>> >>> >>> On 10/14/19 6:35 PM, William Brown wrote: > On 15 Oct 2019, at 06:58, Mark Reynolds wrote: > > So we are finding these race conditions (leading to heap-use-after-free) > when you stop the server while an import task is running. The current > code aborts the task which leaves the database unusable until it is fully > reinitialized at a later time. Unfortunately the code that handles this > is complex, and very error prone. > > I'd like to allow the import task to complete before fully shutting the > server down. Code fix is trivial, but do we want the import to finish, > or should the import be aborted (and database left broken)? Thoughts? > Opinions? The question is "what does the admin expect"? I could envisage if you start an import and then cancel the task, you expect: * The task to be immediately stopped * The db content rolled back. Shouldn't we be in a betxn or similar during an import so we can revert? Failing this, I'd assume the user would expect a ctrl-c to immediately cancel the task. What kind of use-after-frees are we seeing? >>> See >>> >>> https://pagure.io/389-ds-base/issue/50646 >>> >>> Pretty sure the first thing the import does is delete the db directory, but >>> I have not found that in the code yet, but there are definitely no >>> transactions used during an "import". It's a very different process. Now >>> rolling back the database would be nice, but I can imagine very large >>> databases(100+ million entries) where disk space could be an issue if you >>> have to keep the old database around until the new one is imported. >>> >>> As for aborting, currently there is no abort mechanism except for stopping >>> the server. So a ctrl-C is not really an option at this time. Keep in >>> mind I can still easily keep the current abort behavior during a shutdown, >>> but in the current design if you abort an import the database is hosed. >> What about at least hosing it but nicely somehow? So we rm db, then we >> start the import into some temp location, so on ctrl-c even if we crash, db >> is an empty dir and still mildly hosed? >> >> I can see your point though about the db size stuff. So I guess my thinking >> then is: >> >> * Could we just fix the use after free *and* make the db hosing nicer >> somehow? > I can fix the use-after-free, that's easy, but fixing the "hosing" is not > trivial, and it is out of the scope of what I am trying to do in 1.3.x. fair >> * What happens if we ignore the ctrl-c and just block on import? I think >> someone would reach for ctrl+\ pretty fast ... > The import code launches a thread to the do the work. So hitting control-C > on a CLI tool will not stop the import, and there is currently no abort > process for Slapi Tasks that I am aware of besides creating a new "Abort > Import" task. This is something we could do in 1.4.3, or 1.5.x along with > the new backend work. Yeah, that seems like the possible solution to have each thread check periodically for a cancel signal in it's work loop? >> >> >> > Thanks, > > Mark > > -- > > 389 Directory Server Development Team > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org >>> -- >>> >>> 389 Directory Server Development Team >> — >> Sincerely, >> >> William Brown >> >> Senior Software Engineer, 389 Directory Server >> SUSE Labs >> > -- > > 389 Directory Server Development Team — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
[389-devel] Re: Should stopping the server interrupt an import job?
On 10/14/19 11:13 PM, William Brown wrote: On 15 Oct 2019, at 15:51, Mark Reynolds wrote: On 10/14/19 6:35 PM, William Brown wrote: On 15 Oct 2019, at 06:58, Mark Reynolds wrote: So we are finding these race conditions (leading to heap-use-after-free) when you stop the server while an import task is running. The current code aborts the task which leaves the database unusable until it is fully reinitialized at a later time. Unfortunately the code that handles this is complex, and very error prone. I'd like to allow the import task to complete before fully shutting the server down. Code fix is trivial, but do we want the import to finish, or should the import be aborted (and database left broken)? Thoughts? Opinions? The question is "what does the admin expect"? I could envisage if you start an import and then cancel the task, you expect: * The task to be immediately stopped * The db content rolled back. Shouldn't we be in a betxn or similar during an import so we can revert? Failing this, I'd assume the user would expect a ctrl-c to immediately cancel the task. What kind of use-after-frees are we seeing? See https://pagure.io/389-ds-base/issue/50646 Pretty sure the first thing the import does is delete the db directory, but I have not found that in the code yet, but there are definitely no transactions used during an "import". It's a very different process. Now rolling back the database would be nice, but I can imagine very large databases(100+ million entries) where disk space could be an issue if you have to keep the old database around until the new one is imported. As for aborting, currently there is no abort mechanism except for stopping the server. So a ctrl-C is not really an option at this time. Keep in mind I can still easily keep the current abort behavior during a shutdown, but in the current design if you abort an import the database is hosed. What about at least hosing it but nicely somehow? So we rm db, then we start the import into some temp location, so on ctrl-c even if we crash, db is an empty dir and still mildly hosed? I can see your point though about the db size stuff. So I guess my thinking then is: * Could we just fix the use after free *and* make the db hosing nicer somehow? I can fix the use-after-free, that's easy, but fixing the "hosing" is not trivial, and it is out of the scope of what I am trying to do in 1.3.x. * What happens if we ignore the ctrl-c and just block on import? I think someone would reach for ctrl+\ pretty fast ... The import code launches a thread to the do the work. So hitting control-C on a CLI tool will not stop the import, and there is currently no abort process for Slapi Tasks that I am aware of besides creating a new "Abort Import" task. This is something we could do in 1.4.3, or 1.5.x along with the new backend work. Thanks, Mark -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org -- 389 Directory Server Development Team — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Should stopping the server interrupt an import job?
> On 15 Oct 2019, at 15:51, Mark Reynolds wrote: > > > On 10/14/19 6:35 PM, William Brown wrote: >> >>> On 15 Oct 2019, at 06:58, Mark Reynolds wrote: >>> >>> So we are finding these race conditions (leading to heap-use-after-free) >>> when you stop the server while an import task is running. The current code >>> aborts the task which leaves the database unusable until it is fully >>> reinitialized at a later time. Unfortunately the code that handles this is >>> complex, and very error prone. >>> >>> I'd like to allow the import task to complete before fully shutting the >>> server down. Code fix is trivial, but do we want the import to finish, or >>> should the import be aborted (and database left broken)? Thoughts? >>> Opinions? >> The question is "what does the admin expect"? I could envisage if you start >> an import and then cancel the task, you expect: >> >> * The task to be immediately stopped >> * The db content rolled back. >> >> Shouldn't we be in a betxn or similar during an import so we can revert? >> >> Failing this, I'd assume the user would expect a ctrl-c to immediately >> cancel the task. >> >> What kind of use-after-frees are we seeing? > See > > https://pagure.io/389-ds-base/issue/50646 > > Pretty sure the first thing the import does is delete the db directory, but I > have not found that in the code yet, but there are definitely no transactions > used during an "import". It's a very different process. Now rolling back > the database would be nice, but I can imagine very large databases(100+ > million entries) where disk space could be an issue if you have to keep the > old database around until the new one is imported. > > As for aborting, currently there is no abort mechanism except for stopping > the server. So a ctrl-C is not really an option at this time. Keep in mind > I can still easily keep the current abort behavior during a shutdown, but in > the current design if you abort an import the database is hosed. What about at least hosing it but nicely somehow? So we rm db, then we start the import into some temp location, so on ctrl-c even if we crash, db is an empty dir and still mildly hosed? I can see your point though about the db size stuff. So I guess my thinking then is: * Could we just fix the use after free *and* make the db hosing nicer somehow? * What happens if we ignore the ctrl-c and just block on import? I think someone would reach for ctrl+\ pretty fast ... > >>> Thanks, >>> >>> Mark >>> >>> -- >>> >>> 389 Directory Server Development Team >>> ___ >>> 389-devel mailing list -- 389-devel@lists.fedoraproject.org >>> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >>> Fedora Code of Conduct: >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org >> — >> Sincerely, >> >> William Brown >> >> Senior Software Engineer, 389 Directory Server >> SUSE Labs >> ___ >> 389-devel mailing list -- 389-devel@lists.fedoraproject.org >> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > > -- > > 389 Directory Server Development Team — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Should stopping the server interrupt an import job?
On 10/14/19 6:35 PM, William Brown wrote: On 15 Oct 2019, at 06:58, Mark Reynolds wrote: So we are finding these race conditions (leading to heap-use-after-free) when you stop the server while an import task is running. The current code aborts the task which leaves the database unusable until it is fully reinitialized at a later time. Unfortunately the code that handles this is complex, and very error prone. I'd like to allow the import task to complete before fully shutting the server down. Code fix is trivial, but do we want the import to finish, or should the import be aborted (and database left broken)? Thoughts? Opinions? The question is "what does the admin expect"? I could envisage if you start an import and then cancel the task, you expect: * The task to be immediately stopped * The db content rolled back. Shouldn't we be in a betxn or similar during an import so we can revert? Failing this, I'd assume the user would expect a ctrl-c to immediately cancel the task. What kind of use-after-frees are we seeing? See https://pagure.io/389-ds-base/issue/50646 Pretty sure the first thing the import does is delete the db directory, but I have not found that in the code yet, but there are definitely no transactions used during an "import". It's a very different process. Now rolling back the database would be nice, but I can imagine very large databases(100+ million entries) where disk space could be an issue if you have to keep the old database around until the new one is imported. As for aborting, currently there is no abort mechanism except for stopping the server. So a ctrl-C is not really an option at this time. Keep in mind I can still easily keep the current abort behavior during a shutdown, but in the current design if you abort an import the database is hosed. Thanks, Mark -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org -- 389 Directory Server Development Team ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Should stopping the server interrupt an import job?
> On 15 Oct 2019, at 06:58, Mark Reynolds wrote: > > So we are finding these race conditions (leading to heap-use-after-free) when > you stop the server while an import task is running. The current code aborts > the task which leaves the database unusable until it is fully reinitialized > at a later time. Unfortunately the code that handles this is complex, and > very error prone. > > I'd like to allow the import task to complete before fully shutting the > server down. Code fix is trivial, but do we want the import to finish, or > should the import be aborted (and database left broken)? Thoughts? Opinions? The question is "what does the admin expect"? I could envisage if you start an import and then cancel the task, you expect: * The task to be immediately stopped * The db content rolled back. Shouldn't we be in a betxn or similar during an import so we can revert? Failing this, I'd assume the user would expect a ctrl-c to immediately cancel the task. What kind of use-after-frees are we seeing? > > Thanks, > > Mark > > -- > > 389 Directory Server Development Team > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
