Re: libstdc++: Fix deadlock in debug iterator increment [PR108288]
Committed with the idiomatic approach.
I'll work on this additional check later.
On 12/01/23 22:35, Jonathan Wakely wrote:
On Thu, 12 Jan 2023 at 18:25, François Dumont wrote:
On 12/01/23 13:00, Jonathan Wakely wrote:
On Thu, 12 Jan 2023 at 05:52, François Dumont wrote:
Small update for an obvious compilation issue and to review new test
case that could have lead to an infinite loop if the increment issue was
not detected.
I also forgot to ask if there is more chance for the instantiation to be
elided when it is implemented like in the _Safe_local_iterator:
return { __cur, this->_M_sequence };
No, that doesn't make any difference.
than in the _Safe_iterator:
return _Safe_iterator(__cur, this->_M_sequence);
In the case where the user code do not use it ?
Fully tested now, ok to commit ?
François
On 11/01/23 07:03, François Dumont wrote:
Thanks for fixing this.
Here is the extension of the fix to all post-increment/decrement
operators we have on _GLIBCXX_DEBUG iterator.
Thanks, I completely forgot we have other partial specializations, I
just fixed the one that showed a deadlock in the user's example!
I prefer to restore somehow previous implementation to continue to
have _GLIBCXX_DEBUG post operators implemented in terms of normal post
operators.
Why?
Implementing post-increment as:
auto tmp = *this;
++*this;
return tmp;
is the idiomatic way to write it, and it works fine in this case. I
don't think it performs any more work than your version, does it?
Why not use the idiomatic form?
Is it just so that post-inc of a debug iterator uses post-inc of the
underlying iterator? Why does that matter?
A little yes, but that's a minor reason that is just making me happy.
Main reason is that this form could produce a __msg_init_copy_singular
before the __msg_bad_inc.
Ah yes, I see. That's a shame. I find the idiomatic form much simpler
to read, and it will generate better code (because it just reuses
existing functions, instead of adding new ones).
We could do this though, right?
_GLIBCXX_DEBUG_VERIFY(this->_M_incrementable(),
_M_message(__msg_bad_inc)
._M_iterator(*this, "this"));
_Safe_iterator __tmp = *this;
++*this;
return __tmp;
That does the VERIFY check twice though.
And moreover I plan to propose a patch later to skip any check in the
call to _Safe_iterator(__cur, _M_sequence) as we already know that __cur
is ok here like anywhere else in the lib.
There will still be one in the constructor normally elided unless
--no-elide-constructors but there is not much I can do about it.
Don't worry about it. Nobody should ever use -fno-elide-constructors
in any real cases (except maybe debugging some very strange corner
cases, and in that case the extra safe iterator checks are not going
to be their biggest problem).
The patch is OK for trunk then.
Re: libstdc++: Fix deadlock in debug iterator increment [PR108288]
On Thu, 12 Jan 2023 at 18:25, François Dumont wrote:
>
> On 12/01/23 13:00, Jonathan Wakely wrote:
> > On Thu, 12 Jan 2023 at 05:52, François Dumont wrote:
> >> Small update for an obvious compilation issue and to review new test
> >> case that could have lead to an infinite loop if the increment issue was
> >> not detected.
> >>
> >> I also forgot to ask if there is more chance for the instantiation to be
> >> elided when it is implemented like in the _Safe_local_iterator:
> >> return { __cur, this->_M_sequence };
> > No, that doesn't make any difference.
> >
> >> than in the _Safe_iterator:
> >> return _Safe_iterator(__cur, this->_M_sequence);
> >>
> >> In the case where the user code do not use it ?
> >>
> >> Fully tested now, ok to commit ?
> >>
> >> François
> >>
> >> On 11/01/23 07:03, François Dumont wrote:
> >>> Thanks for fixing this.
> >>>
> >>> Here is the extension of the fix to all post-increment/decrement
> >>> operators we have on _GLIBCXX_DEBUG iterator.
> > Thanks, I completely forgot we have other partial specializations, I
> > just fixed the one that showed a deadlock in the user's example!
> >
> >>> I prefer to restore somehow previous implementation to continue to
> >>> have _GLIBCXX_DEBUG post operators implemented in terms of normal post
> >>> operators.
> > Why?
> >
> > Implementing post-increment as:
> >
> > auto tmp = *this;
> > ++*this;
> > return tmp;
> >
> > is the idiomatic way to write it, and it works fine in this case. I
> > don't think it performs any more work than your version, does it?
> > Why not use the idiomatic form?
> >
> > Is it just so that post-inc of a debug iterator uses post-inc of the
> > underlying iterator? Why does that matter?
> >
> A little yes, but that's a minor reason that is just making me happy.
>
> Main reason is that this form could produce a __msg_init_copy_singular
> before the __msg_bad_inc.
Ah yes, I see. That's a shame. I find the idiomatic form much simpler
to read, and it will generate better code (because it just reuses
existing functions, instead of adding new ones).
We could do this though, right?
_GLIBCXX_DEBUG_VERIFY(this->_M_incrementable(),
_M_message(__msg_bad_inc)
._M_iterator(*this, "this"));
_Safe_iterator __tmp = *this;
++*this;
return __tmp;
That does the VERIFY check twice though.
> And moreover I plan to propose a patch later to skip any check in the
> call to _Safe_iterator(__cur, _M_sequence) as we already know that __cur
> is ok here like anywhere else in the lib.
>
> There will still be one in the constructor normally elided unless
> --no-elide-constructors but there is not much I can do about it.
Don't worry about it. Nobody should ever use -fno-elide-constructors
in any real cases (except maybe debugging some very strange corner
cases, and in that case the extra safe iterator checks are not going
to be their biggest problem).
The patch is OK for trunk then.
Re: libstdc++: Fix deadlock in debug iterator increment [PR108288]
On 12/01/23 13:00, Jonathan Wakely wrote:
On Thu, 12 Jan 2023 at 05:52, François Dumont wrote:
Small update for an obvious compilation issue and to review new test
case that could have lead to an infinite loop if the increment issue was
not detected.
I also forgot to ask if there is more chance for the instantiation to be
elided when it is implemented like in the _Safe_local_iterator:
return { __cur, this->_M_sequence };
No, that doesn't make any difference.
than in the _Safe_iterator:
return _Safe_iterator(__cur, this->_M_sequence);
In the case where the user code do not use it ?
Fully tested now, ok to commit ?
François
On 11/01/23 07:03, François Dumont wrote:
Thanks for fixing this.
Here is the extension of the fix to all post-increment/decrement
operators we have on _GLIBCXX_DEBUG iterator.
Thanks, I completely forgot we have other partial specializations, I
just fixed the one that showed a deadlock in the user's example!
I prefer to restore somehow previous implementation to continue to
have _GLIBCXX_DEBUG post operators implemented in terms of normal post
operators.
Why?
Implementing post-increment as:
auto tmp = *this;
++*this;
return tmp;
is the idiomatic way to write it, and it works fine in this case. I
don't think it performs any more work than your version, does it?
Why not use the idiomatic form?
Is it just so that post-inc of a debug iterator uses post-inc of the
underlying iterator? Why does that matter?
A little yes, but that's a minor reason that is just making me happy.
Main reason is that this form could produce a __msg_init_copy_singular
before the __msg_bad_inc.
And moreover I plan to propose a patch later to skip any check in the
call to _Safe_iterator(__cur, _M_sequence) as we already know that __cur
is ok here like anywhere else in the lib.
There will still be one in the constructor normally elided unless
--no-elide-constructors but there is not much I can do about it.
Re: libstdc++: Fix deadlock in debug iterator increment [PR108288]
On Thu, 12 Jan 2023 at 05:52, François Dumont wrote:
>
> Small update for an obvious compilation issue and to review new test
> case that could have lead to an infinite loop if the increment issue was
> not detected.
>
> I also forgot to ask if there is more chance for the instantiation to be
> elided when it is implemented like in the _Safe_local_iterator:
> return { __cur, this->_M_sequence };
No, that doesn't make any difference.
>
> than in the _Safe_iterator:
> return _Safe_iterator(__cur, this->_M_sequence);
>
> In the case where the user code do not use it ?
>
> Fully tested now, ok to commit ?
>
> François
>
> On 11/01/23 07:03, François Dumont wrote:
> > Thanks for fixing this.
> >
> > Here is the extension of the fix to all post-increment/decrement
> > operators we have on _GLIBCXX_DEBUG iterator.
Thanks, I completely forgot we have other partial specializations, I
just fixed the one that showed a deadlock in the user's example!
> > I prefer to restore somehow previous implementation to continue to
> > have _GLIBCXX_DEBUG post operators implemented in terms of normal post
> > operators.
Why?
Implementing post-increment as:
auto tmp = *this;
++*this;
return tmp;
is the idiomatic way to write it, and it works fine in this case. I
don't think it performs any more work than your version, does it?
Why not use the idiomatic form?
Is it just so that post-inc of a debug iterator uses post-inc of the
underlying iterator? Why does that matter?
Re: libstdc++: Fix deadlock in debug iterator increment [PR108288]
Small update for an obvious compilation issue and to review new test
case that could have lead to an infinite loop if the increment issue was
not detected.
I also forgot to ask if there is more chance for the instantiation to be
elided when it is implemented like in the _Safe_local_iterator:
return { __cur, this->_M_sequence };
than in the _Safe_iterator:
return _Safe_iterator(__cur, this->_M_sequence);
In the case where the user code do not use it ?
Fully tested now, ok to commit ?
François
On 11/01/23 07:03, François Dumont wrote:
Thanks for fixing this.
Here is the extension of the fix to all post-increment/decrement
operators we have on _GLIBCXX_DEBUG iterator.
I prefer to restore somehow previous implementation to continue to
have _GLIBCXX_DEBUG post operators implemented in terms of normal post
operators.
I also plan to remove the debug check in the _Safe_iterator
constructor from base iterator to avoid the redundant check we have
now. But I need to make sure first that we are never calling it with
an unchecked base iterator. And it might not be the right moment to do
such a change.
libstdc++: Fix deadlock in debug local_iterator increment [PR108288]
Complete fix on all _Safe_iterator post-increment and
post-decrement implementations
and on _Safe_local_iterator.
libstdc++-v3/ChangeLog:
* include/debug/safe_iterator.h
(_Safe_iterator<>::operator++(int)): Extend deadlock fix to
other iterator category.
(_Safe_iterator<>::operator--(int)): Likewise.
* include/debug/safe_local_iterator.h
(_Safe_local_iterator<>::operator++(int)): Fix deadlock.
* testsuite/util/debug/unordered_checks.h
(invalid_local_iterator_pre_increment): New.
(invalid_local_iterator_post_increment): New.
*
testsuite/23_containers/unordered_map/debug/invalid_local_iterator_post_increment_neg.cc:
New test.
*
testsuite/23_containers/unordered_map/debug/invalid_local_iterator_pre_increment_neg.cc:
New test.
Tested under Linux x86_64.
Ok to commit ?
François
On 06/01/23 12:54, Jonathan Wakely via Libstdc++ wrote:
Tested x86_64-linux. Pushed to trunk.
I think we should backport this too, after some soak time on trunk.
-- >8 --
With -fno-elide-constructors the debug iterator post-increment and
post-decrement operators are susceptible to deadlock. They take a mutex
lock and then return a temporary, which also attempts to take a lock to
attach itself to the sequence. If the return value and *this happen to
Note that the chosen mutex depends on the sequence so there is no need
for conditional sentense here, it will necessarily be the same mutex.
collide and use the same mutex from the pool, then you get a deadlock
trying to lock a mutex that is already held by the current thread.
diff --git a/libstdc++-v3/include/debug/safe_iterator.h b/libstdc++-v3/include/debug/safe_iterator.h
index f9068eaf8d6..f8b46826b7c 100644
--- a/libstdc++-v3/include/debug/safe_iterator.h
+++ b/libstdc++-v3/include/debug/safe_iterator.h
@@ -129,14 +129,6 @@ namespace __gnu_debug
typename _Sequence::_Base::iterator,
typename _Sequence::_Base::const_iterator>::__type _OtherIterator;
- struct _Attach_single
- { };
-
- _Safe_iterator(_Iterator __i, _Safe_sequence_base* __seq, _Attach_single)
- _GLIBCXX_NOEXCEPT
- : _Iter_base(__i)
- { _M_attach_single(__seq); }
-
public:
typedef _Iterator iterator_type;
typedef typename _Traits::iterator_category iterator_category;
@@ -347,8 +339,13 @@ namespace __gnu_debug
_GLIBCXX_DEBUG_VERIFY(this->_M_incrementable(),
_M_message(__msg_bad_inc)
._M_iterator(*this, "this"));
- __gnu_cxx::__scoped_lock __l(this->_M_get_mutex());
- return _Safe_iterator(base()++, this->_M_sequence, _Attach_single());
+ _Iter_base __cur;
+ {
+ __gnu_cxx::__scoped_lock __l(this->_M_get_mutex());
+ __cur = base()++;
+ }
+
+ return _Safe_iterator(__cur, this->_M_sequence);
}
// -- Utilities --
@@ -520,12 +517,6 @@ namespace __gnu_debug
protected:
typedef typename _Safe_base::_OtherIterator _OtherIterator;
- typedef typename _Safe_base::_Attach_single _Attach_single;
-
- _Safe_iterator(_Iterator __i, _Safe_sequence_base* __seq, _Attach_single)
- _GLIBCXX_NOEXCEPT
- : _Safe_base(__i, __seq, _Attach_single())
- { }
public:
/// @post the iterator is singular and unattached
@@ -609,9 +600,13 @@ namespace __gnu_debug
_GLIBCXX_DEBUG_VERIFY(this->_M_incrementable(),
_M_message(__msg_bad_inc)
._M_iterator(*this, "this"));
- __gnu_cxx::__scoped_lock __l(this->_M_get_mutex());
- return _Safe_iterator(this->base()++, this->_M_sequence,
- _Attach_single());
+ _Iterator __cur;
+ {
+ __gnu_cxx::__scoped_lock __l(this->_M_get_mutex());
+ __cur = this->base()++;
+ }
+
+ return _Safe_iterator(__cur,
