Re: [Development] Important recent changes in QList/QString/QByteArray

[Andrei Golubev]

The "flag" is basically "a call to reserve was made"? How do I clear the flag?
shrink_to_fit()/squeeze() do this along the way.

--
Best Regards,
Andrei

From: Ville Voutilainen 
Sent: Thursday, September 10, 2020 4:38 PM
To: Andrei Golubev 
Cc: development@qt-project.org 
Subject: Re: [Development] Important recent changes in QList/QString/QByteArray

On Thu, 10 Sep 2020 at 16:04, Andrei Golubev  wrote:
>
> Should I expect to do a reserve call with a *smaller* value than my
> current size is before removing elements from
> a container, in order to set a "target size"? For any standard
> container, such a reserve is a complete noop because
> it can't be responded to by shrinking a capacity lower than the size,
> but with QList it apparently records a preferred size?
>
> No. If the value passed to reserve is smaller than capacity, we do not 
> shrink*.
> For any passed value, we set a specific flag that the capacity is reserved. 
> There is no notion of "target size" and "target capacity" is out of scope of 
> the container (this is something that the user knows about).
> Instead, having the flag in place, we can decide whether allocation should 
> take place or not. When allocation is not obligatory and the capacity 
> reservation flag is set, we do not reallocate but instead do "something 
> else": in case of erase - just remove elements, in case of prepend - move 
> data around (if necessary).
> The flag is a mere part of QCC's state.

The "flag" is basically "a call to reserve was made"? How do I clear the flag?
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Giuseppe D'Angelo via Development]

On 10/09/2020 15:38, Ville Voutilainen wrote:

The "flag" is basically "a call to reserve was made"? How do I clear the flag?


Yes (CapacityReserved, in Qt 5). squeeze() / shrink_to_fit() were 
supposed to clear it.


HTH,
--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts



smime.p7s
Description: S/MIME Cryptographic Signature
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Ville Voutilainen]

On Thu, 10 Sep 2020 at 16:04, Andrei Golubev  wrote:
>
> Should I expect to do a reserve call with a *smaller* value than my
> current size is before removing elements from
> a container, in order to set a "target size"? For any standard
> container, such a reserve is a complete noop because
> it can't be responded to by shrinking a capacity lower than the size,
> but with QList it apparently records a preferred size?
>
> No. If the value passed to reserve is smaller than capacity, we do not 
> shrink*.
> For any passed value, we set a specific flag that the capacity is reserved. 
> There is no notion of "target size" and "target capacity" is out of scope of 
> the container (this is something that the user knows about).
> Instead, having the flag in place, we can decide whether allocation should 
> take place or not. When allocation is not obligatory and the capacity 
> reservation flag is set, we do not reallocate but instead do "something 
> else": in case of erase - just remove elements, in case of prepend - move 
> data around (if necessary).
> The flag is a mere part of QCC's state.

The "flag" is basically "a call to reserve was made"? How do I clear the flag?
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Andrei Golubev]

Should I expect to do a reserve call with a *smaller* value than my
current size is before removing elements from
a container, in order to set a "target size"? For any standard
container, such a reserve is a complete noop because
it can't be responded to by shrinking a capacity lower than the size,
but with QList it apparently records a preferred size?
No. If the value passed to reserve is smaller than capacity, we do not shrink*.
For any passed value, we set a specific flag that the capacity is reserved. 
There is no notion of "target size" and "target capacity" is out of scope of 
the container (this is something that the user knows about).
Instead, having the flag in place, we can decide whether allocation should take 
place or not. When allocation is not obligatory and the capacity reservation 
flag is set, we do not reallocate but instead do "something else": in case of 
erase - just remove elements, in case of prepend - move data around (if 
necessary).
The flag is a mere part of QCC's state.

* let aside the case when QCC is implicitly shared

--
Best Regards,
Andrei

From: Ville Voutilainen 
Sent: Thursday, September 10, 2020 3:43 PM
To: Andrei Golubev 
Cc: development@qt-project.org 
Subject: Re: [Development] Important recent changes in QList/QString/QByteArray

On Thu, 10 Sep 2020 at 11:46, Andrei Golubev  wrote:
>
> That's interesting. So the container remembers what sort of a reserve
> request I made on it, and uses
> that as the preferred size whenever the element count of the container 
> changes?
>
> Yes. Calling reserve typically means that you do not want to have extra 
> reallocations while updating the container's content (instead you allocate 
> sufficient space upfront). In case of QList, we make sure there are no 
> reallocations happening unless absolutely necessary, e.g. appending 1 element 
> when size == capacity would allocate bigger memory chunk - reservation is a 
> hint, but not an obligation. Consequently, QList's shrinking erase also does 
> not shrink when capacity is reserved.

Well, yeah - calling reserve before growing a container avoids extra
allocations. Calling reserve before removing elements
doesn't have such a "typical" effect.

Should I expect to do a reserve call with a *smaller* value than my
current size is before removing elements from
a container, in order to set a "target size"? For any standard
container, such a reserve is a complete noop because
it can't be responded to by shrinking a capacity lower than the size,
but with QList it apparently records a preferred size?
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Ville Voutilainen]

On Thu, 10 Sep 2020 at 11:46, Andrei Golubev  wrote:
>
> That's interesting. So the container remembers what sort of a reserve
> request I made on it, and uses
> that as the preferred size whenever the element count of the container 
> changes?
>
> Yes. Calling reserve typically means that you do not want to have extra 
> reallocations while updating the container's content (instead you allocate 
> sufficient space upfront). In case of QList, we make sure there are no 
> reallocations happening unless absolutely necessary, e.g. appending 1 element 
> when size == capacity would allocate bigger memory chunk - reservation is a 
> hint, but not an obligation. Consequently, QList's shrinking erase also does 
> not shrink when capacity is reserved.

Well, yeah - calling reserve before growing a container avoids extra
allocations. Calling reserve before removing elements
doesn't have such a "typical" effect.

Should I expect to do a reserve call with a *smaller* value than my
current size is before removing elements from
a container, in order to set a "target size"? For any standard
container, such a reserve is a complete noop because
it can't be responded to by shrinking a capacity lower than the size,
but with QList it apparently records a preferred size?
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Andrei Golubev]

But reallocating on erase still falls under the problem that it
invalidates everything, including the part before the erased range.

Indeed. This is why I mentioned it in the first place. This can be "disabled" 
by calling reserve().
Anyhow, I am not against any of the options: have shrinking optimization or not 
in erase(). It would just be good to know why it was needed (or thought of) 
initially.


--
Best Regards,
Andrei

From: Development  on behalf of Giuseppe 
D'Angelo via Development 
Sent: Thursday, September 10, 2020 12:29 PM
To: development@qt-project.org 
Subject: Re: [Development] Important recent changes in QList/QString/QByteArray

Il 10/09/20 08:48, Andrei Golubev ha scritto:
>
> That's the dilemma that Andrei was talking about. Either solution is
> valid and
> both have a way for you to tell QList to do what you want.
>
> Oh, but it's not a question of "if". It is already done in latest dev
> this way, the shrinking erase, I mean (not sure who was an author of the
> original change).

But reallocating on erase still falls under the problem that it
invalidates everything, including the part before the erased range.

--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Giuseppe D'Angelo via Development]

Il 10/09/20 08:48, Andrei Golubev ha scritto:


That's the dilemma that Andrei was talking about. Either solution is
valid and
both have a way for you to tell QList to do what you want.

Oh, but it's not a question of "if". It is already done in latest dev 
this way, the shrinking erase, I mean (not sure who was an author of the 
original change).


But reallocating on erase still falls under the problem that it 
invalidates everything, including the part before the erased range.


--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts



smime.p7s
Description: Firma crittografica S/MIME
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Giuseppe D'Angelo via Development]

Il 10/09/20 04:31, Thiago Macieira ha scritto:

This is another against automatic shrinking; it may invalidate
everything rather than just the erased area. I don't understand the
"detach" part, is it about holding references across a detach, so they
refer to the original container, not the detached one? That's not
different from we always had, isn't it?

It means that if you try to erase from a container that is currently shared,
it will detach. When it does, the detached container's size will be evaluated
in terms of the size of the container, not the capacity the shared container
had.



Why isn't the capacity carried across?


QVector v;
v.reserve(100);
v.push_back(~~~); // fill it up

// someone takes a copy
auto v2 = v;

// erase into the original
v.erase(~~~); 


// now v's capacity is no longer reserved to be >= 100?


My 2 c,

--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts



smime.p7s
Description: Firma crittografica S/MIME
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Andrei Golubev]

That's interesting. So the container remembers what sort of a reserve
request I made on it, and uses
that as the preferred size whenever the element count of the container changes?

Yes. Calling reserve typically means that you do not want to have extra 
reallocations while updating the container's content (instead you allocate 
sufficient space upfront). In case of QList, we make sure there are no 
reallocations happening unless absolutely necessary, e.g. appending 1 element 
when size == capacity would allocate bigger memory chunk - reservation is a 
hint, but not an obligation. Consequently, QList's shrinking erase also does 
not shrink when capacity is reserved.

--
Best Regards,
Andrei


From: Ville Voutilainen 
Sent: Thursday, September 10, 2020 10:39 AM
To: Andrei Golubev 
Cc: development@qt-project.org 
Subject: Re: [Development] Important recent changes in QList/QString/QByteArray

On Wed, 9 Sep 2020 at 16:38, Andrei Golubev  wrote:
>
> I don't understand what this means. Am I supposed to reserve a
> container to its current size before erasing elements
> from it, if I don't want the erase to shrink it?
>
> Yes.

That's interesting. So the container remembers what sort of a reserve
request I made on it, and uses
that as the preferred size whenever the element count of the container changes?
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Ville Voutilainen]

On Wed, 9 Sep 2020 at 16:38, Andrei Golubev  wrote:
>
> I don't understand what this means. Am I supposed to reserve a
> container to its current size before erasing elements
> from it, if I don't want the erase to shrink it?
>
> Yes.

That's interesting. So the container remembers what sort of a reserve
request I made on it, and uses
that as the preferred size whenever the element count of the container changes?
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Andrei Golubev]

If we implement this behaviour.

That's the dilemma that Andrei was talking about. Either solution is valid and
both have a way for you to tell QList to do what you want.
Oh, but it's not a question of "if". It is already done in latest dev this way, 
the shrinking erase, I mean (not sure who was an author of the original change).

--
Best Regards,
Andrei

From: Development  on behalf of Thiago 
Macieira 
Sent: Thursday, September 10, 2020 5:33 AM
To: development@qt-project.org 
Subject: Re: [Development] Important recent changes in QList/QString/QByteArray

On Wednesday, 9 September 2020 06:38:01 PDT Andrei Golubev wrote:
> > I don't understand what this means. Am I supposed to reserve a
> > container to its current size before erasing elements
> > from it, if I don't want the erase to shrink it?
> Yes.

If we implement this behaviour.

That's the dilemma that Andrei was talking about. Either solution is valid and
both have a way for you to tell QList to do what you want.

--
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel DPG Cloud Engineering



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Julien Cugnière]

Le mer. 9 sept. 2020 à 15:38, Andrei Golubev  a écrit :
>
> I don't understand what this means. Am I supposed to reserve a
> container to its current size before erasing elements
> from it, if I don't want the erase to shrink it?
>
> Yes.

For what it's worth, as a user, I think this would violate the
principle of least surprise. I would expect QCC in Qt6 to behave like
std::vector, and like QCC in Qt5 : an erase never shrinks the
allocated memory. If I want to reclaim memory, I must explicitly use
shrink_to_fit/squeeze.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Thiago Macieira]

On Wednesday, 9 September 2020 06:38:01 PDT Andrei Golubev wrote:
> > I don't understand what this means. Am I supposed to reserve a
> > container to its current size before erasing elements
> > from it, if I don't want the erase to shrink it?
> Yes.

If we implement this behaviour.

That's the dilemma that Andrei was talking about. Either solution is valid and 
both have a way for you to tell QList to do what you want.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel DPG Cloud Engineering



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Thiago Macieira]

On Wednesday, 9 September 2020 02:17:52 PDT Giuseppe D'Angelo via Development 
wrote:
> > *- except when detach is necessary or shrinking happens
> 
> This is another against automatic shrinking; it may invalidate
> everything rather than just the erased area. I don't understand the
> "detach" part, is it about holding references across a detach, so they
> refer to the original container, not the detached one? That's not
> different from we always had, isn't it?

It means that if you try to erase from a container that is currently shared, 
it will detach. When it does, the detached container's size will be evaluated 
in terms of the size of the container, not the capacity the shared container 
had.

Obviously all iterators in a recently-detached container point to newly 
allocated memory, not to the original memory.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel DPG Cloud Engineering



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Andrei Golubev]

I don't understand what this means. Am I supposed to reserve a
container to its current size before erasing elements
from it, if I don't want the erase to shrink it?
Yes.

--
Best Regards,
Andrei

From: Ville Voutilainen 
Sent: Wednesday, September 9, 2020 4:20 PM
To: Andrei Golubev 
Cc: development@qt-project.org 
Subject: Re: [Development] Important recent changes in QList/QString/QByteArray

On Wed, 9 Sep 2020 at 11:58, Andrei Golubev  wrote:
> On the other hand, "please do not free memory, I still need it" use-case is 
> also justified. However, chances are that when you really need a certain 
> memory to be allocated/preserved, there is a call to QList::reserve() prior 
> to insertions/deletions. And since we do not shrink in case of reserved 
> memory, users should be safe in this case, in general.

I don't understand what this means. Am I supposed to reserve a
container to its current size before erasing elements
from it, if I don't want the erase to shrink it?
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Ville Voutilainen]

On Wed, 9 Sep 2020 at 11:58, Andrei Golubev  wrote:
> On the other hand, "please do not free memory, I still need it" use-case is 
> also justified. However, chances are that when you really need a certain 
> memory to be allocated/preserved, there is a call to QList::reserve() prior 
> to insertions/deletions. And since we do not shrink in case of reserved 
> memory, users should be safe in this case, in general.

I don't understand what this means. Am I supposed to reserve a
container to its current size before erasing elements
from it, if I don't want the erase to shrink it?
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Andrei Golubev]

This is another against automatic shrinking; it may invalidate
everything rather than just the erased area. I don't understand the
"detach" part, is it about holding references across a detach, so they
refer to the original container, not the detached one? That's not
different from we always had, isn't it?
You are correct about detach.
Are the new behaviors autotested for all the leaf classes, incl. things
like detaching a reserve()d container? The usual problem is that the
code duplication in QVector/QString/QByteArray gave them completely
different behaviors (there's a number of bug reports open).
Unlikely fully tested. From what I see, there are certain tests for QList's 
erase (incl. reserve + erase), though there is no behavior validation of what 
happens with capacity.
QList/QString/QBA certainly have behavior differences now. For instance, 
shrinking erase is only done for QList (seems to be an extra point against 
auto-shrinking). Also remember null-termination of QString/QBA and 
QString/QBA::_empty - this stuff affects things like capacity()/size().
Prepend optimization forced to make behavior more aligned, QBA's modifying 
functions were updated to use the same "backend" as QList does. QString (and 
QString-using code) is a bit tougher one, so not fully converted yet. Anyhow, 
the devil is in the details around the operations on elements (e.g. detach, 
reserve).
Guessing that by "detaching a reserve()d container" you ask what the detached 
container knows about prior reservation - detached contained knows that the 
capacity was reserved. Couldn't find tests for that, however.

--
Best Regards,
Andrei

From: Development  on behalf of Giuseppe 
D'Angelo via Development 
Sent: Wednesday, September 9, 2020 12:17 PM
To: development@qt-project.org 
Subject: Re: [Development] Important recent changes in QList/QString/QByteArray

On 09/09/2020 10:55, Andrei Golubev wrote:
> Small update on QList::erase: the iterator invalidation model will be
> aligned with std::vector's erase (in progress of merging the change to
> dev and it got approved already). This means that, when erasing, only
> the iterators that correspond to erased elements and elements after them
> (until end()) will be invalidated*. The corner case is erasing from the
> beginning where formal definition seems to allow right-shifting erasure
> (feel free to prove me wrong) and the only concern I have is when
> someone holds QList::constData()/data() pointer, since this will be
> invalidated in the corner case.

data() is invalidated if one erases begin(), so no problem.

>
> *- except when detach is necessary or shrinking happens

This is another against automatic shrinking; it may invalidate
everything rather than just the erased area. I don't understand the
"detach" part, is it about holding references across a detach, so they
refer to the original container, not the detached one? That's not
different from we always had, isn't it?

> As to whether we should or should not shrink when erasing too many
> elements, there are clearly two opposite opinions (see the "replied to"
> conversation).
> My assumption would be that shrinking in erase has some justification
> (otherwise, why do it in the first place?). Thus, there are already
> Qt-relevant cases that benefit from having smaller memory footprint.
> On the other hand, "please do not free memory, I still need it" use-case
> is also justified. However, chances are that when you really need a
> certain memory to be allocated/preserved, there is a call to
> QList::reserve() prior to insertions/deletions. And since we do not
> shrink in case of reserved memory, users should be safe in this case, in
> general.

Are the new behaviors autotested for all the leaf classes, incl. things
like detaching a reserve()d container? The usual problem is that the
code duplication in QVector/QString/QByteArray gave them completely
different behaviors (there's a number of bug reports open).

--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Giuseppe D'Angelo via Development]

On 09/09/2020 10:55, Andrei Golubev wrote:
Small update on QList::erase: the iterator invalidation model will be 
aligned with std::vector's erase (in progress of merging the change to 
dev and it got approved already). This means that, when erasing, only 
the iterators that correspond to erased elements and elements after them 
(until end()) will be invalidated*. The corner case is erasing from the 
beginning where formal definition seems to allow right-shifting erasure 
(feel free to prove me wrong) and the only concern I have is when 
someone holds QList::constData()/data() pointer, since this will be 
invalidated in the corner case.


data() is invalidated if one erases begin(), so no problem.



*- except when detach is necessary or shrinking happens


This is another against automatic shrinking; it may invalidate 
everything rather than just the erased area. I don't understand the 
"detach" part, is it about holding references across a detach, so they 
refer to the original container, not the detached one? That's not 
different from we always had, isn't it?


As to whether we should or should not shrink when erasing too many 
elements, there are clearly two opposite opinions (see the "replied to" 
conversation).
My assumption would be that shrinking in erase has some justification 
(otherwise, why do it in the first place?). Thus, there are already 
Qt-relevant cases that benefit from having smaller memory footprint.
On the other hand, "please do not free memory, I still need it" use-case 
is also justified. However, chances are that when you really need a 
certain memory to be allocated/preserved, there is a call to 
QList::reserve() prior to insertions/deletions. And since we do not 
shrink in case of reserved memory, users should be safe in this case, in 
general.


Are the new behaviors autotested for all the leaf classes, incl. things 
like detaching a reserve()d container? The usual problem is that the 
code duplication in QVector/QString/QByteArray gave them completely 
different behaviors (there's a number of bug reports open).


--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts



smime.p7s
Description: S/MIME Cryptographic Signature
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Andrei Golubev]

Hello,

Small update on QList::erase: the iterator invalidation model will be aligned 
with std::vector's erase (in progress of merging the change to dev and it got 
approved already). This means that, when erasing, only the iterators that 
correspond to erased elements and elements after them (until end()) will be 
invalidated*. The corner case is erasing from the beginning where formal 
definition seems to allow right-shifting erasure (feel free to prove me wrong) 
and the only concern I have is when someone holds QList::constData()/data() 
pointer, since this will be invalidated in the corner case.

*- except when detach is necessary or shrinking happens

As to whether we should or should not shrink when erasing too many elements, 
there are clearly two opposite opinions (see the "replied to" conversation).
My assumption would be that shrinking in erase has some justification 
(otherwise, why do it in the first place?). Thus, there are already Qt-relevant 
cases that benefit from having smaller memory footprint.
On the other hand, "please do not free memory, I still need it" use-case is 
also justified. However, chances are that when you really need a certain memory 
to be allocated/preserved, there is a call to QList::reserve() prior to 
insertions/deletions. And since we do not shrink in case of reserved memory, 
users should be safe in this case, in general.

Feel free to share your opinion or ask questions.

--
Best Regards,
Andrei


From: Development  on behalf of Иван 
Комиссаров 
Sent: Thursday, September 3, 2020 2:34 PM
To: Jaroslaw Kobus 
Cc: Qt development mailing list 
Subject: Re: [Development] Important recent changes in QList/QString/QByteArray

In the awful world of the standard library we call vector.shrink_to_fit() to 
ensure that.
Note that while you have your «please free the memory» use-case someone might 
have the «please do not free memory, I still need it» use-case.

Ivan

3 сент. 2020 г., в 09:40, Jaroslaw Kobus 
mailto:jaroslaw.ko...@qt.io>> написал(а):


From: Development 
mailto:development-boun...@qt-project.org>> 
on behalf of Giuseppe D'Angelo via Development 
mailto:development@qt-project.org>>
Sent: Wednesday, September 2, 2020 9:37 PM
To: Andrei Golubev; 
development@qt-project.org<mailto:development@qt-project.org>; Ville Voutilainen
Subject: Re: [Development] Important recent changes in  QList/QString/QByteArray

On 02/09/2020 21:18, Andrei Golubev wrote:
Also not sure whether it is an implementation detail or the behavior
that should always be anticipated.

People build performance sensitive code assuming the cost of certain
operations -- like, assuming that erasing elements from a vector never
reallocates it; and that the only operation that sheds capacity is
squeeze(), everything else (incl. clear(), incl. resize(0)) keeps the
capacity (*). We should stop backstabbing them...

People sometimes care about memory consumption, too. If user's object
contains a vector consisting of exactly one element, he may be surprised
that it still consumes a place for one million elements, just because one
minute ago he removed 999.999 items. If he is creating hundreds
of such objects sequentially, his app may not run at all (however, will perform 
very well).

Jarek
___
Development mailing list
Development@qt-project.org<mailto:Development@qt-project.org>
https://lists.qt-project.org/listinfo/development

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Иван Комиссаров]

In the awful world of the standard library we call vector.shrink_to_fit() to 
ensure that.
Note that while you have your «please free the memory» use-case someone might 
have the «please do not free memory, I still need it» use-case.

Ivan

> 3 сент. 2020 г., в 09:40, Jaroslaw Kobus  написал(а):
> 
>> 
>> From: Development  on behalf of Giuseppe 
>> D'Angelo via Development 
>> Sent: Wednesday, September 2, 2020 9:37 PM
>> To: Andrei Golubev; development@qt-project.org; Ville Voutilainen
>> Subject: Re: [Development] Important recent changes in  
>> QList/QString/QByteArray
>> 
>> On 02/09/2020 21:18, Andrei Golubev wrote:
>>> Also not sure whether it is an implementation detail or the behavior
>>> that should always be anticipated.
>> 
>> People build performance sensitive code assuming the cost of certain
>> operations -- like, assuming that erasing elements from a vector never
>> reallocates it; and that the only operation that sheds capacity is
>> squeeze(), everything else (incl. clear(), incl. resize(0)) keeps the
>> capacity (*). We should stop backstabbing them...
> 
> People sometimes care about memory consumption, too. If user's object
> contains a vector consisting of exactly one element, he may be surprised
> that it still consumes a place for one million elements, just because one
> minute ago he removed 999.999 items. If he is creating hundreds
> of such objects sequentially, his app may not run at all (however, will 
> perform very well).
> 
> Jarek
> ___
> Development mailing list
> Development@qt-project.org <mailto:Development@qt-project.org>
> https://lists.qt-project.org/listinfo/development 
> <https://lists.qt-project.org/listinfo/development>
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[André Pönitz]

On Wed, Sep 02, 2020 at 09:37:49PM +0200, Giuseppe D'Angelo via Development 
wrote:
> On 02/09/2020 21:18, Andrei Golubev wrote:
> > Also not sure whether it is an implementation detail or the behavior
> > that should always be anticipated.
> 
> People build performance sensitive code assuming the cost of certain
> operations -- like, assuming that erasing elements from a vector never
> reallocates it; and that the only operation that sheds capacity is
> squeeze(), everything else (incl. clear(), incl. resize(0)) keeps the
> capacity (*). We should stop backstabbing them...

People do not build *truly* performance sensitive code using Qt containers.

The reference counting is very convenient, but it comes with a price tag
for non-const item accesses for checking the reference count. 

In the (absolutley rare) cases of "performance trumps everything" *this*
is not acceptable.

The only backstabbing that happens in this area is the removal of Qt
container convenience functionality based on performance arguments that
are not applicable in almost all cases, or on a "does not exist in the
Standard Library so it can't be good" base.

Andre'
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Giuseppe D'Angelo via Development]

On 02/09/2020 21:18, Andrei Golubev wrote:
Also not sure whether it is an implementation detail or the behavior 
that should always be anticipated.


People build performance sensitive code assuming the cost of certain 
operations -- like, assuming that erasing elements from a vector never 
reallocates it; and that the only operation that sheds capacity is 
squeeze(), everything else (incl. clear(), incl. resize(0)) keeps the 
capacity (*). We should stop backstabbing them...



(*) Which of course opens an interesting discussion in case of these 
operations detach



Thanks,
--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts



smime.p7s
Description: S/MIME Cryptographic Signature
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Andrei Golubev]

Even if the capacity has been reserved by the user?
No, in the case of reserved capacity we do not shrink. However, unlikely every 
call to erase that is to break is covered by a reserve somewhere earlier in the 
code. Also not sure whether it is an implementation detail or the behavior that 
should always be anticipated.
> With this setup, one might be tempted to optimize erasure in the first
> half of the container by shifting elements towards the end (rather than
> from the end towards the beginning), as it would be cheaper. I guess
> that's what's happening here?
>
>
> Yes, this is what's happening now. But I am considering switching back to 
> std::vector::erase mode, there's a patch for that already. Let's see how it 
> goes.

After feature freeze? :)
Well, technically there were no explicit guarantees on erase beforehand (also 
see the shrinking thing), so my suggestion is to not even consider new update 
(if it is going to happen) as a behavior change in the first place (read as can 
be done after FF). Once we decide upon it, the plan is to document it => 
effectively fix the behavior at least until Qt 7.

--
Best Regards,
Andrei


From: Development  on behalf of Giuseppe 
D'Angelo via Development 
Sent: Wednesday, September 2, 2020 8:21 PM
To: development@qt-project.org 
Subject: Re: [Development] Important recent changes in QList/QString/QByteArray

Il 02/09/20 17:38, Andrei Golubev ha scritto:
>
> But yes, good job bringing this up. In Qt 6 code base we also do the
> shrinking in erase. So this already is not aligned with std::vector.

Even if the capacity has been reserved by the user?

Thanks,
--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Ville Voutilainen]

On Wed, 2 Sep 2020 at 17:16, Andrei Golubev  wrote:
>
> With this setup, one might be tempted to optimize erasure in the first
> half of the container by shifting elements towards the end (rather than
> from the end towards the beginning), as it would be cheaper. I guess
> that's what's happening here?
>
>
> Yes, this is what's happening now. But I am considering switching back to 
> std::vector::erase mode, there's a patch for that already. Let's see how it 
> goes.


After feature freeze? :)
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Giuseppe D'Angelo via Development]

Il 02/09/20 17:38, Andrei Golubev ha scritto:


But yes, good job bringing this up. In Qt 6 code base we also do the 
shrinking in erase. So this already is not aligned with std::vector.


Even if the capacity has been reserved by the user?

Thanks,
--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts



smime.p7s
Description: Firma crittografica S/MIME
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Andrei Golubev]

I thought QVector and the older QList could decide that the container's
allocated space was too big compared to the the number of elements and
reallocate on erase() too. This doesn't seem like a new requirement.
Actually, this is something not done in Qt 5.15 as far as I can see: QList does 
not seem to have such a code path at all (and no mentions of such), QVector has 
a fixme yet follows std::vector policy looks like.
But yes, good job bringing this up. In Qt 6 code base we also do the shrinking 
in erase. So this already is not aligned with std::vector.

--
Best Regards,
Andrei

From: Development  on behalf of Thiago 
Macieira 
Sent: Wednesday, September 2, 2020 5:37 PM
To: development@qt-project.org 
Subject: Re: [Development] Important recent changes in QList/QString/QByteArray

On Wednesday, 2 September 2020 05:28:33 PDT Ville Voutilainen wrote:
> No. I mean "before", not "after". That's how std::vector works. An
> erase is a shrinking operation
> that keeps everything before the erased position untouched and
> left-shifts everything after it. If you
> want a prepend optimization, you can get it without changing the way
> erase works and what the
> invalidation guarantees of erase are, so I'd like to understand what
> in our implementation strategy
> necessitates this change to those guarantees.

I thought QVector and the older QList could decide that the container's
allocated space was too big compared to the the number of elements and
reallocate on erase() too. This doesn't seem like a new requirement.

--
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel DPG Cloud Engineering



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Thiago Macieira]

On Wednesday, 2 September 2020 05:28:33 PDT Ville Voutilainen wrote:
> No. I mean "before", not "after". That's how std::vector works. An
> erase is a shrinking operation
> that keeps everything before the erased position untouched and
> left-shifts everything after it. If you
> want a prepend optimization, you can get it without changing the way
> erase works and what the
> invalidation guarantees of erase are, so I'd like to understand what
> in our implementation strategy
> necessitates this change to those guarantees.

I thought QVector and the older QList could decide that the container's 
allocated space was too big compared to the the number of elements and 
reallocate on erase() too. This doesn't seem like a new requirement.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel DPG Cloud Engineering



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Thiago Macieira]

On Wednesday, 2 September 2020 00:39:04 PDT Иван Комиссаров wrote:
> If you write a vector and your vector doesn’t do what the standard says (and
> cppref literally quotes the paper here
> http://www.open-std.org/JTC1/SC22/WG21/docs/papers/2011/n3242.pdf
> ) than
> you have to have slightly better arguments than that.

QVector does not comply and does not intend to comply with the standard's 
requirements. In fact, it intends to deviate from some of the requirements, 
especially those relating to exception-safety and some of the complexity 
requirements. And obviously QList does not comply with the standard's list 
requirements, since it's the same as QVector in Qt 6.  It's intentional.

Therefore, cppreference is irrelevant for the discussion.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel DPG Cloud Engineering



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Andrei Golubev]

With this setup, one might be tempted to optimize erasure in the first
half of the container by shifting elements towards the end (rather than
from the end towards the beginning), as it would be cheaper. I guess
that's what's happening here?

Yes, this is what's happening now. But I am considering switching back to 
std::vector::erase mode, there's a patch for that already. Let's see how it 
goes.

--
Best Regards,
Andrei

From: Development  on behalf of Giuseppe 
D'Angelo via Development 
Sent: Wednesday, September 2, 2020 4:47 PM
To: Ville Voutilainen 
Cc: Qt development mailing list 
Subject: Re: [Development] Important recent changes in QList/QString/QByteArray

Il 02/09/20 10:04, Ville Voutilainen ha scritto:
> Interesting. I'm curious what sort of repacking happens on erase, and why
> it wasn't done in such a way that e.g. QVector is 4 pointers instead
> of 3, so that
> the element storage wouldn't necessarily be at the beginning of the
> allocated block;
> in that approach, a pop_front would merely bump the begin, and erase
> still wouldn't
> invalidate anything before the erased position.

With this setup, one might be tempted to optimize erasure in the first
half of the container by shifting elements towards the end (rather than
from the end towards the beginning), as it would be cheaper. I guess
that's what's happening here?

Thanks,

--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Giuseppe D'Angelo via Development]

Il 02/09/20 10:04, Ville Voutilainen ha scritto:

Interesting. I'm curious what sort of repacking happens on erase, and why
it wasn't done in such a way that e.g. QVector is 4 pointers instead
of 3, so that
the element storage wouldn't necessarily be at the beginning of the
allocated block;
in that approach, a pop_front would merely bump the begin, and erase
still wouldn't
invalidate anything before the erased position.


With this setup, one might be tempted to optimize erasure in the first 
half of the container by shifting elements towards the end (rather than 
from the end towards the beginning), as it would be cheaper. I guess 
that's what's happening here?


Thanks,

--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts



smime.p7s
Description: Firma crittografica S/MIME
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Giuseppe D'Angelo via Development]

Il 02/09/20 15:32, Dmitriy Purgin ha scritto:
As far as I remember, until recently, for years we were told in this 
mailing list to move away from QList and use QVector as default, because 
QList will be deprecated/removed in Qt 6 anyway.


Search in this ML's archives. The short story:

* QList became QVector, meaning it's got QVector behavior
* To avoid changing a ton of code, QList is the type used in API and 
"list" suffix is kept in names


My 2 c,

--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts



smime.p7s
Description: Firma crittografica S/MIME
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Dmitriy Purgin]

By the way, a bit off topic but what happened to the plan to make QVector
the primary container in Qt 6, and why has it been turned the other way
round? The official documentation [1] still refers to Marc's article [2]
that says that QVector should be preferred over QList. As far as I
remember, until recently, for years we were told in this mailing list to
move away from QList and use QVector as default, because QList will be
deprecated/removed in Qt 6 anyway. A quick search reveals [3, 4, 5].

[1] https://doc.qt.io/qt-5/qlist.html
[2] https://marcmutz.wordpress.com/effective-qt/containers/
[3] https://lists.qt-project.org/pipermail/interest/2017-May/027055.html
[4] https://lists.qt-project.org/pipermail/development/2015-July/022283.html
[5] https://lists.qt-project.org/pipermail/development/2019-May/036130.html

Cheers
Dmitriy

On Wed, Sep 2, 2020 at 3:06 PM NIkolai Marchenko 
wrote:

> >   This is the difference between QList and std::vector.
> You are aliasing QVector into a QList if I understand it correctly. This
> means a major compatibility break in a ton of code. Unless QVector also
> worked like that which I doubt was the case.
>
> On Wed, Sep 2, 2020 at 3:53 PM Andrei Golubev 
> wrote:
>
>> > Interesting. I'm curious what sort of repacking happens on erase
>> >
>> > The iterators before or after may be invalidated. Exactly which of the
>> two (before or after) is going to happen depends on the position of the
>> to-be-erased range with regards to the position of the full range.
>>
>> I don't quite understand why a vector erase would invalidate vector
>> elements before the erased position.
>>
>> This is the difference between QList and std::vector.
>>
>> > the element storage wouldn't necessarily be at the beginning of the
>> > allocated block;
>> > in that approach, a pop_front would merely bump the begin, and erase
>> > still wouldn't
>> > invalidate anything before the erased position.
>> >
>> > I guess you mean "erase still wouldn't invalidate anything after the
>> erased position".
>>
>> No. I mean "before", not "after". That's how std::vector works. An
>> erase is a shrinking operation
>> that keeps everything before the erased position untouched and
>> left-shifts everything after it. If you
>> want a prepend optimization, you can get it without changing the way
>> erase works and what the
>> invalidation guarantees of erase are, so I'd like to understand what
>> in our implementation strategy
>> necessitates this change to those guarantees.
>>
>> I do not have use cases at hand so cannot really back up my words by
>> evidence. The erase strategy with invalidation either "before" or "after"
>> is the strategy of Qt 5 QList. Similar thing is done with Qt 6 QList
>> (a.k.a. QVector). Indeed, it's not aligned with std::vector::erase().
>>
>> >But this is effectively undefined (or implementation-specific, as you
>> wish) and may change in future if we discover a more performing strategy.
>> Again, any exceptions should be documented. Please do not assume any
>> specific behavior otherwise.
>>
>> This seems like a potentially quite significant compatibility break,
>> that chopping a vector invalidates all references
>> and iterators to parts of the vector that the chopping didn't touch
>> before.
>>
>>
>> --
>> Best Regards,
>> Andrei
>>
>> --
>> *From:* Ville Voutilainen 
>> *Sent:* Wednesday, September 2, 2020 3:28 PM
>> *To:* Andrei Golubev 
>> *Cc:* Giuseppe D'Angelo ;
>> development@qt-project.org ; Ville
>> Voutilainen 
>> *Subject:* Re: [Development] Important recent changes in
>> QList/QString/QByteArray
>>
>> On Wed, 2 Sep 2020 at 15:19, Andrei Golubev  wrote:
>> > Ville:
>> >
>> > Interesting. I'm curious what sort of repacking happens on erase
>> >
>> > The iterators before or after may be invalidated. Exactly which of the
>> two (before or after) is going to happen depends on the position of the
>> to-be-erased range with regards to the position of the full range.
>>
>> I don't quite understand why a vector erase would invalidate vector
>> elements before the erased position.
>>
>> > the element storage wouldn't necessarily be at the beginning of the
>> > allocated block;
>> > in that approach, a pop_front would merely bump the begin, and erase
>> > still wouldn't
>> > invalidate anything before the erased position.
>> >
>> > I guess you mean 

Re: [Development] Important recent changes in QList/QString/QByteArray

[NIkolai Marchenko]

>   This is the difference between QList and std::vector.
You are aliasing QVector into a QList if I understand it correctly. This
means a major compatibility break in a ton of code. Unless QVector also
worked like that which I doubt was the case.

On Wed, Sep 2, 2020 at 3:53 PM Andrei Golubev  wrote:

> > Interesting. I'm curious what sort of repacking happens on erase
> >
> > The iterators before or after may be invalidated. Exactly which of the
> two (before or after) is going to happen depends on the position of the
> to-be-erased range with regards to the position of the full range.
>
> I don't quite understand why a vector erase would invalidate vector
> elements before the erased position.
>
> This is the difference between QList and std::vector.
>
> > the element storage wouldn't necessarily be at the beginning of the
> > allocated block;
> > in that approach, a pop_front would merely bump the begin, and erase
> > still wouldn't
> > invalidate anything before the erased position.
> >
> > I guess you mean "erase still wouldn't invalidate anything after the
> erased position".
>
> No. I mean "before", not "after". That's how std::vector works. An
> erase is a shrinking operation
> that keeps everything before the erased position untouched and
> left-shifts everything after it. If you
> want a prepend optimization, you can get it without changing the way
> erase works and what the
> invalidation guarantees of erase are, so I'd like to understand what
> in our implementation strategy
> necessitates this change to those guarantees.
>
> I do not have use cases at hand so cannot really back up my words by
> evidence. The erase strategy with invalidation either "before" or "after"
> is the strategy of Qt 5 QList. Similar thing is done with Qt 6 QList
> (a.k.a. QVector). Indeed, it's not aligned with std::vector::erase().
>
> >But this is effectively undefined (or implementation-specific, as you
> wish) and may change in future if we discover a more performing strategy.
> Again, any exceptions should be documented. Please do not assume any
> specific behavior otherwise.
>
> This seems like a potentially quite significant compatibility break,
> that chopping a vector invalidates all references
> and iterators to parts of the vector that the chopping didn't touch before.
>
>
> --
> Best Regards,
> Andrei
>
> ----------
> *From:* Ville Voutilainen 
> *Sent:* Wednesday, September 2, 2020 3:28 PM
> *To:* Andrei Golubev 
> *Cc:* Giuseppe D'Angelo ;
> development@qt-project.org ; Ville
> Voutilainen 
> *Subject:* Re: [Development] Important recent changes in
> QList/QString/QByteArray
>
> On Wed, 2 Sep 2020 at 15:19, Andrei Golubev  wrote:
> > Ville:
> >
> > Interesting. I'm curious what sort of repacking happens on erase
> >
> > The iterators before or after may be invalidated. Exactly which of the
> two (before or after) is going to happen depends on the position of the
> to-be-erased range with regards to the position of the full range.
>
> I don't quite understand why a vector erase would invalidate vector
> elements before the erased position.
>
> > the element storage wouldn't necessarily be at the beginning of the
> > allocated block;
> > in that approach, a pop_front would merely bump the begin, and erase
> > still wouldn't
> > invalidate anything before the erased position.
> >
> > I guess you mean "erase still wouldn't invalidate anything after the
> erased position".
>
> No. I mean "before", not "after". That's how std::vector works. An
> erase is a shrinking operation
> that keeps everything before the erased position untouched and
> left-shifts everything after it. If you
> want a prepend optimization, you can get it without changing the way
> erase works and what the
> invalidation guarantees of erase are, so I'd like to understand what
> in our implementation strategy
> necessitates this change to those guarantees.
>
> >But this is effectively undefined (or implementation-specific, as you
> wish) and may change in future if we discover a more performing strategy.
> Again, any exceptions should be documented. Please do not assume any
> specific behavior otherwise.
>
> This seems like a potentially quite significant compatibility break,
> that chopping a vector invalidates all references
> and iterators to parts of the vector that the chopping didn't touch before.
> ___
> Development mailing list
> Development@qt-project.org
> https://lists.qt-project.org/listinfo/development
>
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Andrei Golubev]

> Interesting. I'm curious what sort of repacking happens on erase
>
> The iterators before or after may be invalidated. Exactly which of the two 
> (before or after) is going to happen depends on the position of the 
> to-be-erased range with regards to the position of the full range.

I don't quite understand why a vector erase would invalidate vector
elements before the erased position.
This is the difference between QList and std::vector.
> the element storage wouldn't necessarily be at the beginning of the
> allocated block;
> in that approach, a pop_front would merely bump the begin, and erase
> still wouldn't
> invalidate anything before the erased position.
>
> I guess you mean "erase still wouldn't invalidate anything after the erased 
> position".

No. I mean "before", not "after". That's how std::vector works. An
erase is a shrinking operation
that keeps everything before the erased position untouched and
left-shifts everything after it. If you
want a prepend optimization, you can get it without changing the way
erase works and what the
invalidation guarantees of erase are, so I'd like to understand what
in our implementation strategy
necessitates this change to those guarantees.
I do not have use cases at hand so cannot really back up my words by evidence. 
The erase strategy with invalidation either "before" or "after" is the strategy 
of Qt 5 QList. Similar thing is done with Qt 6 QList (a.k.a. QVector). Indeed, 
it's not aligned with std::vector::erase().
>But this is effectively undefined (or implementation-specific, as you wish) 
>and may change in future if we discover a more performing strategy. Again, any 
>exceptions should be documented. Please do not assume any specific behavior 
>otherwise.

This seems like a potentially quite significant compatibility break,
that chopping a vector invalidates all references
and iterators to parts of the vector that the chopping didn't touch before.

--
Best Regards,
Andrei


From: Ville Voutilainen 
Sent: Wednesday, September 2, 2020 3:28 PM
To: Andrei Golubev 
Cc: Giuseppe D'Angelo ; development@qt-project.org 
; Ville Voutilainen 
Subject: Re: [Development] Important recent changes in QList/QString/QByteArray

On Wed, 2 Sep 2020 at 15:19, Andrei Golubev  wrote:
> Ville:
>
> Interesting. I'm curious what sort of repacking happens on erase
>
> The iterators before or after may be invalidated. Exactly which of the two 
> (before or after) is going to happen depends on the position of the 
> to-be-erased range with regards to the position of the full range.

I don't quite understand why a vector erase would invalidate vector
elements before the erased position.

> the element storage wouldn't necessarily be at the beginning of the
> allocated block;
> in that approach, a pop_front would merely bump the begin, and erase
> still wouldn't
> invalidate anything before the erased position.
>
> I guess you mean "erase still wouldn't invalidate anything after the erased 
> position".

No. I mean "before", not "after". That's how std::vector works. An
erase is a shrinking operation
that keeps everything before the erased position untouched and
left-shifts everything after it. If you
want a prepend optimization, you can get it without changing the way
erase works and what the
invalidation guarantees of erase are, so I'd like to understand what
in our implementation strategy
necessitates this change to those guarantees.

>But this is effectively undefined (or implementation-specific, as you wish) 
>and may change in future if we discover a more performing strategy. Again, any 
>exceptions should be documented. Please do not assume any specific behavior 
>otherwise.

This seems like a potentially quite significant compatibility break,
that chopping a vector invalidates all references
and iterators to parts of the vector that the chopping didn't touch before.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Ville Voutilainen]

On Wed, 2 Sep 2020 at 15:19, Andrei Golubev  wrote:
> Ville:
>
> Interesting. I'm curious what sort of repacking happens on erase
>
> The iterators before or after may be invalidated. Exactly which of the two 
> (before or after) is going to happen depends on the position of the 
> to-be-erased range with regards to the position of the full range.

I don't quite understand why a vector erase would invalidate vector
elements before the erased position.

> the element storage wouldn't necessarily be at the beginning of the
> allocated block;
> in that approach, a pop_front would merely bump the begin, and erase
> still wouldn't
> invalidate anything before the erased position.
>
> I guess you mean "erase still wouldn't invalidate anything after the erased 
> position".

No. I mean "before", not "after". That's how std::vector works. An
erase is a shrinking operation
that keeps everything before the erased position untouched and
left-shifts everything after it. If you
want a prepend optimization, you can get it without changing the way
erase works and what the
invalidation guarantees of erase are, so I'd like to understand what
in our implementation strategy
necessitates this change to those guarantees.

>But this is effectively undefined (or implementation-specific, as you wish) 
>and may change in future if we discover a more performing strategy. Again, any 
>exceptions should be documented. Please do not assume any specific behavior 
>otherwise.

This seems like a potentially quite significant compatibility break,
that chopping a vector invalidates all references
and iterators to parts of the vector that the chopping didn't touch before.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Andrei Golubev]

Hello,

Let me try to improve the wording from the original mail and clarify certain 
things. Sorry for not being pedantic enough at the beginning.

"Non-detaching non-const modifying operations" mean non-detaching non-const 
member functions of the QCCs*, sorry for ambiguation.

*Qt Contiguous Container
(Thanks for the abbreviation, by the way).


Now, let's distinguish between non-const member functions:

  *   Mutating - affect the number of elements in container or space 
arrangement:
 *   Number of elements: erase, insert, append, prepend (also operator 
overloads - e.g. operator+=), etc.
 *   Space arrangement: squeeze, resize, reserve
  *   Non-mutating - still non-const but do not change the number of elements 
or space arrangement:
 *   Data accessors: begin(), end(), data(), operator[], at() and so on

With the difference in mind, more precise definition of "non-detaching 
non-const modifying operation" is any mutating member function of the QCC.
So, any mutating member function of QCC can invalidate 
iterators/pointers/references to elements.
For non-mutating member functions pre-existing requirements should hold without 
changes (at least from the point of prepend optimization and caused 
disruptions), for instance, begin() may cause detach and invalidate your 
iterators.
So, now, formally, std::sort(v.begin(), v.end()) risks undefined
behavior? E.g. begin() returns the begin iterator without touching
anything, but end() decides to invalidate all the iterators.
This should be a defined behavior regardless of whether v needs to be detached 
or not. If it's suddenly UB, it's a bug in the QCC. (In detach case, we would 
detach before returning begin/end so this should also be safe or am I wrong?)

However, doing something like the following is UB:

auto begin = v.begin(); auto end = b.end();
std::sort(begin, end);
v.copyAppend(42);   // assume 42 is of suitable type and can be appended to v
std::sort(begin, end);  // UB, since copyAppend() _may have invalidated_ the 
iterators

Note: the case is UB even when there is enough capacity to store 42 without 
reallocation, this is an example of what changed. Yet, from what I see, there 
were no guarantees in documentation beforehand, so most likely:
Your code was incorrect in the past, it just happened to work.

Now, this "_may have invalidated_" depends a lot on a) specific member function 
called; b) the way QCC is filled with data; c) previous modifying (member) 
operations done on the QCC and this is probably not an exhaustive list of 
things to consider.
Since the behavior of invalidation is rather non-deterministic, at this moment 
it is easier (for library developers, not for users) to consider the 
invalidation to happen always. Any exceptions or cases in which 
iterators/pointers/references would stay valid should be documented explicitly.

As for Guiseppe's question:
This is the contract, and it's OK. I don't think however that this is
what was intended by OP. Rather, that calling a non-const function may
repack/reallocate a QCC (*), causing invalidation of all references and
iterators, even if the container was NOT shared in the first place.

Did I misunderstand the problem?
If the function called is a mutating member function, then it causes the 
invalidation of all references and iterators, yes. It is correct (and only 
correct at this stage) to assume that it happens _always_ for any mutating 
member function.

Ville:
Interesting. I'm curious what sort of repacking happens on erase
The iterators before or after may be invalidated. Exactly which of the two 
(before or after) is going to happen depends on the position of the 
to-be-erased range with regards to the position of the full range.
the element storage wouldn't necessarily be at the beginning of the
allocated block;
in that approach, a pop_front would merely bump the begin, and erase
still wouldn't
invalidate anything before the erased position.
I guess you mean "erase still wouldn't invalidate anything after the erased 
position".  But this is effectively undefined (or implementation-specific, as 
you wish) and may change in future if we discover a more performing strategy. 
Again, any exceptions should be documented. Please do not assume any specific 
behavior otherwise.

--
Best Regards,
Andrei

From: Development  on behalf of Ville 
Voutilainen 
Sent: Wednesday, September 2, 2020 11:04 AM
To: Giuseppe D'Angelo 
Cc: Qt development mailing list 
Subject: Re: [Development] Important recent changes in QList/QString/QByteArray

On Tue, 1 Sep 2020 at 20:44, Giuseppe D'Angelo via Development
 wrote:
>
> Il 01/09/20 19:33, Thiago Macieira ha scritto:
> > All non-const functions that may detach should be coded so they DO detach.
> > That is, after any and all non-const functions, the refcount of the 
> > container
> > should be 1.
>
> This is the contract, and i

Re: [Development] Important recent changes in QList/QString/QByteArray

[Ville Voutilainen]

On Tue, 1 Sep 2020 at 20:44, Giuseppe D'Angelo via Development
 wrote:
>
> Il 01/09/20 19:33, Thiago Macieira ha scritto:
> > All non-const functions that may detach should be coded so they DO detach.
> > That is, after any and all non-const functions, the refcount of the 
> > container
> > should be 1.
>
> This is the contract, and it's OK. I don't think however that this is
> what was intended by OP. Rather, that calling a non-const function may
> repack/reallocate a QCC (*), causing invalidation of all references and
> iterators, even if the container was NOT shared in the first place.
>
> Did I misunderstand the problem?

Interesting. I'm curious what sort of repacking happens on erase, and why
it wasn't done in such a way that e.g. QVector is 4 pointers instead
of 3, so that
the element storage wouldn't necessarily be at the beginning of the
allocated block;
in that approach, a pop_front would merely bump the begin, and erase
still wouldn't
invalidate anything before the erased position.
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Иван Комиссаров]

If you write a vector and your vector doesn’t do what the standard says (and 
cppref literally quotes the paper here 
http://www.open-std.org/JTC1/SC22/WG21/docs/papers/2011/n3242.pdf 
) than you 
have to have slightly better arguments than that.

Ivan

> 1 сент. 2020 г., в 22:31, Thiago Macieira  
> написал(а):
> 
> On Tuesday, 1 September 2020 13:06:39 PDT Иван Комиссаров wrote:
>> Sorry, but you’re wrong here. Cppreference clearly says
> 
> Cppreference is irrelevant.
> 
> -- 
> Thiago Macieira - thiago.macieira (AT) intel.com
>  Software Architect - Intel DPG Cloud Engineering
> 
> 
> 
> ___
> Development mailing list
> Development@qt-project.org
> https://lists.qt-project.org/listinfo/development

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Thiago Macieira]

On Tuesday, 1 September 2020 13:06:39 PDT Иван Комиссаров wrote:
> Sorry, but you’re wrong here. Cppreference clearly says

Cppreference is irrelevant.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel DPG Cloud Engineering



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Иван Комиссаров]

Sorry, but you’re wrong here. Cppreference clearly says

Invalidates iterators and references at or after the point of the erase, 
including the end()  
iterator.

https://en.cppreference.com/w/cpp/container/vector/erase 


It says nothing about invalidating references before the point of insertion

I expect QVector/QList behave exactly like that (if it is not shared like in 
the example in the first mail).

I’d rather prefer not to read another 10 blog posts «how not to shoot in the 
leg with Qt6 QList»

Ivan

> 1 сент. 2020 г., в 19:31, Thiago Macieira  
> написал(а):
> 
> On Tuesday, 1 September 2020 09:05:48 PDT NIkolai Marchenko wrote:
>> Wait what? The switch to Qt6 was supposed to become a painless process yet
>> you're introducing important and hard to notice in real code changes that
>> can result in undefined behaviour?
>> What? WHAT?!
> 
> There's no violation. Your code was incorrect in the past, it just happened 
> to 
> work.
> 
> Assume any and all non-const function will invalidate iterators.
> 
> -- 
> Thiago Macieira - thiago.macieira (AT) intel.com
>  Software Architect - Intel DPG Cloud Engineering
> 
> 
> 
> ___
> Development mailing list
> Development@qt-project.org
> https://lists.qt-project.org/listinfo/development

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Thiago Macieira]

On Tuesday, 1 September 2020 10:41:33 PDT Giuseppe D'Angelo via Development 
wrote:
> This is the contract, and it's OK. I don't think however that this is
> what was intended by OP. Rather, that calling a non-const function may
> repack/reallocate a QCC (*), causing invalidation of all references and
> iterators, even if the container was NOT shared in the first place.
> 
> Did I misunderstand the problem?

I hope you are. Calling begin() on a container that is already detached() 
shouldn't cause it to reallocate if it was already unshared. Calling prepend() 
or append() might.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel DPG Cloud Engineering



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Thiago Macieira]

On Tuesday, 1 September 2020 10:46:21 PDT Giuseppe D'Angelo via Development 
wrote:
> Pedantically: if any and all non-const function will invalidate
> iterators, then calling any algorithm that takes an iterator range
> becomes formally impossible (unless one obtains an iterator range in one
> function call, but QVector doesn't have that).

Ok, the wording was incorrect then.

Non-const mutating functions must be assumed to invalidate all iterators. If 
you remove or append or insert an item in the container, the iterators are 
gone.

Non-const non-mutating functions (like begin()) invalidate iterators obtained 
from const functions only. This was the QT_STRICT_ITERATORS usage and remains 
so. Those functions must ensure the reference count is 1 after the call. That 
means a second non-const non-mutating function does not have to detach again. 
Therefore, your requirement that (begin(), end()) is a valid pair is met, 
regardless of the order in which they are called.

This is most visible in functions like find(): if they don't find the item, 
did they detach?

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel DPG Cloud Engineering



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Giuseppe D'Angelo via Development]

Il 01/09/20 19:31, Thiago Macieira ha scritto:

There's no violation. Your code was incorrect in the past, it just happened to
work.


This isn't true...



Assume any and all non-const function will invalidate iterators.


... because this is sketchy. It has never been 100% documented, but 
de-facto "the" behavor everyone is relying upon, due to Hyrum's law.


I don't have the energy to fight this, but my question was same lines -- 
legalese / documentation: what's the guaranteed behavior?


Pedantically: if any and all non-const function will invalidate 
iterators, then calling any algorithm that takes an iterator range 
becomes formally impossible (unless one obtains an iterator range in one 
function call, but QVector doesn't have that).


My 2 c,
--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts



smime.p7s
Description: Firma crittografica S/MIME
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[NIkolai Marchenko]

> There's no violation. Your code was incorrect in the past, it just
happened to work.
I have to remind you about the Qt5.5 and the QDebug behaviour change that
had to be rolled back because of the backlash.
This has ramifications way bigger than a simple logging breakage.
If a mission critical application triggers a bug because of the code that
worked previously but is UB then, you're going to have a massive reputation
hit on Qt as a whole.
This also goes completely contrary to the goal of making the switch to qt6
painless.

On Tue, Sep 1, 2020 at 8:35 PM Thiago Macieira 
wrote:

> On Tuesday, 1 September 2020 09:05:48 PDT NIkolai Marchenko wrote:
> > Wait what? The switch to Qt6 was supposed to become a painless process
> yet
> > you're introducing important and hard to notice in real code changes that
> > can result in undefined behaviour?
> > What? WHAT?!
>
> There's no violation. Your code was incorrect in the past, it just
> happened to
> work.
>
> Assume any and all non-const function will invalidate iterators.
>
> --
> Thiago Macieira - thiago.macieira (AT) intel.com
>   Software Architect - Intel DPG Cloud Engineering
>
>
>
> ___
> Development mailing list
> Development@qt-project.org
> https://lists.qt-project.org/listinfo/development
>
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Giuseppe D'Angelo via Development]

Il 01/09/20 19:33, Thiago Macieira ha scritto:

All non-const functions that may detach should be coded so they DO detach.
That is, after any and all non-const functions, the refcount of the container
should be 1.


This is the contract, and it's OK. I don't think however that this is 
what was intended by OP. Rather, that calling a non-const function may 
repack/reallocate a QCC (*), causing invalidation of all references and 
iterators, even if the container was NOT shared in the first place.


Did I misunderstand the problem?


(*) Qt Contiguous Container

Thanks,
--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts



smime.p7s
Description: Firma crittografica S/MIME
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Thiago Macieira]

On Tuesday, 1 September 2020 08:53:36 PDT Giuseppe D'Angelo via Development 
wrote:
> So, now, formally, std::sort(v.begin(), v.end()) risks undefined
> behavior? E.g. begin() returns the begin iterator without touching
> anything, but end() decides to invalidate all the iterators.

We've seen this in QCborMap and QJsonObject too: trying to optimise code for 
not detaching causes problems.

All non-const functions that may detach should be coded so they DO detach. 
That is, after any and all non-const functions, the refcount of the container 
should be 1.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel DPG Cloud Engineering



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Thiago Macieira]

On Tuesday, 1 September 2020 09:05:48 PDT NIkolai Marchenko wrote:
> Wait what? The switch to Qt6 was supposed to become a painless process yet
> you're introducing important and hard to notice in real code changes that
> can result in undefined behaviour?
> What? WHAT?!

There's no violation. Your code was incorrect in the past, it just happened to 
work.

Assume any and all non-const function will invalidate iterators.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel DPG Cloud Engineering



___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[NIkolai Marchenko]

>  A more subtle issue concerns the users of QVector.
Wait what? The switch to Qt6 was supposed to become a painless process yet
you're introducing important and hard to notice in real code changes that
can result in undefined behaviour?
What? WHAT?!

On Tue, Sep 1, 2020 at 6:58 PM Giuseppe D'Angelo via Development <
development@qt-project.org> wrote:

> Hi,
>
> Thanks for the heads up!
>
> Il 31/08/20 13:50, Andrei Golubev ha scritto:
> > The invalidation existed before for cases when a container could detach
> > (due to copy-on-write) or reallocate (due to growing or squeezing).
>
> This sounds incorrect? Which invalidation did happen due to COW?
>
> > Now
> > this is also true for non-detaching, non-reallocating modifying
> operations.
>
> So, now, formally, std::sort(v.begin(), v.end()) risks undefined
> behavior? E.g. begin() returns the begin iterator without touching
> anything, but end() decides to invalidate all the iterators.
>
> Yes, I assume that in practice begin() would already invalidate, and
> end() wouldn't, so it would work, but I'm asking what's formal model
> now. Is there a way to know that the next non-const call is going to
> invalidate everything?
>
>
> Side question, does anyone see a problem with begin() / data() / etc. no
> longer be noexcept O(1) operations?
>
>
> Thanks,
>
> --
> Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
> KDAB (France) S.A.S., a KDAB Group company
> Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
> KDAB - The Qt, C++ and OpenGL Experts
>
> ___
> Development mailing list
> Development@qt-project.org
> https://lists.qt-project.org/listinfo/development
>
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Giuseppe D'Angelo via Development]

Hi,

Thanks for the heads up!

Il 31/08/20 13:50, Andrei Golubev ha scritto:
The invalidation existed before for cases when a container could detach 
(due to copy-on-write) or reallocate (due to growing or squeezing). 


This sounds incorrect? Which invalidation did happen due to COW?

Now 
this is also true for non-detaching, non-reallocating modifying operations.


So, now, formally, std::sort(v.begin(), v.end()) risks undefined 
behavior? E.g. begin() returns the begin iterator without touching 
anything, but end() decides to invalidate all the iterators.


Yes, I assume that in practice begin() would already invalidate, and 
end() wouldn't, so it would work, but I'm asking what's formal model 
now. Is there a way to know that the next non-const call is going to 
invalidate everything?



Side question, does anyone see a problem with begin() / data() / etc. no 
longer be noexcept O(1) operations?



Thanks,

--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Senior Software Engineer
KDAB (France) S.A.S., a KDAB Group company
Tel. France +33 (0)4 90 84 08 53, http://www.kdab.com
KDAB - The Qt, C++ and OpenGL Experts



smime.p7s
Description: Firma crittografica S/MIME
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development

Re: [Development] Important recent changes in QList/QString/QByteArray

[Daniel Engelke]

Hey,


ASan for x86 Release targets is available in MSVC since 
https://devblogs.microsoft.com/cppblog/addresssanitizer-asan-for-windows-with-msvc/
 , and recently also for x64 + Debug builds 
https://devblogs.microsoft.com/cppblog/asan-for-windows-x64-and-debug-build-support/
To add some details.


BR,
Dan


---

Daniel EngelkeSoftware Engineer
basysKom GmbH
Robert-Bosch-Str. 7 | 64293 Darmstadt | Germany
Tel: +49 6151 870 589 -0 | Fax: -199
daniel.enge...@basyskom.com | www.basyskom.com


Handelsregister: Darmstadt HRB 9352
Geschaeftsfuehrende Partner: Heike Ziegler, Alexander Sorg



 From:   Andrei Golubev  
 To:   "development@qt-project.org"  
 Sent:   8/31/2020 2:01 PM 
 Subject:   [Development] Important recent changes in QList/QString/QByteArray 


 
 
 
 Hello all, 
 
 
 
 
 Over the last week I pushed several patches to enable prepend optimization in 
some core containers, namely, QList, QString, QByteArray. 
 In the light of these changes, there may occur new subtle issues in your code. 
 
 
 
 
 New model
 
 ---
 
 
 
 In a nutshell, prepend optimization code does some magic with the memory 
layout and the stored elements location. 
 This, in turn, affects how you can and cannot use 
iterators/pointers/references to container's elements and their invalidation. 
 
 
 As of today, please consider that any non-const member function of the 
aforementioned container can invalidate "remembered" iterators, pointers or 
references to elements of that container. For example, an iterator is returned 
by QList::begin(), a pointer is  returned by QString::data() (and 
QString::constData()), a reference is returned by QList::at() - this is, of 
course, not an exhaustive list.
 
 The invalidation existed before for cases when a container could detach (due 
to copy-on-write) or reallocate (due to growing or squeezing). Now this is also 
true for non-detaching, non-reallocating, modifying operations. Examples of 
such operations: append,  prepend, insert, emplace, remove, etc.
 
 
 
 
 
 How this affects existing code? 
 --- 
 
 
 To the best of my knowledge, at least certain documentation pieces of QString 
and QByteArray already explained such a thing. So, hopefully, diligent users of 
these 2 containers should be safe.
 
 
 
 QList's behavior changed with regards to the iterators/pointers: as described 
above, using an iterator or a pointer to an element of a QList between 
non-const modifying operations is an undefined behavior (same true for 
references).
 
 
 
 A more subtle issue concerns the users of QVector. Since QVector got unified 
with QList and QList's implementation/behavior changed, old code that was valid 
before may now be a subject to UB. Consider a simple example of the difference: 
 
 
 QVector ints { 0, 1, 2, 3, 4, 5, ... };  // now: effectively QList
 
 auto thirdElemIt = ints.begin() + 2; 
 ints.remove(6);   // removing element at position 6. before: thirdElemIt 
remains valid 
    //  
 now: thirdElemIt may be invalidated
 
 print(*thirdElemIt);  // before: valid enough behavior; now: undefined
 
 
 
 
 
 
 Aiding the debugging process
 
 --- 
 
 
 Below are some basic steps one can do to simplify the debugging of issues:
 
 
 
 1) Enable assertions in the code. This should be possible even if you have a 
release build, there's a macro definition QT_FORCE_ASSERTS that should 
force-enable assertions (albeit not sure how exactly to specify it during build 
stage). 
 
 
 2) Build with address sanitizer. Even basic "sanitize=address" should suffice 
initially. It should be available both for GCC and Clang (guessing that any 
recent GCC/Clang that supports C++17 also supports ASan). MSVC developers some 
time ago also reported that  ASan is now supported. To enable ASan locally with 
CMake build toolchain, one can use "-DFEATURE_sanitize_address=ON" flag as part 
of the CMake input arguments (when building Qt sources). Don't know an 
alternative if you use configure script + qmake. 
 
 
 3) Valgrind's memory checker should cover roughly same issues as address 
sanitizer.
 
 
 
 Note that you do not need to rebuild Qt/your application to use valgrind which 
may be a preferred option sometimes. 
 
 
 
 
 
 
 Feel free to contact me in case you find an issue caused by new changes and 
need a help in resolving it. 
 Sorry for possible inconveniences!
 
 
 
 
 
 --
 
 Best Regards, 
 Andrei
 

___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development
___
Development mailing list
Development@qt-project.org
https://lists.qt-project.org/listinfo/development