Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 10:05:40 Curtis Mitch wrote: consider Q_DECLARE_METATYPE with the appropriate flag (yes, even I'm guessing that you meant to write Q_DECLARE_TYPEINFO here? Yes, _TYPEINFO is what I meant. Thanks, Marc -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
It might be more instructive to describe these by the effects they have: On Friday 10 July 2015 10:05:40 Curtis Mitch wrote: Q_PRIMITIVE_TYPE Imples Q_MOVABLE_TYPE. In addition, the default ctor is replaced by memset(0) and the dtor call with nothing. Q_MOVABLE_TYPE Has nothing to do with C++11 move semantics! The default ctor and the dtor are run, but for relocating an object in memory, e.g. when growing a container, memcpy is used and the old memory does not have it's dtors called. IOW: realloc() can be used on array of these types. Q_COMPLEX_TYPE (the default) the default ctor and dtor are called. In addition, for relocating the type in memory, the copy ctor is used (and the dtor is called on the old memory). The copy ctor could be replaced by the move ctor, but our containers don't do that, yet. Thanks, Marc PS: the docs for QList are an outright lie. QList is not a good default container, and it does not produce less executable code than QVector, but changing the docs is a lot of work and I prefer to focus on fixing the code. But if someone would give the docs a good dose of reality back, I'd be more than willing to review such a change. -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
[Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
Hi, I'll never manage to eradicate inefficient QLists if people continue to add new ones :) So here are some things I'd like _everyone_ to watch out for in reviews and mercilessly -1: - using QList - not using Q_DECLARE_TYPEINFO Let me qualify that a bit: QListC is a _very_ bad idea for virtually any C, because the minutest of mistakes (and the default behaviour _is_ a mistake) can render it utterly and unacceptably inefficient. I won't give you the whole story (google QList harmful for that), but never, ever use QListC *unless*: - sizeof(C) == sizeof(void*) (*both* on 64 and 32-bit platforms!) _and_ C has been declared Q_MOVABLE_TYPE or Q_PRIMITIVE_TYPE -or- - the use is unescapable, because other parts of Qt use it (e.g. QVariant, QModelIndex). The latter doesn't mean that you should _always_ use QList for QVariant or QModelIndex. If all you're doing is local to your class, then by all means use a QVector. QVector is the correct default container. If you actually need list behaviour (ie. linked lists), use QLinkedList instead. It tends to be _faster_ than QList. In private implementation, also consider using std::vectorC, esp. if you never copy it and C is (explictly or implicitly) move-enabled. It typically produces up to 1KiB less executable code than QVector. *Per instantiation*. If in *any* kind of doubt whether QList is acceptable, or any of your container choices, feel free to add me as a reviewer. And please, whenever you add a new type (not just class, *any* type, incl. enums, but excluding QFlags (which are automatically primitive)), strongly consider Q_DECLARE_METATYPE with the appropriate flag (yes, even Q_COMPLEX_TYPE). About the only time this can be considered optional is for polymorphic classes. It should become second nature to Q_DECLARE_TYPEINFO. It should be considered a _must_ to Q_DECLARE_TYPEINFO any type that is put into a QVector, QList or QVariant. In fact, my local copy enforces this at compile-time. I hope to push this work at some point, but of course, that means that each and every occurrence in Qt itself first needs to be fixed, and even then, we can only enable it as an opt-in to not break user code (even if it deserves to be broken). Thanks, Marc -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Relevant industrial buses
PROFINET would be nice... :) it is very complex... but wide used in industrial automation... greetings jan Am 03.07.2015 um 12:13 schrieb Turunen Tuukka: Hi, As you are probably aware, we are developing a new Qt SerialBus module. Target is to have a technology preview in Qt 5.6 with support to CANBus. You can look how it is shaping up at: http://code.qt.io/cgit/qt/qtserialbus.git/ The architecture of the new Qt SerialBus is modular, and we would like to add support to new buses relevant for Qt based systems. One area that Qt is widely used is industrial automation, and therefore we are interested in knowing which are the buses you would like to have supported with this module. We are of course also very much open to contributions to get more buses supported. Yours, Tuukka Turunen Director, RD The Qt Company Piippukatu 11, 40100 Jyväskylä, Finland Email: tuukka.turu...@theqtcompany.com | Mobile: + 358 40 7655 800 www.qt.io |Qt Blog: http://blog.qt.io | Twitter: @QtbyDigia, @Qtproject | Facebook: www.facebook.com/qt ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
-Original Message- From: development-bounces+mitch.curtis=theqtcompany@qt-project.org [mailto:development-bounces+mitch.curtis=theqtcompany.com@qt- project.org] On Behalf Of Marc Mutz Sent: Friday, 10 July 2015 11:04 AM To: development@qt-project.org Subject: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO Hi, I'll never manage to eradicate inefficient QLists if people continue to add new ones :) So here are some things I'd like _everyone_ to watch out for in reviews and mercilessly -1: - using QList - not using Q_DECLARE_TYPEINFO Let me qualify that a bit: QListC is a _very_ bad idea for virtually any C, because the minutest of mistakes (and the default behaviour _is_ a mistake) can render it utterly and unacceptably inefficient. I won't give you the whole story (google QList harmful for that), but never, ever use QListC *unless*: - sizeof(C) == sizeof(void*) (*both* on 64 and 32-bit platforms!) _and_ C has been declared Q_MOVABLE_TYPE or Q_PRIMITIVE_TYPE -or- - the use is unescapable, because other parts of Qt use it (e.g. QVariant, QModelIndex). The latter doesn't mean that you should _always_ use QList for QVariant or QModelIndex. If all you're doing is local to your class, then by all means use a QVector. QVector is the correct default container. If you actually need list behaviour (ie. linked lists), use QLinkedList instead. It tends to be _faster_ than QList. In private implementation, also consider using std::vectorC, esp. if you never copy it and C is (explictly or implicitly) move-enabled. It typically produces up to 1KiB less executable code than QVector. *Per instantiation*. If in *any* kind of doubt whether QList is acceptable, or any of your container choices, feel free to add me as a reviewer. And please, whenever you add a new type (not just class, *any* type, incl. enums, but excluding QFlags (which are automatically primitive)), strongly consider Q_DECLARE_METATYPE with the appropriate flag (yes, even I'm guessing that you meant to write Q_DECLARE_TYPEINFO here? For those (like me) wondering why Marc is advocating the use of this macro, it's briefly explained in the documentation [1]: You can use this macro to specify information about a custom type Type. With accurate type information, Qt's generic containers can choose appropriate storage methods and algorithms. Flags can be one of the following: Q_PRIMITIVE_TYPE specifies that Type is a POD (plain old data) type with no constructor or destructor, or else a type where every bit pattern is a valid object and memcpy() creates a valid independent copy of the object. Q_MOVABLE_TYPE specifies that Type has a constructor and/or a destructor but can be moved in memory using memcpy(). Q_COMPLEX_TYPE (the default) specifies that Type has constructors and/or a destructor and that it may not be moved in memory. [1] http://doc.qt.io/qt-5/qtglobal.html#Q_DECLARE_TYPEINFO Q_COMPLEX_TYPE). About the only time this can be considered optional is for polymorphic classes. It should become second nature to Q_DECLARE_TYPEINFO. It should be considered a _must_ to Q_DECLARE_TYPEINFO any type that is put into a QVector, QList or QVariant. In fact, my local copy enforces this at compile-time. I hope to push this work at some point, but of course, that means that each and every occurrence in Qt itself first needs to be fixed, and even then, we can only enable it as an opt-in to not break user code (even if it deserves to be broken). Thanks, Marc -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
Il 10/07/2015 11:54, Smith Martin ha scritto:
Then I don't see why it is so inherently inefficient. The QList entry is allocated on
the heap anyway. Doesn't QListC just allocate a bigger entry? And if I don't
have the C object stored anywhere else, it has to be somewhere, so why not keep it in
the QList entry?
Because for a wrong type C (*) QList will allocate an array of
pointers to C (not an array of Cs!), then proceed to allocate on the
heap every single C object you put in it (by new'ing them); the array
will then store the pointer to the allocated object.
In other words:
struct C {};
QListC list;
for (auto i = 0; i 100; ++i)
list C{};
Calls operator new at least 101 times.
(*) not movable, or bigger than a void*. And user-defined types are not
movable by default (they're complex). And if we forget
Q_DECLARE_TYPEINFO on public types, we can't add it back because it's
binary incompatible.
Cheers,
--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Software Engineer
KDAB (UK) Ltd., a KDAB Group company | Tel: UK +44-1625-809908
KDAB - The Qt Experts
smime.p7s
Description: Firma crittografica S/MIME
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
-Original Message- From: development-bounces+mitch.curtis=theqtcompany@qt-project.org [mailto:development-bounces+mitch.curtis=theqtcompany.com@qt- project.org] On Behalf Of Giuseppe D'Angelo Sent: Friday, 10 July 2015 12:05 PM To: Smith Martin; development@qt-project.org Subject: Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO [snip] (*) not movable, or bigger than a void*. And user-defined types are not movable by default (they're complex). And if we forget Q_DECLARE_TYPEINFO on public types, we can't add it back because it's binary incompatible. How is it binary incompatible? Cheers, -- Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Software Engineer KDAB (UK) Ltd., a KDAB Group company | Tel: UK +44-1625-809908 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
Do you mean that if I declare this:
typedef QListParsedParameter ParsedParameterList;
where ParsedParameter is:
struct ParsedParameter {
bool qPrivateSignal_;
QString dataType_;
QString name_;
QString defaultValue_;
ParsedParameter() : qPrivateSignal_(false) { }
};
...it will create each list entry as a QListParsedParameter* even though I
told it not to do that?
martin
From: giuseppe.dang...@kdab.com giuseppe.dang...@kdab.com on behalf of
Giuseppe D'Angelo giuseppe.dang...@kdab.com
Sent: Friday, July 10, 2015 12:05 PM
To: Smith Martin; development@qt-project.org
Subject: Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
Il 10/07/2015 11:54, Smith Martin ha scritto:
Then I don't see why it is so inherently inefficient. The QList entry is
allocated on the heap anyway. Doesn't QListC just allocate a bigger entry?
And if I don't have the C object stored anywhere else, it has to be
somewhere, so why not keep it in the QList entry?
Because for a wrong type C (*) QList will allocate an array of
pointers to C (not an array of Cs!), then proceed to allocate on the
heap every single C object you put in it (by new'ing them); the array
will then store the pointer to the allocated object.
In other words:
struct C {};
QListC list;
for (auto i = 0; i 100; ++i)
list C{};
Calls operator new at least 101 times.
(*) not movable, or bigger than a void*. And user-defined types are not
movable by default (they're complex). And if we forget
Q_DECLARE_TYPEINFO on public types, we can't add it back because it's
binary incompatible.
Cheers,
--
Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Software Engineer
KDAB (UK) Ltd., a KDAB Group company | Tel: UK +44-1625-809908
KDAB - The Qt Experts
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
Le vendredi 10 juillet 2015 à 10:12 +, Curtis Mitch a écrit : (*) not movable, or bigger than a void*. And user-defined types are not movable by default (they're complex). And if we forget Q_DECLARE_TYPEINFO on public types, we can't add it back because it's binary incompatible. How is it binary incompatible? I believe this is because of the allocation strategy change in QList. I wonder, however, if explicit template declaration / instanciation (which is a new C++11 feature) would solve this issue. Regards, Julien ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
1) you put pressure on the memory allocator, by asking it to allocate on the heap each individual C object you put in the list. Every single allocation has a cost, plus the overhead you need for bookkeeping, plus the costs of getting your memory fragmented, etc.; This pressure on the memory allocator, what is that? My memory doesn't get fragmented, because I only build lists; I never destroy them. But I do have to put each C somewhere, and if I use QVectorC, doesn't it allocate a lot of them up front? I don't know how many i will need, but the number is usually 0, 1, or 2 -- the number of parameters in a function signature. 2) you make the compiler produce more inefficient code by introducing the layer of indirection; But it will process the list one time only, from beginning to end. 3) you kill caching, as potentially every single access will result in a cache miss (even in the common scenario of a simple forward iteration over your list); Why? All the entries in the list are created at the same time (well, during the parsing of the function signature) martin From: development-bounces+martin.smith=theqtcompany@qt-project.org development-bounces+martin.smith=theqtcompany@qt-project.org on behalf of Giuseppe D'Angelo giuseppe.dang...@kdab.com Sent: Friday, July 10, 2015 1:26 PM To: development@qt-project.org Subject: Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO Il 10/07/2015 13:21, Smith Martin ha scritto: But I'm still not clear on one point. If I use QListC, and it is implemented as QListC* because my C is not a good C, why is this inefficent if the only copy of each C is the one that gets allocated on the heap? I just create the list to store things. It won't ever be moved. I will just process the list when it's time comes. Why is that inefficient? Because 1) you put pressure on the memory allocator, by asking it to allocate on the heap each individual C object you put in the list. Every single allocation has a cost, plus the overhead you need for bookkeeping, plus the costs of getting your memory fragmented, etc.; 2) you make the compiler produce more inefficient code by introducing the layer of indirection; 3) you kill caching, as potentially every single access will result in a cache miss (even in the common scenario of a simple forward iteration over your list); and so on. Cheers, -- Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Software Engineer KDAB (UK) Ltd., a KDAB Group company | Tel: UK +44-1625-809908 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 11:35:34 Smith Martin wrote: 1) you put pressure on the memory allocator, by asking it to allocate on the heap each individual C object you put in the list. Every single allocation has a cost, plus the overhead you need for bookkeeping, plus the costs of getting your memory fragmented, etc.; This pressure on the memory allocator, what is that? My memory doesn't get fragmented, because I only build lists; I never destroy them. But I do have to put each C somewhere, and if I use QVectorC, doesn't it allocate a lot of them up front? I don't know how many i will need, but the number is usually 0, 1, or 2 -- the number of parameters in a function signature. Calling malloc/free is not a cheap operation, when you profile most Qt applications, you'll find these functions nearly always in the top 10, often even the top 5, of functions where most time is spent. 2) you make the compiler produce more inefficient code by introducing the layer of indirection; But it will process the list one time only, from beginning to end. 3) you kill caching, as potentially every single access will result in a cache miss (even in the common scenario of a simple forward iteration over your list); Why? All the entries in the list are created at the same time (well, during the parsing of the function signature) Huh?! I really don't follow your reasoning to both of these points here. I suggest you go watch https://channel9.msdn.com/Events/Build/2013/4-329 and similar talks out there to educate yourself? Whenever you access/traverse a list of pointers, you'll incur cache misses, except in the rare case where you are lucky and the pointers point to contiguous memory locations. And even then, you'd have an indirection that Peppe mentions. Bye -- Milian Wolff | milian.wo...@kdab.com | Software Engineer KDAB (Deutschland) GmbHCo KG, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
Calling malloc/free is not a cheap operation, when you profile most Qt applications, you'll find these functions nearly always in the top 10, often even the top 5, of functions where most time is spent. But, Marc made QVector for QList swap in qdoc several weeks ago for the same kind of thing, but I couldn't see any difference in the run time for qdoc. Huh?! I really don't follow your reasoning to both of these points here. I suggest you go watch https://channel9.msdn.com/Events/Build/2013/4-329 and similar talks out there to educate yourself? Whenever you access/traverse a list of pointers, you'll incur cache misses, except in the rare case where you are lucky and the pointers point to contiguous memory locations. And even then, you'd have an indirection that Peppe mentions. In this case, qdoc creates a QListParsedParameter as it parses a function signature. All the calls to malloc occur one right after another, and then the list is complete. There hasn't been any intervening heap work, so all these ParsedParameters will be sequential and will be allocated right after the QList itself. But more importantly, this was just a temporary mechanism to correct the parsing of friend function declarations, which broke in qdoc, because good old Marc started making friend functions be friend inline functions with the body defined in the friend inline declaration inside the class. I think Marc's goal is to be the first ever software engineer to use every possible combination of C++ features. Still, this one is certainly legal, but qdoc couldn't handle it (qdoc still doesn't have a real C++ parser *** see below***), so I rewrote the function that parses function declarations so that it waits to create the list of parameters until it knows whether it should create the function node in the first place. This QListParsedParameter was meant to be a temporary solution, because that list gets copied into a QListParameter if qdoc finally decides to create the function node. That, of course, is inefficient, but I generally don'[[t try to optimize until I have a working model. *** If anyone wants to explain how I can use the Creator clang C++ parser in qdoc, I could get started on that upgrade to qdoc. It's a big job I don't want to start until I know it makes sense. martin From: milian on behalf of Milian Wolff milian.wo...@kdab.com Sent: Friday, July 10, 2015 2:04 PM To: development@qt-project.org Cc: Smith Martin; Giuseppe D'Angelo Subject: Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO On Friday 10 July 2015 11:35:34 Smith Martin wrote: 1) you put pressure on the memory allocator, by asking it to allocate on the heap each individual C object you put in the list. Every single allocation has a cost, plus the overhead you need for bookkeeping, plus the costs of getting your memory fragmented, etc.; This pressure on the memory allocator, what is that? My memory doesn't get fragmented, because I only build lists; I never destroy them. But I do have to put each C somewhere, and if I use QVectorC, doesn't it allocate a lot of them up front? I don't know how many i will need, but the number is usually 0, 1, or 2 -- the number of parameters in a function signature. Calling malloc/free is not a cheap operation, when you profile most Qt applications, you'll find these functions nearly always in the top 10, often even the top 5, of functions where most time is spent. 2) you make the compiler produce more inefficient code by introducing the layer of indirection; But it will process the list one time only, from beginning to end. 3) you kill caching, as potentially every single access will result in a cache miss (even in the common scenario of a simple forward iteration over your list); Why? All the entries in the list are created at the same time (well, during the parsing of the function signature) Huh?! I really don't follow your reasoning to both of these points here. I suggest you go watch https://channel9.msdn.com/Events/Build/2013/4-329 and similar talks out there to educate yourself? Whenever you access/traverse a list of pointers, you'll incur cache misses, except in the rare case where you are lucky and the pointers point to contiguous memory locations. And even then, you'd have an indirection that Peppe mentions. Bye -- Milian Wolff | milian.wo...@kdab.com | Software Engineer KDAB (Deutschland) GmbHCo KG, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 13:29:30 Marc Mutz wrote: On Friday 10 July 2015 12:18:04 Smith Martin wrote: ...it will create each list entry as a QListParsedParameter* even though I told it not to do that? Yes. But you told it to do that. You used a _list_. Ok, I take the bait: QList is primarily a list, not an array (thus the name). That means that, by default, references to elements in the list are never invalidated until the element is removed again. The problem started when someone optimised QList to contain the elements in an array, but only for *some* types. So, depending on the type and the QTypeInfo for that flag, QList is either like a std::list or like a std::vector (QLinkedList / QVector in Qt speak). And it takes great mental discipline to use it in the proper way, and the resulting code will be highly fragile and/or extremely inefficient. Thanks, Marc -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
Maybe it's just me, but I'm still not understanding your explanation, and I don't think you are answering my question: I have a struct C which is bigger than a pointer. I declare QListC. Does your explanation mean that sometimes (most of the time, in fact), the list will become an array of C* instead of an array of C? martin From: development-bounces+martin.smith=theqtcompany@qt-project.org development-bounces+martin.smith=theqtcompany@qt-project.org on behalf of Marc Mutz marc.m...@kdab.com Sent: Friday, July 10, 2015 1:34 PM To: development@qt-project.org Subject: Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO On Friday 10 July 2015 13:29:30 Marc Mutz wrote: On Friday 10 July 2015 12:18:04 Smith Martin wrote: ...it will create each list entry as a QListParsedParameter* even though I told it not to do that? Yes. But you told it to do that. You used a _list_. Ok, I take the bait: QList is primarily a list, not an array (thus the name). That means that, by default, references to elements in the list are never invalidated until the element is removed again. The problem started when someone optimised QList to contain the elements in an array, but only for *some* types. So, depending on the type and the QTypeInfo for that flag, QList is either like a std::list or like a std::vector (QLinkedList / QVector in Qt speak). And it takes great mental discipline to use it in the proper way, and the resulting code will be highly fragile and/or extremely inefficient. Thanks, Marc -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 12:18:04 Smith Martin wrote: ...it will create each list entry as a QListParsedParameter* even though I told it not to do that? Yes. But you told it to do that. You used a _list_. -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
Il 10/07/2015 12:12, Curtis Mitch ha scritto: How is it binary incompatible? Because the code produced by QListC (which is inlined) is incompatible if C becomes a good type (from a bad one) or viceversa. The code for a bad C involves allocating every C object on the heap, putting the pointer in the backing array, and dereferencing that pointer to get the C back; the code for a good C involves putting and retrieving C straight into/from the backing array. What happens if you have a library which creates a QListC assuming C to be bad, and then passes that QListC to your app, which assumes C to be good? This is what happens if you add a typeinfo after you forgot. Cheers, -- Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Software Engineer KDAB (UK) Ltd., a KDAB Group company | Tel: UK +44-1625-809908 KDAB - The Qt Experts smime.p7s Description: Firma crittografica S/MIME ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On 10 Jul 2015, at 14:58, Marc Mutz marc.m...@kdab.com wrote: On Friday 10 July 2015 13:37:40 Andreas Aardal Hanssen wrote: QListQImage. You just proved my point. sizeof(QImage) sizeof(void*). If even you as a long-time Troll don't understand what QList actually does, there's a problem. Don't you agree? At the very worst it would be a bug if QList created a list of pointers to new QImages. I can’t say when that bug was introduced though. Andreas ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
Can we please avoid turning this thread into why is QList bad? Sorry, you're the one who said it's bad; I'm quite happy with it. I'm asking if QListC becomes QListC* even if I declare it as QListC. martin From: development-bounces+martin.smith=theqtcompany@qt-project.org development-bounces+martin.smith=theqtcompany@qt-project.org on behalf of Marc Mutz marc.m...@kdab.com Sent: Friday, July 10, 2015 1:25 PM To: development@qt-project.org Subject: Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO On Friday 10 July 2015 11:54:16 Smith Martin wrote: Then I don't see why it is so inherently inefficient. The QList entry is allocated on the heap anyway. Doesn't QListC just allocate a bigger entry? And if I don't have the C object stored anywhere else, it has to be somewhere, so why not keep it in the QList entry? Can we please avoid turning this thread into why is QList bad? QList *is* bad. If you don't believe, read http://marcmutz.wordpress.com/effective-qt/ If you then _still_ don't believe, do your own benchmarks. Then you will believe. Thanks, Marc -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 13:35:34 Smith Martin wrote: 1) you put pressure on the memory allocator, by asking it to allocate on the heap each individual C object you put in the list. Every single allocation has a cost, plus the overhead you need for bookkeeping, plus the costs of getting your memory fragmented, etc.; This pressure on the memory allocator, what is that? My memory doesn't get fragmented, because I only build lists; I never destroy them. But I do have to put each C somewhere, and if I use QVectorC, doesn't it allocate a lot of them up front? I don't know how many i will need, but the number is usually 0, 1, or 2 -- the number of parameters in a function signature. 2) you make the compiler produce more inefficient code by introducing the layer of indirection; But it will process the list one time only, from beginning to end. 3) you kill caching, as potentially every single access will result in a cache miss (even in the common scenario of a simple forward iteration over your list); Why? All the entries in the list are created at the same time (well, during the parsing of the function signature) http://www.akkadia.org/drepper/cpumemory.pdf Sections 3 and 6. Please, read it. -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 11:54:16 Smith Martin wrote: Then I don't see why it is so inherently inefficient. The QList entry is allocated on the heap anyway. Doesn't QListC just allocate a bigger entry? And if I don't have the C object stored anywhere else, it has to be somewhere, so why not keep it in the QList entry? Can we please avoid turning this thread into why is QList bad? QList *is* bad. If you don't believe, read http://marcmutz.wordpress.com/effective-qt/ If you then _still_ don't believe, do your own benchmarks. Then you will believe. Thanks, Marc -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 12:24:35 Smith Martin wrote: Yes. But you told it to do that. You used a _list_. But I told it to make a lis of C, not a list of C*. You told it to make a list of C and it chose to implement it as an array of C* -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On 10 Jul 2015, at 14:24, Marc Mutz marc.m...@kdab.com wrote: That just goes to show how bad of an influence QList has. But it never hid its design, and, indeed, the default container in Qt 3 was QValueList, which actually *was* a doubly-linked list. Why Qt always nominates a list as the default container while the STL recommends std::vector is one of the mysteries to which only Trolls know the answer :) QList is the perfect default container for applications that value convenience and intuitiveness over best performance (a typical feature for most APIs in Qt), but it still outperforms arrays and linked lists for the most common operations for complex types, such as insertion and removal. The two most useful examples of QList usage in Qt applications are QListQWidget *, and QListQImage. To cater for the problem of custom complex types, all types in Qt were made shallow (the shallowness of containers and smartness of QList when used with shallow types go hand in hand, which is another typical feature for APIs in Qt). It should be no surprise to users that creating a QList of 100 QImages does *not* lead to 100 news or mallocs. Qt APIs also help programmers avoid making mistakes. QList has the undisputable advantage of allowing lists of dynamic sized elements. It’s a common programmer’s mistake to run into trouble with lists of dynamic elements when those elements are stored in arrays. Trouble leads to hacks leads to bugs. It doesn’t take too much imagination to understand how much of a pain you can run into, and what horrors could be applied to work around, creating a QVector of QVectors. The problem is a bit hard to grasp for some developers, and using QList you just don’t run into those problems. So, I dispute the general -1 of any use of QList. Use yours brains, and -1 where brain has not been applied. ;-) Andreas ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 12:45:51 Bo Thorsen wrote: I'm a big fan of naming things in ways that says what I'm looking at. To me this suggests that we should rename this one to Q_MEMCOPIABLE_TYPE or something else. The only established name that I know of for that concept is trivially_relocatable, carefully chosen by the EASTL guys to avoid connotation with C++11 move semantics. -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 12:20:55 Smith Martin wrote: I'm asking if QListC becomes QListC* even if I declare it as QListC. And I said in the initial mail: On Friday 10 July 2015 11:03:49 Marc Mutz wrote: I won't give you the whole story (google QList harmful for that) Which you clearly haven't done. Otherwise you'd have found the first hit https://marcmutz.wordpress.com/2010/07/29/sneak-preview-qlist-considered- harmful/ whose second sentence reads: The problem boils down to the fact that for a lot of types T, QListT is needlessly inefficient by allocating elements on the heap and storing pointers to them instead of storing the elements in-place, like e.g. QVector does. But yes, Peppe already provided the TL;DR: *grmbl* He's the nicer of the two of us :) -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
I think the impedance mismatch here is that you use list to mean the same thing as array or vector (in STL terms, not mathematically) while I only use it to mean linked list, in accordance with the STL. I actually just mean it's a list, and I don't care how it is implemented. I can see that it makes a difference how it is implemented, depending on how it is used, but if the use is simply: 1) build the list once; 2) process the list once, and every time I add an element to the list I have to create it somewhere, I don't see why creating it on the heap is inefficient. martin From: development-bounces+martin.smith=theqtcompany@qt-project.org development-bounces+martin.smith=theqtcompany@qt-project.org on behalf of Marc Mutz marc.m...@kdab.com Sent: Friday, July 10, 2015 2:24 PM To: development@qt-project.org Subject: Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO On Friday 10 July 2015 12:32:45 Smith Martin wrote: Maybe it's just me, but I'm still not understanding your explanation, and I don't think you are answering my question: I think the impedance mismatch here is that you use list to mean the same thing as array or vector (in STL terms, not mathematically) while I only use it to mean linked list, in accordance with the STL. That just goes to show how bad of an influence QList has. But it never hid its design, and, indeed, the default container in Qt 3 was QValueList, which actually *was* a doubly-linked list. Why Qt always nominates a list as the default container while the STL recommends std::vector is one of the mysteries to which only Trolls know the answer :) -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 14:19:54 Andreas Aardal Hanssen wrote: On 10 Jul 2015, at 14:58, Marc Mutz marc.m...@kdab.com wrote: On Friday 10 July 2015 13:37:40 Andreas Aardal Hanssen wrote: QListQImage. You just proved my point. sizeof(QImage) sizeof(void*). If even you as a long-time Troll don't understand what QList actually does, there's a problem. Don't you agree? At the very worst it would be a bug if QList created a list of pointers to new QImages. I can’t say when that bug was introduced though. It was introduced with the introduction of QList. :) Maybe QListQImage outperforms Q3ValueListQImage, but QImage is most certainly not in the QList sweet spot. BTW: Even in that sweet spot, when QListC uses almost exactly the same memory layout as a QVectorC (which, incidentally, invalidates your argument about QList being safer than QVector), it performs worse, probably because it has one more addition to perform to adjust for the prepend area. -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
Smith Martin schreef op 10-7-2015 om 13:27: I think the impedance mismatch here is that you use list to mean the same thing as array or vector (in STL terms, not mathematically) while I only use it to mean linked list, in accordance with the STL. I actually just mean it's a list, and I don't care how it is implemented. I can see that it makes a difference how it is implemented, depending on how it is used, but if the use is simply: 1) build the list once; 2) process the list once, and every time I add an element to the list I have to create it somewhere, I don't see why creating it on the heap is inefficient. Ah, but you _should_ care. It is a problem that programmers use (sometimes very complex) containers without having a clue of how they work, and what the performance implications of that complexity is. How are you going to choose the right container for your use case then? In your case, a vector _would_ be more efficient. The problem with the list setup is the double indirection and the memory fragmentation that comes with that. Please realize that memory is no longer fast. Memory was about as fast as CPU's 20 years ago. Now, memory is about 400x slower than your CPU. That means, that if your CPU needs data that is not in a cache (which is an effort to fix the slowness of your RAM), it needs to wait for about 400 cycles until it can proceed. That is slow. That means that you really want to be efficient in how you access your data. A data structure that puts all relevant data next to each other and doesn't waste any space (read: a vector or a simple array) makes it much easier for the computer to make sure that all memory needed for a loop is actually loaded into cache once it is needed. It becomes much harder to do that if you have an array of pointers to who-knows-where in memory where the actual data resides. That means that the CPU first has to follow the pointer to the array, and then every time the pointer to actual data. That means a memory access more, and a big chance of the data you really need not yet being in your cache, incurring the wait time to load it. That is not going to be fast. So yes, you _should_ case about what is going on. You are going to choose what the right container is by applying your knowledge on the performance trade-offs the classes you use. The situation is even worse with using datastructures like QMap and QSet by the way... André ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
-Original Message- From: giuseppe.dang...@kdab.com [mailto:giuseppe.dang...@kdab.com] Sent: Friday, 10 July 2015 12:22 PM To: Curtis Mitch; Smith Martin; development@qt-project.org Subject: Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO Il 10/07/2015 12:12, Curtis Mitch ha scritto: How is it binary incompatible? Because the code produced by QListC (which is inlined) is incompatible if C becomes a good type (from a bad one) or viceversa. The code for a bad C involves allocating every C object on the heap, putting the pointer in the backing array, and dereferencing that pointer to get the C back; the code for a good C involves putting and retrieving C straight into/from the backing array. What happens if you have a library which creates a QListC assuming C to be bad, and then passes that QListC to your app, which assumes C to be good? This is what happens if you add a typeinfo after you forgot. Cheers, -- Giuseppe D'Angelo | giuseppe.dang...@kdab.com | Software Engineer KDAB (UK) Ltd., a KDAB Group company | Tel: UK +44-1625-809908 KDAB - The Qt Experts Ah, I see. Thank you. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 12:48:26 Smith Martin wrote: Calling malloc/free is not a cheap operation, when you profile most Qt applications, you'll find these functions nearly always in the top 10, often even the top 5, of functions where most time is spent. But, Marc made QVector for QList swap in qdoc several weeks ago for the same kind of thing, but I couldn't see any difference in the run time for qdoc. Try perf stat, but even then maybe the impact is not high. But if we'd do it everywhere, the overall performance goes up. Bad performance is often a death by a thousand cuts. Huh?! I really don't follow your reasoning to both of these points here. I suggest you go watch https://channel9.msdn.com/Events/Build/2013/4-329 and similar talks out there to educate yourself? Whenever you access/traverse a list of pointers, you'll incur cache misses, except in the rare case where you are lucky and the pointers point to contiguous memory locations. And even then, you'd have an indirection that Peppe mentions. In this case, qdoc creates a QListParsedParameter as it parses a function signature. All the calls to malloc occur one right after another, and then the list is complete. There hasn't been any intervening heap work, so all these ParsedParameters will be sequential and will be allocated right after the QList itself. This is not necessarily true. Malloc implementations are free to return random pointers. Maybe you are lucky and the stuff is sequential, most probably not. And you'd still have the indirection. Also, maybe it's not a big deal for this one specific use case, but in the general usecase you have other allocations inbetween the list appends, thereby rendering your above reasoning invalid. But more importantly, this was just a temporary mechanism to correct the parsing of friend function declarations, which broke in qdoc, because good old Marc started making friend functions be friend inline functions with the body defined in the friend inline declaration inside the class. I think Marc's goal is to be the first ever software engineer to use every possible combination of C++ features. Still, this one is certainly legal, but qdoc couldn't handle it (qdoc still doesn't have a real C++ parser *** see below***), so I rewrote the function that parses function declarations so that it waits to create the list of parameters until it knows whether it should create the function node in the first place. This QListParsedParameter was meant to be a temporary solution, because that list gets copied into a QListParameter if qdoc finally decides to create the function node. That, of course, is inefficient, but I generally don'[[t try to optimize until I have a working model. You are talking about a specific code path in qdoc, while this thread is about QList in general. This does not help at all. Bye 2) you make the compiler produce more inefficient code by introducing the layer of indirection; But it will process the list one time only, from beginning to end. 3) you kill caching, as potentially every single access will result in a cache miss (even in the common scenario of a simple forward iteration over your list); Why? All the entries in the list are created at the same time (well, during the parsing of the function signature) Huh?! I really don't follow your reasoning to both of these points here. I suggest you go watch https://channel9.msdn.com/Events/Build/2013/4-329 and similar talks out there to educate yourself? Whenever you access/traverse a list of pointers, you'll incur cache misses, except in the rare case where you are lucky and the pointers point to contiguous memory locations. And even then, you'd have an indirection that Peppe mentions. Bye -- Milian Wolff | milian.wo...@kdab.com | Software Engineer KDAB (Deutschland) GmbHCo KG, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt Experts -- Milian Wolff | milian.wo...@kdab.com | Software Engineer KDAB (Deutschland) GmbHCo KG, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On 2015-07-10 06:37, Smith Martin wrote: ok, thanks, G. That explains it. And then there is the further point that it is unwise to use Q_DECLARE_TYPEINFO to declare C to be good in this case. What was the reasoning there? It really depends. The issue with Q_DECLARE_TYPEINFO is that adding/changing it breaks binary compatibility, so whether fiddling with Q_DECLARE_TYPEINFO is unwise or not really comes down to whether breaking binary compatibility for your project is unwise or not. Is your project an application with no exported symbols? Is it part of an application where you tightly control the end product and have no external users linking to your libraries? Then probably there is no issue. OTOH if your project is a widely used library, then breaking BC is almost certainly going to... irritate (to put it mildly) your users. In terms of adding/changing Q_DECLARE_TYPEINFO for classes *within Qt*, then you're definitely in unwise territory. HTH, -- Matthew ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Relevant industrial buses
-Original Message- From: development-bounces+alexander.blasche=theqtcompany.com@qt- project.org [mailto:development- bounces+alexander.blasche=theqtcompany@qt-project.org] On Behalf Of Tom Isaacson Sent: Friday, 10 July 2015 11:13 To: development@qt-project.org Subject: Re: [Development] Relevant industrial buses Not sure this counts as industrial but NMEA2000 for marine would be good. http://doc.qt.io/qt-5/qnmeapositioninfosource.html Even if this is not sufficient, I would not put it into the category of industrial buses. -- Alex ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 14:48:26 Smith Martin wrote: but I generally don'[[t try to optimize until I have a working model. This is not premature optimisation. This is avoiding premature pessimisation (google it!). QVector has exactly the same API these days as QList (except that QVector requires a default ctor and QList doesn't; if something else is missing, shout), so code using QVector is equally readable as code using QList. If that is the case, the more efficient alternative is what should naturally flow out of your fingers. And another thing: By using a QVector, not only do you avoid an additional heap allocation per element inserted, you also avoid readers of your code guessing whether you're using the QList for its list features (like the QListPage in QToolBoxPrivate - go, read it, it's subtle!) or just as a container. -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] Relevant industrial buses
There are a lot more commands in NMEA2000 than just position. As far as I can see QNmeaPositionInfoSource just allows you to enter data you've already parsed, there's no bus support there. Tom Isaacson -Original Message- From: Blasche Alexander [mailto:alexander.blas...@theqtcompany.com] Sent: Saturday, 11 July 2015 1:34 a.m. To: Tom Isaacson; development@qt-project.org Subject: RE: [Development] Relevant industrial buses -Original Message- From: development-bounces+alexander.blasche=theqtcompany.com@qt- project.org [mailto:development- bounces+alexander.blasche=theqtcompany@qt-project.org] On Behalf Of Tom Isaacson Sent: Friday, 10 July 2015 11:13 To: development@qt-project.org Subject: Re: [Development] Relevant industrial buses Not sure this counts as industrial but NMEA2000 for marine would be good. http://doc.qt.io/qt-5/qnmeapositioninfosource.html Even if this is not sufficient, I would not put it into the category of industrial buses. -- Alex ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 17:00:22 Smith Martin wrote: This is not premature optimisation. It is premature because the entire structure will be removed. I'm sorry, my static checking code doesn't yet read my mind, much less other people's. :) -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 13:26:47 Giuseppe D'Angelo wrote: 3) you kill caching, as potentially every single access will result in a cache miss (even in the common scenario of a simple forward iteration over your list); Even if it isn't a cache miss now, it might become later as there will be more cachelines touched by the iteration. That means some older cachelines were evicted from L1, which means you'lll most likely get a cache miss elsewhere that could've been a cache hit. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 15:22:13 Milian Wolff wrote: In this case, qdoc creates a QListParsedParameter as it parses a function signature. All the calls to malloc occur one right after another, and then the list is complete. There hasn't been any intervening heap work, so all these ParsedParameters will be sequential and will be allocated right after the QList itself. This is not necessarily true. Malloc implementations are free to return random pointers. Maybe you are lucky and the stuff is sequential, most probably not. Don't forget the overhead of each allocation. If you're allocating 16 bytes, for example, then Glibc's malloc will return pointers that are offset 32 bytes from each other. That means you get two elements per cacheline, as opposed to four. A QList of 8 elements of 16 bytes each occupies: 8 * sizeof(void*) + sizeof(QListData) + overhead = 8 * 8 + 16 + 16 8 * (sizeof(element) + overhead) = 8 * 32 - 352 bytes (6 to 19 cachelines)[*] A QVector of the same 8 elements is 8 * sizeof(element) + sizeof(QArrayData) + overhead = 8 * 16 + 16 + 16 --- 160 bytes (75% chance of 3 cachelines, 25% of 4 cachelines) The best of QList is still 50% worse than the worst of QVector. [*] How I came up with these numbers: the array block is 96 bytes long, so in the best case it spans 2 cachelines. In the best case, each of the 32-byte blocks for the elements is contiguous, so we have one single 352-byte block. That's 6 cachelines. in the worst case, the array block spans 3 cachelines due to alignment and each of the 8 elements is not only in a different cacheline, each also spans 2 cachelines without sharing them with other elements or the array header. Now, to be fair, iteration does not touch the region for malloc's overhead, so iterating would touch at most 2 + 8 = 10 cachelines in the worst case, but still 6 on the best. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
You are talking about a specific code path in qdoc, while this thread is about QList in general. This does not help at all. But that was the point. The argument is that QListC should not be used at all, but in the specific code path where it is being changed, it was ok. And apparently QVector has the same API as QList now, so why don't we deprecate QList. Let it always create a QVector. martin From: milian on behalf of Milian Wolff milian.wo...@kdab.com Sent: Friday, July 10, 2015 3:22 PM To: Smith Martin Cc: development@qt-project.org; Giuseppe D'Angelo Subject: Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO On Friday 10 July 2015 12:48:26 Smith Martin wrote: Calling malloc/free is not a cheap operation, when you profile most Qt applications, you'll find these functions nearly always in the top 10, often even the top 5, of functions where most time is spent. But, Marc made QVector for QList swap in qdoc several weeks ago for the same kind of thing, but I couldn't see any difference in the run time for qdoc. Try perf stat, but even then maybe the impact is not high. But if we'd do it everywhere, the overall performance goes up. Bad performance is often a death by a thousand cuts. Huh?! I really don't follow your reasoning to both of these points here. I suggest you go watch https://channel9.msdn.com/Events/Build/2013/4-329 and similar talks out there to educate yourself? Whenever you access/traverse a list of pointers, you'll incur cache misses, except in the rare case where you are lucky and the pointers point to contiguous memory locations. And even then, you'd have an indirection that Peppe mentions. In this case, qdoc creates a QListParsedParameter as it parses a function signature. All the calls to malloc occur one right after another, and then the list is complete. There hasn't been any intervening heap work, so all these ParsedParameters will be sequential and will be allocated right after the QList itself. This is not necessarily true. Malloc implementations are free to return random pointers. Maybe you are lucky and the stuff is sequential, most probably not. And you'd still have the indirection. Also, maybe it's not a big deal for this one specific use case, but in the general usecase you have other allocations inbetween the list appends, thereby rendering your above reasoning invalid. But more importantly, this was just a temporary mechanism to correct the parsing of friend function declarations, which broke in qdoc, because good old Marc started making friend functions be friend inline functions with the body defined in the friend inline declaration inside the class. I think Marc's goal is to be the first ever software engineer to use every possible combination of C++ features. Still, this one is certainly legal, but qdoc couldn't handle it (qdoc still doesn't have a real C++ parser *** see below***), so I rewrote the function that parses function declarations so that it waits to create the list of parameters until it knows whether it should create the function node in the first place. This QListParsedParameter was meant to be a temporary solution, because that list gets copied into a QListParameter if qdoc finally decides to create the function node. That, of course, is inefficient, but I generally don'[[t try to optimize until I have a working model. You are talking about a specific code path in qdoc, while this thread is about QList in general. This does not help at all. Bye 2) you make the compiler produce more inefficient code by introducing the layer of indirection; But it will process the list one time only, from beginning to end. 3) you kill caching, as potentially every single access will result in a cache miss (even in the common scenario of a simple forward iteration over your list); Why? All the entries in the list are created at the same time (well, during the parsing of the function signature) Huh?! I really don't follow your reasoning to both of these points here. I suggest you go watch https://channel9.msdn.com/Events/Build/2013/4-329 and similar talks out there to educate yourself? Whenever you access/traverse a list of pointers, you'll incur cache misses, except in the rare case where you are lucky and the pointers point to contiguous memory locations. And even then, you'd have an indirection that Peppe mentions. Bye -- Milian Wolff | milian.wo...@kdab.com | Software Engineer KDAB (Deutschland) GmbHCo KG, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt Experts -- Milian Wolff | milian.wo...@kdab.com | Software Engineer KDAB (Deutschland) GmbHCo KG, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 17:01:04 Smith Martin wrote: And apparently QVector has the same API as QList now, so why don't we deprecate QList. Let it always create a QVector. Because it's used in our API and we cannot deprecate such large chunks of it. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
I’m surprised nobody has mentioned that, because of the private data nature of most Qt classes, they are a) already allocating the private data object on the heap all the time and incurring all that overhead and memory allocation badness, and b) are essentially a pointer to an object already — thus QList adds another redundant pointer on top of that, and hence QVector functions much like a QList might have worked for objects with significant amounts of their own member data. Seems like a compelling argument in favor of QVector to me.. - Randy On Jul 10, 2015, at 1:33 PM, Thiago Macieira thiago.macie...@intel.com wrote: On Friday 10 July 2015 09:50:40 Thiago Macieira wrote: A QList of 8 elements of 16 bytes each occupies: 8 * sizeof(void*) + sizeof(QListData) + overhead = 8 * 8 + 16 + 16 8 * (sizeof(element) + overhead) = 8 * 32 - 352 bytes (6 to 19 cachelines)[*] A QVector of the same 8 elements is 8 * sizeof(element) + sizeof(QArrayData) + overhead = 8 * 16 + 16 + 16 --- 160 bytes (75% chance of 3 cachelines, 25% of 4 cachelines) The best of QList is still 50% worse than the worst of QVector. BTW, if you benchmark the same QLinkedList: 16 + overhead = 32 bytes[1 or 2 cachelines] 8 * (sizeof(QLinkedListNodeT) + overhead) = 8 * (2 * sizeof(void*) + sizeof(T) + overhead) = 8 * (16 + 16 + 16) = 384 [8 to 16 cachelines) - 416 bytes [7 to 18 cachelines] Comparing on cachelines, QLinkedList's best is worse than QList's best, but its worst is best than QList's worst. Comparing on memory allocation, QLinkedList is 18% worse than QList and 160% worse than QVector. QList is a better list than QLinkedList. It's not a vector. You really want to use a vector for everything except if: - you NEED O(1) insertion or deletion anywhere→ use QLinkedList - you want stable element pointers even if the container is modified - copying your type is expensive and your container gets modified a lot The first two are extremely rare. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
On Friday 10 July 2015 09:50:40 Thiago Macieira wrote: A QList of 8 elements of 16 bytes each occupies: 8 * sizeof(void*) + sizeof(QListData) + overhead = 8 * 8 + 16 + 16 8 * (sizeof(element) + overhead) = 8 * 32 - 352 bytes (6 to 19 cachelines)[*] A QVector of the same 8 elements is 8 * sizeof(element) + sizeof(QArrayData) + overhead = 8 * 16 + 16 + 16 --- 160 bytes (75% chance of 3 cachelines, 25% of 4 cachelines) The best of QList is still 50% worse than the worst of QVector. BTW, if you benchmark the same QLinkedList: 16 + overhead = 32 bytes[1 or 2 cachelines] 8 * (sizeof(QLinkedListNodeT) + overhead) = 8 * (2 * sizeof(void*) + sizeof(T) + overhead) = 8 * (16 + 16 + 16) = 384 [8 to 16 cachelines) - 416 bytes [7 to 18 cachelines] Comparing on cachelines, QLinkedList's best is worse than QList's best, but its worst is best than QList's worst. Comparing on memory allocation, QLinkedList is 18% worse than QList and 160% worse than QVector. QList is a better list than QLinkedList. It's not a vector. You really want to use a vector for everything except if: - you NEED O(1) insertion or deletion anywhere → use QLinkedList - you want stable element pointers even if the container is modified - copying your type is expensive and your container gets modified a lot The first two are extremely rare. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO
This is not premature optimisation. It is premature because the entire structure will be removed. From: development-bounces+martin.smith=theqtcompany@qt-project.org development-bounces+martin.smith=theqtcompany@qt-project.org on behalf of Marc Mutz marc.m...@kdab.com Sent: Friday, July 10, 2015 5:32 PM To: development@qt-project.org Subject: Re: [Development] HEADS UP: Don't use QList, use Q_DECLARE_TYPEINFO On Friday 10 July 2015 14:48:26 Smith Martin wrote: but I generally don'[[t try to optimize until I have a working model. This is not premature optimisation. This is avoiding premature pessimisation (google it!). QVector has exactly the same API these days as QList (except that QVector requires a default ctor and QList doesn't; if something else is missing, shout), so code using QVector is equally readable as code using QList. If that is the case, the more efficient alternative is what should naturally flow out of your fingers. And another thing: By using a QVector, not only do you avoid an additional heap allocation per element inserted, you also avoid readers of your code guessing whether you're using the QList for its list features (like the QListPage in QToolBoxPrivate - go, read it, it's subtle!) or just as a container. -- Marc Mutz marc.m...@kdab.com | Senior Software Engineer KDAB (Deutschland) GmbH Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
