Re: [Development] RFD: plugins vs QStringLiterals
On Thursday 05 November 2015 11:44:33 Thiago Macieira wrote: > 3) Never unload any plugins, possibly also compiling our own libraries and > plugins with -z nodelete. Solves most of the problems, including the C++ > vtable case. Patch for QPluginLoader and QFactoryLoader: https://codereview.qt-project.org/140750 Do we want to add -z nodelete? -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Thursday 12 November 2015 09:58:15 Thiago Macieira wrote:
> Ok, thanks. I'll disable plugin unloading in QFactoryLoader in 5.6. We can
> look into merging it with QPluginLoader later.
Actually, I've just seen the difference.
~QPluginLoader:
if (d)
d->release();
~QFactoryLoaderPrivate:
for (int i = 0; i < libraryList.count(); ++i) {
QLibraryPrivate *library = libraryList.at(i);
library->unload();
library->release();
}
A QPluginLoader loads one plugin. A QFactoryLoader loads all plugins matching
a given filter.
--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Thursday 12 November 2015 11:54:14 Knoll Lars wrote: > On 12/11/15 12:28, "Development on behalf of Paul Olav Tvete" wrote: > >On Friday 6. November 2015 10.10.52 Thiago Macieira wrote: > >> But before I go and modify QFactoryLoader... what is that class for? Can > >> anyone find out from the old history? It traces its existence back to > >> "Long > >> live Qt 4.5". > > > >Added by Matthias in commit 89df363e4a795d67342d04e478af592618e16363 at Thu > >May 6 13:28:09 2004 > > > >"added the new plugin stuff. Not yet used, not yet compiled." > > > >Hope that helps. :p > > > >But seriously, there does not seem to be a lot more information in the old > >history than what's in the code today. > > Most of that code is 8 years old by now, and some of the reasons for having > these things separated have most probably disappeared by now. Let's simply > look at things from todays perspective :) Ok, thanks. I'll disable plugin unloading in QFactoryLoader in 5.6. We can look into merging it with QPluginLoader later. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On 12/11/15 12:28, "Development on behalf of Paul Olav Tvete" wrote: >On Friday 6. November 2015 10.10.52 Thiago Macieira wrote: >> But before I go and modify QFactoryLoader... what is that class for? Can >> anyone find out from the old history? It traces its existence back to "Long >> live Qt 4.5". > >Added by Matthias in commit 89df363e4a795d67342d04e478af592618e16363 at Thu >May 6 13:28:09 2004 > >"added the new plugin stuff. Not yet used, not yet compiled." > >Hope that helps. :p > >But seriously, there does not seem to be a lot more information in the old >history than what's in the code today. Most of that code is 8 years old by now, and some of the reasons for having these things separated have most probably disappeared by now. Let's simply look at things from todays perspective :) Cheers, Lars ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Friday 6. November 2015 10.10.52 Thiago Macieira wrote: > But before I go and modify QFactoryLoader... what is that class for? Can > anyone find out from the old history? It traces its existence back to "Long > live Qt 4.5". Added by Matthias in commit 89df363e4a795d67342d04e478af592618e16363 at Thu May 6 13:28:09 2004 "added the new plugin stuff. Not yet used, not yet compiled." Hope that helps. :p But seriously, there does not seem to be a lot more information in the old history than what's in the code today. - Paul ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Friday 06 November 2015 09:47:51 Koehne Kai wrote: > The easiest way to achieve this is to ban QStringLiteral and > QByteArrayLiteral inside Qt's own code. Somehow I doubt that this would > have noticeable impact anyway, at least if the replacement tries to cache > strings once constructed, instead of mindlessly calling > QString::fromLatin1() a million times in a row. Doesn't help with the other types of static data we might keep pointers to (meta objects, meta types, vtables, etc.) -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Friday 06 November 2015 16:45:33 Knoll Lars wrote: > To me this points strongly to opt for never unloading plugins. We can of > course offer a way to force the unloading if the user does it's own stuff > and knows what he's doing. But Qt should probably never unload any plugins > it loads, as we do not control those and don't know whether they are safe > to unload. > > With that we're ok for the plugins we load. And if someone else insists on > unloading something it's his problem. I'm leaning towards that too. But before I go and modify QFactoryLoader... what is that class for? Can anyone find out from the old history? It traces its existence back to "Long live Qt 4.5". -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Friday 06 November 2015 08:52:29 Welbourne Edward wrote: > > SEP? > > Sorry, reference to hitch-hiker's guide to the galaxy: > Someone Else's Problem. https://en.wikipedia.org/wiki/SEP_field "... is a psychological effect where people choose to dissociate themselves from an issue that may be in critical need of recognition. Such issues may be of large concern to the population as a whole but can easily be a choice of ignorance by an individual" -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
> On 06 Nov 2015, at 17:45, Knoll Lars wrote: > > On 06/11/15 17:38, "Development on behalf of Giuseppe D'Angelo" > wrote: Hi, > To me this points strongly to opt for never unloading plugins. We can of > course offer a way to force the unloading if the user does it's own stuff and > knows what he's doing. But Qt should probably never unload any plugins it > loads, as we do not control those and don't know whether they are safe to > unload. FYI the musl libc has already made dlclose a no-op[1] out of concerns for the robustness of applications. If you want to add a force option we would probably need a way to flush "caches" in all of qt as well? kind regards holger [1] http://wiki.musl-libc.org/wiki/Functional_differences_from_glibc#Unloading_libraries ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On 06/11/15 17:38, "Development on behalf of Giuseppe D'Angelo" wrote: >On Fri, Nov 6, 2015 at 5:20 PM, Kevin Kofler wrote: >> They just need to deep-copy the >> strings they return in the plugin interface with: >> QString(orig.constData(), orig.size()) >> if they can be QStringLiterals and are susceptible of outliving the plugin. > >Given the problem is a specific case of dangling pointers to resources >(in the plugin) about to get destroyed... in the general case, are we >playing it "nice" and offering a way for the plugin to clean up after >itself, by removing any resouce it may have given out to other >libraries which will outlive it? Unfortunately in many ways that's pretty hard to do. QStringLiteral is only one part of the problem as others have noted. We have many other things that could potentially still be referenced when the plugin gets unloaded: * virtual tables * doc generated data * metatype registrations * string and byte array literals * any pointers to static data that leave the scope of the plugin To me this points strongly to opt for never unloading plugins. We can of course offer a way to force the unloading if the user does it's own stuff and knows what he's doing. But Qt should probably never unload any plugins it loads, as we do not control those and don't know whether they are safe to unload. With that we're ok for the plugins we load. And if someone else insists on unloading something it's his problem. Cheers, Lars ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Fri, Nov 6, 2015 at 5:20 PM, Kevin Kofler wrote: > They just need to deep-copy the > strings they return in the plugin interface with: > QString(orig.constData(), orig.size()) > if they can be QStringLiterals and are susceptible of outliving the plugin. Given the problem is a specific case of dangling pointers to resources (in the plugin) about to get destroyed... in the general case, are we playing it "nice" and offering a way for the plugin to clean up after itself, by removing any resouce it may have given out to other libraries which will outlive it? -- Giuseppe D'Angelo ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
Thiago Macieira wrote: > 1) Declare it SEP and only apply workarounds for the places where > QStringLiteral is used in our plugins, suggesting that people do the same > in their plugins. > > Problems: libraries loaded by plugins, fragile. +1. Caveat emptor. Deep copying destroys the point of QStringLiteral, never unloading plugins is a crude hack. There is only really a problem if the strings (or shallow copies thereof) are passed around in the program after unloading the plugin. So this is very much in the control of the plugin author. They just need to deep-copy the strings they return in the plugin interface with: QString(orig.constData(), orig.size()) if they can be QStringLiterals and are susceptible of outliving the plugin. Kevin Kofler ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
> -Original Message- > From: Development [mailto:development-boun...@qt-project.org] On Behalf > [...] > The problem with SEP is that it only works in so far as there *is* something > the > author of client code can do about the situation. They may not have enough > choice to control this. > > Qt is a framework. > It gets used in many ways. > This makes it hard to say what options clients have - or don't have. > In the end, SEP means telling some class of potential clients of Qt that Qt > has > made a design decision that means they have to use something other than Qt. Well ... we'd 'just' have to make sure we don't "leak" QString's created by QStringLiteral across dll/plugin boundaries. Then, it's indeed the choice of the user's (say KDE) whether they want to use it in their libraries and applications, or not. The easiest way to achieve this is to ban QStringLiteral and QByteArrayLiteral inside Qt's own code. Somehow I doubt that this would have noticeable impact anyway, at least if the replacement tries to cache strings once constructed, instead of mindlessly calling QString::fromLatin1() a million times in a row. Regards Kai ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
> SEP? Sorry, reference to hitch-hiker's guide to the galaxy: Someone Else's Problem. Eddy. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
> -Original Message- > From: Development [mailto:development-boun...@qt-project.org] On Behalf Of > Thiago Macieira > Sent: Thursday, 5 November 2015 5:45 PM > To: development@qt-project.org > Subject: [Development] RFD: plugins vs QStringLiterals > [snip] > > 1) Declare it SEP and only apply workarounds for the places where > QStringLiteral is used in our plugins, suggesting that people do the same > in their plugins. SEP? > Problems: libraries loaded by plugins, fragile. > > 2) Deep-copy the QStringLiterals > a) with atom/quark > b) without > > Problem: performance impact, complexity of the atom/quark solution. > > 3) Never unload any plugins, possibly also compiling our own libraries and > plugins with -z nodelete. Solves most of the problems, including the C++ > vtable case. > > Problems: doesn't catch bypassing of QLibrary (dlopen/LoadLibrary calls), > prevents upgrading of plugins without restarting the host application. > > -- > Thiago Macieira - thiago.macieira (AT) intel.com > Software Architect - Intel Open Source Technology Center > > ___ > Development mailing list > Development@qt-project.org > http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
IIUC, the problem Thiago is discussing is a crash on exit(), or after main() has returned. This limits the scope of the problem (we don't make applications crashy while running), but only somewhat. There may be things in the application loading a plugin that rely on clean shutdown, including destructors of the application's globals, which may get skipped if a (long ago unloaded) plugin's globals cause a crash before the application's destructors get their turn. Thiago asked: >> What do we do then? >> >> 1) Declare it SEP and only apply workarounds for the places where >> QStringLiteral is used in our plugins, suggesting that people do the same in >> their plugins. On 05 November 2015 18:02, Sergio Martins replied: > +1 for SEP > Document the circumstances where QSL might crash so the user can > decide for himself what he values most. > > Customers will usually want stability, so they probably won't use QSL, > but for many projects, like KDE, it's important to remove all those > heap allocations and we tolerate a crash now and then. These crashes > are very rare, and when they happen they are easy to diagnose and > fix. Until now I've only seen the dbus crash you fixed and the QRegExp > crash which I debugged in 15 minutes yesterday. The problem with SEP is that it only works in so far as there *is* something the author of client code can do about the situation. They may not have enough choice to control this. Qt is a framework. It gets used in many ways. This makes it hard to say what options clients have - or don't have. In the end, SEP means telling some class of potential clients of Qt that Qt has made a design decision that means they have to use something other than Qt. The client code may be a plugin for an application that the plugin authors can't influence, that unloads whether the plugin authors like it or not. It is not unheard of for an application to include a "plugin blacklist" of plugins known to cause problems; if the application developers dislike crash-on-exit, they may then blacklist the plugin. Authors of plugins in such contexts can't sensibly use Qt, unless we can ensure that "unloading" doesn't actually unload. Of course, such an application is likely using naked dlopen/dlcose, not our helpers. The application may be long-running, loading and unloading plugins on demand: it may end up loading new versions of a plugin it has previously unloaded. (There are at least two APIs for web server plugins; fortunately, these are usually loaded in separate processes; but a lean mean new web-server supporting those APIs to ease migration to it as replacement might handle things differently.) As Thiago mentioned, any don't-really-unload solution breaks this. We likely have to live with some element of SEP: the other solutions only mitigate the problem or limit its scope, so we have to document the remaining cases where Qt won't be a good choice for plugin authors. We should at least try to make that a problem for as few potential clients as possible. I played around with this problem (in my ignorance) last month. It seems *some* globals in plugins do get destructed when the library gets unloaded. This surely is what should happen. However, others did not; I failed to work out why (partly due to getting distracted by a red herring). If we can work out what causes the difference and work out how to get globals to go away on dlclose(), we may have a better solution. However, I failed to see a way to do that. Eddy. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
And every small, movable object kept in QVariant is affected. QMetaType also
keeps pointers to functions defined in plugins if such register a type. I
believe that we had that discussion around Qt4->Qt5 migration. Unloading
plugins is not safe, sometimes it works but still it is very tricky. I agree
with Simon and I'm in favor of 3.
Cheers,
Jędrek
On Thursday 05 of November 2015 16:57:49 Hausmann Simon wrote:
> And moc data is affected in a similar way. I continue to be in favor of (3)
>
> Simon
>
> Original Message
> From: Thiago Macieira
> Sent: Friday, November 6, 2015 00:44
> To: development@qt-project.org
> Subject: [Development] RFD: plugins vs QStringLiterals
>
>
> Proposal: force QStringLiteral uses to always address a heap-allocated
> QString, instead of pointing to the static data. Possibly, this should be an
> interned atom/quark à la GQuark, so two passes on the same QStringLiteral
> would result in the same heap pointers.
>
> Alternative: force QPluginLoader / QFactoryLoader to fake unloading but not
> really unload. Maybe even QLibrary, if we find people use QLibrary to load
> Qt- based plugins instead of our own classes.
>
> Background:
>
> When we created QStringLiteral, we thought it was the best thing since
> sliced bread. We started using it everywhere and so did our users, after
> our recommendations. I even had a session in last years' Dev Days
> explaining when to use QStringLiteral and when not to. The objective of
> that session was to explain performance.
>
> The problem we're seeing is that it's quite easy to violate the precondition
> that the QStringLiteral never, ever disappears. This happens when plugins
> are involved. Any QStringLiteral in a plugin or in a library that gets
> loaded by that plugin is subject to disappearing before the end of the
> program.
>
> [Henceforth, "plugin" is "any dynamically loaded module, whether intended as
> a library or plugin, including all their dependencies that weren't loaded
> before"]
>
> I've said in the past that unloading plugins is a bad idea in C++. The case
> then was that it's easy to leave objects of a polymorphic class type whose
> virtual table is located in the unloaded plugin. This is not very different,
> since the QStringLiteral's data is also global data not expected to
> disappeaar.
>
> We've already worked around two cases of crash-at-exit caused by this, both
> coincidentally related to QtDBus's interface cache. But this can also happen
> for any other types of cache, like:
>
> QRegExp rx(QStringLiteral());
> QPixmapCache::insert(QStringLiteral("foo"), px);
>
> What do we do then?
>
> 1) Declare it SEP and only apply workarounds for the places where
> QStringLiteral is used in our plugins, suggesting that people do the same in
> their plugins.
>
> Problems: libraries loaded by plugins, fragile.
>
> 2) Deep-copy the QStringLiterals
> a) with atom/quark
> b) without
>
> Problem: performance impact, complexity of the atom/quark solution.
>
> 3) Never unload any plugins, possibly also compiling our own libraries and
> plugins with -z nodelete. Solves most of the problems, including the C++
> vtable case.
>
> Problems: doesn't catch bypassing of QLibrary (dlopen/LoadLibrary calls),
> prevents upgrading of plugins without restarting the host application.
>
> --
> Thiago Macieira - thiago.macieira (AT) intel.com
> Software Architect - Intel Open Source Technology Center
>
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development
> ___
> Development mailing list
> Development@qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Thu, 5 Nov 2015, Thiago Macieira wrote: It gets worse if you have KF5-based image format plugins, since kimg_eps.so links to QtPrintSupport. Fortunately, none of them today link to other KF5 libraries, but can we continue to count on that? Well, the KRA and OpenRaster image format plugins that come with Krita need Kf5::Archive to open the zip files containers. At least they no longer link to krita's core libraries, lcms2 and all the rest. Boudewijn ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Thursday 05 November 2015 15:50:17 Thiago Macieira wrote: > On Thursday 05 November 2015 18:23:37 Robin Burchell wrote: > > On Thu, Nov 5, 2015, at 06:13 PM, Oswald Buddenhagen wrote: > > > > Problem: performance impact, complexity of the atom/quark solution. > > > > > > to lessen the performance impact, we could have a #define to control the > > > construction behavior of QSL at compile time. i'm not quite sure whether > > > to make that opt-in or opt-out (safety vs. performance). > > > > Is the problem only apparent inside of plugins? Do we have a way to > > detect whether someone is building plugin code at build time? (CONFIG += > > plugin for qmake, I guess?) If so, couldn't the behaviour of > > QStringLiteral differ for code inside of plugins, to that outside of > > them? > > Plugins and any library loaded by plugins that hadn't been loaded before. Q: How often does this happen? A: It happens for virtually EVERY GUI Qt application because QtGui loads all imageformat plugins, which means QtSvg gets loaded whether you wanted it or not. It happens again on XCB because the platform plugin links to its private QtXcbQpa library, which in turn links to QtDBus. It gets worse if you have KF5-based image format plugins, since kimg_eps.so links to QtPrintSupport. Fortunately, none of them today link to other KF5 libraries, but can we continue to count on that? Note: the imageformats are loaded via QPluginLoader, so they aren't unloaded. The QPA platform plugin is loaded via QFactoryLoader, so it gets unloaded. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Thursday 05 November 2015 21:06:20 Marc Mutz wrote: > On Thursday 05 November 2015 17:44:33 Thiago Macieira wrote: > > Proposal: force QStringLiteral uses to always address a heap-allocated > > QString, instead of pointing to the static data. Possibly, this should be > > an interned atom/quark à la GQuark, so two passes on the same > > QStringLiteral would result in the same heap pointers. > > TL;DR: what's atom/quark in this context? The documentation for XAtom and GQuark probably explains more. Basically it's a string that becomes managed by the library and for which a pointer comparison guarantees string comparison. str1 == str2 ←→ &str1 == &str2 Obviously this is more useful for when you have pointers (GQuark) or you have some other form of identifier (XAtom). The benefit for us is that it would allocate memory only on the first use. After that, it always returns the same pre-allocated string data. We'd implement it using a QSet. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Thursday 05 November 2015 18:23:37 Robin Burchell wrote: > On Thu, Nov 5, 2015, at 06:13 PM, Oswald Buddenhagen wrote: > > > Problem: performance impact, complexity of the atom/quark solution. > > > > to lessen the performance impact, we could have a #define to control the > > construction behavior of QSL at compile time. i'm not quite sure whether > > to make that opt-in or opt-out (safety vs. performance). > > Is the problem only apparent inside of plugins? Do we have a way to > detect whether someone is building plugin code at build time? (CONFIG += > plugin for qmake, I guess?) If so, couldn't the behaviour of > QStringLiteral differ for code inside of plugins, to that outside of > them? Plugins and any library loaded by plugins that hadn't been loaded before. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Thursday 05 November 2015 18:13:36 Oswald Buddenhagen wrote: > On Thu, Nov 05, 2015 at 11:44:33AM -0500, Thiago Macieira wrote: > > Alternative: force QPluginLoader / QFactoryLoader to fake unloading but > > not > > really unload. > > you already did that quite a while ago. was it reverted? or never > integrated? And I did. QPluginLoader does not really unload its plugins. QFactoryLoader still does. Don't ask me why we have two and one doesn't use the other. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Thursday 05 November 2015 18:23:49 Milian Wolff wrote: > When you load a plugin, or compile a QRegExp, or use DBUS, or ... then the > cost of these operations easily dwarfs the string heap allocation. So this > is not that bad, imo. Making a nothrow operation throwing is bad. Avoiding heap allocations is not _just_ about performance -- Marc Mutz | Senior Software Engineer KDAB (Deutschland) GmbH & Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Thursday 05 November 2015 17:44:33 Thiago Macieira wrote: > Proposal: force QStringLiteral uses to always address a heap-allocated > QString, instead of pointing to the static data. Possibly, this should be > an interned atom/quark à la GQuark, so two passes on the same > QStringLiteral would result in the same heap pointers. TL;DR: what's atom/quark in this context? -- Marc Mutz | Senior Software Engineer KDAB (Deutschland) GmbH & Co.KG, a KDAB Group Company Tel: +49-30-521325470 KDAB - The Qt Experts ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Thu, Nov 05, 2015 at 06:23:37PM +0100, Robin Burchell wrote: > On Thu, Nov 5, 2015, at 06:13 PM, Oswald Buddenhagen wrote: > > > Problem: performance impact, complexity of the atom/quark solution. > > > > > to lessen the performance impact, we could have a #define to control the > > construction behavior of QSL at compile time. i'm not quite sure whether > > to make that opt-in or opt-out (safety vs. performance). > > Is the problem only apparent inside of plugins? Do we have a way to > detect whether someone is building plugin code at build time? (CONFIG += > plugin for qmake, I guess?) If so, couldn't the behaviour of > QStringLiteral differ for code inside of plugins, to that outside of > them? > you can also have plugin code which is perfectly safe. only a kinda perfect static analysis would give you a verdict about the safety of your code. On Thu, Nov 05, 2015 at 06:23:49PM +0100, Milian Wolff wrote: > On Donnerstag, 5. November 2015 18:13:36 CET Oswald Buddenhagen wrote: > > On Thu, Nov 05, 2015 at 11:44:33AM -0500, Thiago Macieira wrote: > > > 2) Deep-copy the QStringLiterals > > > > > > a) with atom/quark > > > > this sounds like it could be quite horrible (when people rely on cheap > > instantiation, e.g., for comparisons). > > When you load a plugin, or compile a QRegExp, or use DBUS, or ... then the > cost of these operations easily dwarfs the string heap allocation. So this is > not that bad, imo. > uhm, so? you can still do something "stupid" with QSL in a tight loop. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Donnerstag, 5. November 2015 18:13:36 CET Oswald Buddenhagen wrote: > On Thu, Nov 05, 2015 at 11:44:33AM -0500, Thiago Macieira wrote: > > Alternative: force QPluginLoader / QFactoryLoader to fake unloading but > > not > > really unload. > > you already did that quite a while ago. was it reverted? or never > integrated? > > > I've said in the past that unloading plugins is a bad idea in C++. The > > case > > then was that it's easy to leave objects of a polymorphic class type whose > > virtual table is located in the unloaded plugin. This is not very > > different, since the QStringLiteral's data is also global data not > > expected to disappeaar. > > the generic c++ case isn't that bad, because the user can reasonably > control the lifetime of objects and delete them before unloading > associated plugins. > > obviously, qt's implicit sharing takes away this control, which is > exactly the problem. > > > 2) Deep-copy the QStringLiterals > > > > a) with atom/quark > > this sounds like it could be quite horrible (when people rely on cheap > instantiation, e.g., for comparisons). When you load a plugin, or compile a QRegExp, or use DBUS, or ... then the cost of these operations easily dwarfs the string heap allocation. So this is not that bad, imo. -- Milian Wolff | milian.wo...@kdab.com | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company Tel: +49-30-521325470 KDAB - The Qt Experts smime.p7s Description: S/MIME cryptographic signature ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Thu, Nov 5, 2015, at 06:13 PM, Oswald Buddenhagen wrote: > > Problem: performance impact, complexity of the atom/quark solution. > > > to lessen the performance impact, we could have a #define to control the > construction behavior of QSL at compile time. i'm not quite sure whether > to make that opt-in or opt-out (safety vs. performance). Is the problem only apparent inside of plugins? Do we have a way to detect whether someone is building plugin code at build time? (CONFIG += plugin for qmake, I guess?) If so, couldn't the behaviour of QStringLiteral differ for code inside of plugins, to that outside of them? Robin ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Thu, Nov 05, 2015 at 11:44:33AM -0500, Thiago Macieira wrote: > Alternative: force QPluginLoader / QFactoryLoader to fake unloading but not > really unload. > you already did that quite a while ago. was it reverted? or never integrated? > I've said in the past that unloading plugins is a bad idea in C++. The case > then was that it's easy to leave objects of a polymorphic class type whose > virtual table is located in the unloaded plugin. This is not very different, > since the QStringLiteral's data is also global data not expected to > disappeaar. > the generic c++ case isn't that bad, because the user can reasonably control the lifetime of objects and delete them before unloading associated plugins. obviously, qt's implicit sharing takes away this control, which is exactly the problem. > 2) Deep-copy the QStringLiterals > a) with atom/quark > this sounds like it could be quite horrible (when people rely on cheap instantiation, e.g., for comparisons). > b) without > > Problem: performance impact, complexity of the atom/quark solution. > to lessen the performance impact, we could have a #define to control the construction behavior of QSL at compile time. i'm not quite sure whether to make that opt-in or opt-out (safety vs. performance). ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
On Thursday, November 05, 2015 11:44:33 AM Thiago Macieira wrote:
> Proposal: force QStringLiteral uses to always address a heap-allocated
> QString, instead of pointing to the static data. Possibly, this should be an
> interned atom/quark à la GQuark, so two passes on the same QStringLiteral
> would result in the same heap pointers.
>
> Alternative: force QPluginLoader / QFactoryLoader to fake unloading but not
> really unload. Maybe even QLibrary, if we find people use QLibrary to load
> Qt- based plugins instead of our own classes.
>
> Background:
>
> When we created QStringLiteral, we thought it was the best thing since
> sliced bread. We started using it everywhere and so did our users, after
> our recommendations. I even had a session in last years' Dev Days
> explaining when to use QStringLiteral and when not to. The objective of
> that session was to explain performance.
>
> The problem we're seeing is that it's quite easy to violate the precondition
> that the QStringLiteral never, ever disappears. This happens when plugins
> are involved. Any QStringLiteral in a plugin or in a library that gets
> loaded by that plugin is subject to disappearing before the end of the
> program.
>
> [Henceforth, "plugin" is "any dynamically loaded module, whether intended as
> a library or plugin, including all their dependencies that weren't loaded
> before"]
>
> I've said in the past that unloading plugins is a bad idea in C++. The case
> then was that it's easy to leave objects of a polymorphic class type whose
> virtual table is located in the unloaded plugin. This is not very different,
> since the QStringLiteral's data is also global data not expected to
> disappeaar.
>
> We've already worked around two cases of crash-at-exit caused by this, both
> coincidentally related to QtDBus's interface cache. But this can also happen
> for any other types of cache, like:
>
> QRegExp rx(QStringLiteral());
> QPixmapCache::insert(QStringLiteral("foo"), px);
>
> What do we do then?
>
> 1) Declare it SEP and only apply workarounds for the places where
> QStringLiteral is used in our plugins, suggesting that people do the same in
> their plugins.
+1 for SEP
Document the circumstances where QSL might crash so the user can decide for
himself what he values most.
Customers will usually want stability, so they probably won't use QSL, but for
many projects, like KDE, it's important to remove all those heap allocations
and we tolerate a crash now and then. These crashes are very rare, and when
they happen they are easy to diagnose and fix. Until now I've only seen the
dbus crash you fixed and the QRegExp crash which I debugged in 15 minutes
yesterday.
> Problems: libraries loaded by plugins, fragile.
>
> 2) Deep-copy the QStringLiterals
> a) with atom/quark
> b) without
>
> Problem: performance impact, complexity of the atom/quark solution.
-1
That defeats the purpose of QSL no ?
> 3) Never unload any plugins, possibly also compiling our own libraries and
> plugins with -z nodelete. Solves most of the problems, including the C++
> vtable case.
>
> Problems: doesn't catch bypassing of QLibrary (dlopen/LoadLibrary calls),
> prevents upgrading of plugins without restarting the host application.
+0
Regards,
--
Sérgio Martins | sergio.mart...@kdab.com | Software Engineer
Klarälvdalens Datakonsult AB, a KDAB Group company
Tel: Sweden (HQ) +46-563-540090, USA +1-866-777-KDAB(5322)
KDAB - The Qt Experts
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] RFD: plugins vs QStringLiterals
And moc data is affected in a similar way. I continue to be in favor of (3)
Simon
Original Message
From: Thiago Macieira
Sent: Friday, November 6, 2015 00:44
To: development@qt-project.org
Subject: [Development] RFD: plugins vs QStringLiterals
Proposal: force QStringLiteral uses to always address a heap-allocated
QString, instead of pointing to the static data. Possibly, this should be an
interned atom/quark à la GQuark, so two passes on the same QStringLiteral
would result in the same heap pointers.
Alternative: force QPluginLoader / QFactoryLoader to fake unloading but not
really unload. Maybe even QLibrary, if we find people use QLibrary to load Qt-
based plugins instead of our own classes.
Background:
When we created QStringLiteral, we thought it was the best thing since sliced
bread. We started using it everywhere and so did our users, after our
recommendations. I even had a session in last years' Dev Days explaining when
to use QStringLiteral and when not to. The objective of that session was to
explain performance.
The problem we're seeing is that it's quite easy to violate the precondition
that the QStringLiteral never, ever disappears. This happens when plugins are
involved. Any QStringLiteral in a plugin or in a library that gets loaded by
that plugin is subject to disappearing before the end of the program.
[Henceforth, "plugin" is "any dynamically loaded module, whether intended as a
library or plugin, including all their dependencies that weren't loaded
before"]
I've said in the past that unloading plugins is a bad idea in C++. The case
then was that it's easy to leave objects of a polymorphic class type whose
virtual table is located in the unloaded plugin. This is not very different,
since the QStringLiteral's data is also global data not expected to
disappeaar.
We've already worked around two cases of crash-at-exit caused by this, both
coincidentally related to QtDBus's interface cache. But this can also happen
for any other types of cache, like:
QRegExp rx(QStringLiteral());
QPixmapCache::insert(QStringLiteral("foo"), px);
What do we do then?
1) Declare it SEP and only apply workarounds for the places where
QStringLiteral is used in our plugins, suggesting that people do the same in
their plugins.
Problems: libraries loaded by plugins, fragile.
2) Deep-copy the QStringLiterals
a) with atom/quark
b) without
Problem: performance impact, complexity of the atom/quark solution.
3) Never unload any plugins, possibly also compiling our own libraries and
plugins with -z nodelete. Solves most of the problems, including the C++
vtable case.
Problems: doesn't catch bypassing of QLibrary (dlopen/LoadLibrary calls),
prevents upgrading of plugins without restarting the host application.
--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
[Development] RFD: plugins vs QStringLiterals
Proposal: force QStringLiteral uses to always address a heap-allocated
QString, instead of pointing to the static data. Possibly, this should be an
interned atom/quark à la GQuark, so two passes on the same QStringLiteral
would result in the same heap pointers.
Alternative: force QPluginLoader / QFactoryLoader to fake unloading but not
really unload. Maybe even QLibrary, if we find people use QLibrary to load Qt-
based plugins instead of our own classes.
Background:
When we created QStringLiteral, we thought it was the best thing since sliced
bread. We started using it everywhere and so did our users, after our
recommendations. I even had a session in last years' Dev Days explaining when
to use QStringLiteral and when not to. The objective of that session was to
explain performance.
The problem we're seeing is that it's quite easy to violate the precondition
that the QStringLiteral never, ever disappears. This happens when plugins are
involved. Any QStringLiteral in a plugin or in a library that gets loaded by
that plugin is subject to disappearing before the end of the program.
[Henceforth, "plugin" is "any dynamically loaded module, whether intended as a
library or plugin, including all their dependencies that weren't loaded
before"]
I've said in the past that unloading plugins is a bad idea in C++. The case
then was that it's easy to leave objects of a polymorphic class type whose
virtual table is located in the unloaded plugin. This is not very different,
since the QStringLiteral's data is also global data not expected to
disappeaar.
We've already worked around two cases of crash-at-exit caused by this, both
coincidentally related to QtDBus's interface cache. But this can also happen
for any other types of cache, like:
QRegExp rx(QStringLiteral());
QPixmapCache::insert(QStringLiteral("foo"), px);
What do we do then?
1) Declare it SEP and only apply workarounds for the places where
QStringLiteral is used in our plugins, suggesting that people do the same in
their plugins.
Problems: libraries loaded by plugins, fragile.
2) Deep-copy the QStringLiterals
a) with atom/quark
b) without
Problem: performance impact, complexity of the atom/quark solution.
3) Never unload any plugins, possibly also compiling our own libraries and
plugins with -z nodelete. Solves most of the problems, including the C++
vtable case.
Problems: doesn't catch bypassing of QLibrary (dlopen/LoadLibrary calls),
prevents upgrading of plugins without restarting the host application.
--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
