Re: QStandardPaths::GenericDataLocation on MacOS
Hi, I just remembered that some time ago I wrote a small library that allows me to use binaries compiled against my own patched qt5-kde with an "official" Qt build: it provides the additional symbols that are in qt5-kde but not standard Qt through code injection (DYLD_INSERT_LIBRARIES). That isn't really a kosher thing to do (and actively discouraged on newer Mac OS versions) but it brings me to this: Could we design a kind of Qt plugin that provides an alternative QSP, a bit like how the plasma-integration plugin provides the KDE file dialog? In first testing/internal approach that could use a similar trick as the one I'm using with my QExtStandardPaths (a macro definition given in CPPFLAGS) and if that replaces every QStandardPaths::foo() call with (say) a KStandardLocations::foo() call we could just as well use an additional link library rather than a plugin. But ultimately I'm thinking of a plugin that provides alternative methods in QStandardPaths which user code could decide to use (or not) and which would by default just proxy their standard counterparts. Such a system would provide the same kind of flexibility as a solution where qt.conf is used, but in a way that is probably easier to deploy on other platforms, less easy to tinker with by mere mortal endusers and that probably requires touching less files. It might thus be easier to upstream. A long time ago David Faure pointed out to me that an application-wide solution that toggles QSP behaviour throughout an application is best avoided as it has the potential of doing that toggle at any time when an application is running. Instead, every dynamically loadable component should have its own, fixed QSP behaviour. (This is of course relevant mostly if not only for "linuxy" centralised installs.)
Re: QStandardPaths::GenericDataLocation on MacOS
I'll shuffle the bits in this mail a bit, in an attempt to get the discussion back to the main topic (or at least the thing that I intended to be the main topic): On Thu, 03 May 2018 23:11:56 +0200 René J.V. Bertin wrote: > On Thursday May 03 2018 21:57:03 Thomas Friedrichsmeier wrote: > > Ah, good reminder. But it doesn't install anything under /Library, > > does it? Which is why MacPorts, too, cannot support KF5 with a > > non-patched QStandardPaths, right? > > Good question, actually, because this doesn't apply only to KF5 code. > It's not forbidden to install things outside $prefix, but it's > discouraged. I'm not aware of any "pure Qt5" application which > installs things into /Library/Application Support Exactly. And that returns us to Square 1: Using GenericDataLocation simply does not work for us on Mac, but KF5 apps use it all over the place (encouraged by documentation). We will either have to stop using it, today, or patch Qt (upstream / downstream). So where do we go from here? ^This space intentionally left blank to emphasize the above.^ > Nope; my hope is that this would remove the need for patching QSP. As > you said yourself, files written to a QSP location at runtime can be > found again through QSP. That should apply to files written during > the install step too (as if the application writes them to that > location during its first run). Ok, so you're talking about _installing_ to the writeable location. Would probably work, but 1. It breaks the current separation between "vendor provided" and "user added / customized", including the implicit protection of user files that this provides. 2. It still does not allow for an installation to be limited to a single arbitrary prefix, with the benefits that provides (multiple versions / methods of installation for one application, trivial uninstallation, easy archiving, running from thumb drives, etc.) > > I don't care what it's called > > Still, best to avoid terms that are confusing because others > interpret them differently. Well, sorry for causing confusion, then. So we agree on calling the one thing "standalone apps (bundles)", but what is the complementary term? [...] > The usual content of ApplicationsLocation (.desktop files) probably > makes no sense anyway for standalone app bundles that are not > expected to play nice with applications following FreeDesktop > protocols. IOW, those files could be skipped during the installation > step in standalone build mode (again, not simply `if(APPLE)`). Besides any code querying ApplicationsLocation is _probably_ broken at least on Windows, irrespective of the filesystem location. Which does not turn this into a non-issue, but into a separate issue, that IMO should be discussed separately (and probably requires individual fixes, instead of and "global" measure). Regards Thomas pgpSL4WJcPbxS.pgp Description: OpenPGP digital signature
Re: QStandardPaths::GenericDataLocation on MacOS
On Thu, 03 May 2018 21:09:03 +0200 René J.V. Bertin wrote: > >On Fri, May 4, 2018 at 2:38 AM, Thomas Friedrichsmeier > > >> homebrew, craft). It is also about the fact that this installation > >> option causes problems, e.g. if users are to mix MacPorts and a > >> single-app-bundled KF5 app > > No, that shouldn't happen, and it doesn't with the official Kate > bundle. Standalone & self-sufficient also implies that you don't > clash with another similar application, or even with another copy of > yourself. Bundling DBus within such a standalone app bundle and > running it from there would of course be asking for trouble, but it > would also make no sense at all. Again, this is about QStandardPaths::GenericDataLocation, exclusively. _If_ applications were to actually install to the current paths defined for that on Mac "Library/Application Support", and "~Library/Application Support", on more, no less, _then_ clashes would start to happen. As far as I am aware, _none_ of the current installation approaches for KF5 does so, and as a consequence they all fail GenericDataLocation lookup with an unpatched qt. Which is one reason why I don't think "just install to the offical path, then" is an acceptable solution. Current kate "standalone" is not a good example, because it is _circumventing_ any QStandardPaths::GenericDataLocation lookup problems by shipping all its data compiled in. If it did not, it would fail to find its data files with an unpatched qt, despite following documented code patterns. > I think you're misusing the term monolithic here? Standalone > appbundles are monolithic, an aggregate framework bundle containing > all KF5 frameworks would be monolithic too. Installing under a prefix > that isn't the root is not what I'd call monolithic I don't care what it's called, I just need some terms to refer to this. My monolithic, here, was referring to _all_ in one place, instead of one app and its dependencies in one place. And yes, standalone bundles are not really all that different from "monolithic" installs, technically, except, it would be a typical situation to have more than one of them at the same time. > (and in practice > MacPorts installs its .app bundles under /Applications, not under > $prefix ;)) Ah, good reminder. But it doesn't install anything under /Library, does it? Which is why MacPorts, too, cannot support KF5 with a non-patched QStandardPaths, right? > >> Why would that be a problem? If something is written to the > >> writable > > It's not a problem if you don't care where those files are written. I don't care where files are written at runtime (as long as they can be found again). I would like not to care about where files are written at installation time, but the sad fact is they will not be found unless installed to one specific absolute path. > But if that's true, the whole issue can probably be avoided by > determining the install locations for (shared) resources from QSP > instead of hardcoding them in CMake files... Nope. That will simply lead us into the situation discussed near the top of this mail. Besides, _if_ we are both talking about the option of using an _unpatched_ Qt, then hardcoding or querying QSP makes very little practical difference. I understand you'd like to see ECM to continue to support your "Unix-like" patched QSP, and why not? ECM is under "our" control anyway, but the issue at hand is one that cannot be solved in ECM alone. > 1 problem here: ApplicationsLocation. On Mac that points > to /Applications (probably still without a test version - danger!), > which is not at all comparable to what ApplicationsLocation is used > for elsewhere. True. Any code that relies on this will likely fail on Mac _and_ Windows, so that's a valid concern. Yet, arguably, its typically going to be a milder failure, and clearly, the prevalence is much lower (check GneericDataLocation vs. ApplicationsLocation on lxr.kde.org). So let's keep it in mind, but not make it a key concern while looking for a path forward. (And in fact, I'm a bit at a loss to see how to solve _that_, short of, again, installing everything to a single absolute location.) Regards Thomas pgps7D0cMeK5z.pgp Description: OpenPGP digital signature
Re: QStandardPaths::GenericDataLocation on MacOS
On Thursday May 03 2018 21:57:03 Thomas Friedrichsmeier wrote: > _If_ applications were to actually install to the current paths defined > for that on Mac "Library/Application Support", and "~Library/Application > Support", on more, no less, _then_ clashes would start to happen. As A priori no, because the Application Support directory shouldn't be written to directly (as per Apple's rules) but should contain subdirectories, e.g. /Library/Application Support/Kate . I've never really looked at how that works in practice, but I strongly doubt that code has to append this subdirectory itself, that'd be contrary to the QSP purpose. > far as I am aware, _none_ of the current installation approaches for > KF5 does so, and as a consequence they all fail GenericDataLocation > lookup with an unpatched qt. Nope; my hope is that this would remove the need for patching QSP. As you said yourself, files written to a QSP location at runtime can be found again through QSP. That should apply to files written during the install step too (as if the application writes them to that location during its first run). > I don't care what it's called Still, best to avoid terms that are confusing because others interpret them differently. > Ah, good reminder. But it doesn't install anything under /Library, does > it? Which is why MacPorts, too, cannot support KF5 with a non-patched > QStandardPaths, right? Good question, actually, because this doesn't apply only to KF5 code. It's not forbidden to install things outside $prefix, but it's discouraged. I'm not aware of any "pure Qt5" application which installs things into /Library/Application Support > of this mail. Besides, _if_ we are both talking about the option of > using an _unpatched_ Qt, then hardcoding or querying QSP makes very > little practical difference. One very important difference: querying should work on all platforms, so remove the need for conditional code. > I understand you'd like to see ECM to continue to support your > "Unix-like" patched QSP, and why not? ECM is under "our" control > anyway, but the issue at hand is one that cannot be solved in ECM alone. Forgive me if I'm not entirely reassured that there won't be CMake code at some point that simply checks `if(APPLE)` to decide whether or not to use tweaks for generating standalone app bundles (Marble does, and its devs have simply ignored my proposed patch to reenable "linuxy" builds). > Windows, so that's a valid concern. Yet, arguably, its typically going > to be a milder failure, and clearly, the prevalence is much lower Until you run the unittests and one of those just trashes ApplicationsLocation when it's done. That actually happened to me, and I still can't believe I actually caught that before my entire /Applications had disappeared. > a path forward. (And in fact, I'm a bit at a loss to see how to solve > _that_, short of, again, installing everything to a single absolute > location.) The usual content of ApplicationsLocation (.desktop files) probably makes no sense anyway for standalone app bundles that are not expected to play nice with applications following FreeDesktop protocols. IOW, those files could be skipped during the installation step in standalone build mode (again, not simply `if(APPLE)`). Anyway, I don't want to monopolise this thread and I'm beginning to feel that's exactly what I'm doing. Apologies. R.
Re: QStandardPaths::GenericDataLocation on MacOS
Hi, > On Fri, May 4, 2018 at 2:38 AM, Thomas Friedrichsmeier > wrote: >> Hi, >> >> On Thu, 03 May 2018 16:01:46 +0200 >> René J.V. Bertin wrote: >>> On Thursday May 03 2018 15:19:33 Thomas Friedrichsmeier wrote: >>> >plan is actually to get David Faure to champion a solution (whichever >>> >one) into Qt. >>> >>> I can be wrong, but I don't see that happening beyond his general >>> support. >> >> well, let's hear him on this. >> >>> I don't see what Unix does here. /Library/Application Support is just >>> the OS X equivalent of $prefix/share and maybe parts of >>> $prefix/libexec; both are central, shared location that have a >>> per-user equivalent (~/Library/Application Support and >>> ~/.local/share). The limitations I see are purely and only related to >>> the way software has to be installed. >> >> Yes, it's about the installation options. It is _primarily_ about the >> fact that the one theorically possible installation option on Mac >> matches with _none_ of the major available solutions (macports, >> homebrew, craft). It is also about the fact that this installation >> option causes problems, e.g. if users are to mix MacPorts and a >> single-app-bundled KF5 app, or two versions of a KF5 app, or many other >> not-so-unlikely scenarios. >> >> Unix does support installing software to any prefix >> (because it provides a way to customize lookup). Windows supports >> installing software to any prefix (because it includes an >> installation-relative path in the list of paths for >> GenericDataLocation). Mac does not, because it lacks both (even though >> installation-relative paths are available in other QStandardPaths). >> >>> >It's not even hard for individual apps, today. Just compile in all >>> >data files as qresources, and the problem is done with. >>> >>> That may be true for resources that contain no executable native >>> code, but it probably won't work for libraries and plugins. >> >> Those can be handled in qt.conf (and macdeployqt auto-generates an >> appropriate qt.conf, for instance). That may or may not be our favorite >> solution, but it is an already available option (and if there is a >> concern that this could cause problems for monolithic (macports-style) >> installations, qt.conf can even be provided as a per-app qresource, >> which in principle, should be possible to generate, and compile into >> each binary, automatically. >> >>> >A fully packaged Kate is available at 88MB download for instance, >>> >despite including a whole bunch of frameworks. >>> >>> But with that you get a version of the application that just drops >>> the functionality [...] >> >> I just don't want to get into that discussion, at all, here. I >> absolutely see use-cases for both "monolithic" and "single app" >> installations. So both should be supported. Currently neither is. So >> the issue at hand isn't to discuss which of the two is superior, but to >> find out how to fix the problem affecting them both. > > It is worth noting that at one point we had a much lighter (only about > 20MB or so I think) DMG however that broke icon support. > There is definitely space for optimisation here in terms of what we're > including in the .app bundle. > > If we end up going down the QRC route (which also has the advantage of > improving speed as it eliminates the large number of small files) then > we can also ensure the icons are kept compressed on disk. > >> >>> >Again, I think this is not an either-or situation. Both centralized >>> >and standalone app installations are affected by this problem, unless >>> >relying on .qrc files, exclusively. >>> >>> There *is* an either-or situation: work with the existing >>> QStandardPaths or don't. Kate.app shows that the former is not >>> impossible, and it has the big advantage that you do everything >>> in-house without modifying Qt. >> >> QRC, sing it with me. That's the whole secret, as far as I am aware: >> Circumventing the problem by not installing files. Since I found out >> about this, I sleep a lot better as maintainer of RKWard, but it still I still think going the QRC way is the simplest solution. With that, at least in Kate (that uses a lot of frameworks), I am not aware of any real remaining issue with qstandardpaths. You can go the monolithic app bundle way with that or the "I share stuff" way like on Linux. Actually all frameworks I patched use the QRC way on Linux nowadays, too, which IMHO has no real drawbacks and even avoids a lot of hassle during deployment for e.g. Windows (where a lot of small files do suck incredibly more than on Unices with a decent filesystem). Much harder is the issue of deploying all plugins/... that are bundled with the application in a app bundle, as e.g. stuff like my macqtdeploy patch to patch the kio plugins and co. just rot away in the qt bug tracker. https://bugreports.qt.io/browse/QTBUG-48836 Greetings Christoph -- - Dr.-Ing. Christoph Cullmann - AbsInt Angewandte Informatik GmbH
Re: QStandardPaths::GenericDataLocation on MacOS
>On Fri, May 4, 2018 at 2:38 AM, Thomas Friedrichsmeier >> homebrew, craft). It is also about the fact that this installation >> option causes problems, e.g. if users are to mix MacPorts and a >> single-app-bundled KF5 app No, that shouldn't happen, and it doesn't with the official Kate bundle. Standalone & self-sufficient also implies that you don't clash with another similar application, or even with another copy of yourself. Bundling DBus within such a standalone app bundle and running it from there would of course be asking for trouble, but it would also make no sense at all. >> Unix does support installing software to any prefix >> (because it provides a way to customize lookup). More importantly, the Unix QSP are relative to Qt's install prefix (if memory serves me well; and my patched version is too). >> concern that this could cause problems for monolithic (macports-style) I think you're misusing the term monolithic here? Standalone appbundles are monolithic, an aggregate framework bundle containing all KF5 frameworks would be monolithic too. Installing under a prefix that isn't the root is not what I'd call monolithic (and in practice MacPorts installs its .app bundles under /Applications, not under $prefix ;)) >improving speed as it eliminates the large number of small files) then >we can also ensure the icons are kept compressed on disk. Doesn't Kate already get its icons from the qresource? >> Why would that be a problem? If something is written to the writable It's not a problem if you don't care where those files are written. But if that's true, the whole issue can probably be avoided by determining the install locations for (shared) resources from QSP instead of hardcoding them in CMake files... 1 problem here: ApplicationsLocation. On Mac that points to /Applications (probably still without a test version - danger!), which is not at all comparable to what ApplicationsLocation is used for elsewhere. Cheers, R.
Re: QStandardPaths::GenericDataLocation on MacOS
On Fri, May 4, 2018 at 2:38 AM, Thomas Friedrichsmeier wrote: > Hi, > > On Thu, 03 May 2018 16:01:46 +0200 > René J.V. Bertin wrote: >> On Thursday May 03 2018 15:19:33 Thomas Friedrichsmeier wrote: >> >plan is actually to get David Faure to champion a solution (whichever >> >one) into Qt. >> >> I can be wrong, but I don't see that happening beyond his general >> support. > > well, let's hear him on this. > >> I don't see what Unix does here. /Library/Application Support is just >> the OS X equivalent of $prefix/share and maybe parts of >> $prefix/libexec; both are central, shared location that have a >> per-user equivalent (~/Library/Application Support and >> ~/.local/share). The limitations I see are purely and only related to >> the way software has to be installed. > > Yes, it's about the installation options. It is _primarily_ about the > fact that the one theorically possible installation option on Mac > matches with _none_ of the major available solutions (macports, > homebrew, craft). It is also about the fact that this installation > option causes problems, e.g. if users are to mix MacPorts and a > single-app-bundled KF5 app, or two versions of a KF5 app, or many other > not-so-unlikely scenarios. > > Unix does support installing software to any prefix > (because it provides a way to customize lookup). Windows supports > installing software to any prefix (because it includes an > installation-relative path in the list of paths for > GenericDataLocation). Mac does not, because it lacks both (even though > installation-relative paths are available in other QStandardPaths). > >> >It's not even hard for individual apps, today. Just compile in all >> >data files as qresources, and the problem is done with. >> >> That may be true for resources that contain no executable native >> code, but it probably won't work for libraries and plugins. > > Those can be handled in qt.conf (and macdeployqt auto-generates an > appropriate qt.conf, for instance). That may or may not be our favorite > solution, but it is an already available option (and if there is a > concern that this could cause problems for monolithic (macports-style) > installations, qt.conf can even be provided as a per-app qresource, > which in principle, should be possible to generate, and compile into > each binary, automatically. > >> >A fully packaged Kate is available at 88MB download for instance, >> >despite including a whole bunch of frameworks. >> >> But with that you get a version of the application that just drops >> the functionality [...] > > I just don't want to get into that discussion, at all, here. I > absolutely see use-cases for both "monolithic" and "single app" > installations. So both should be supported. Currently neither is. So > the issue at hand isn't to discuss which of the two is superior, but to > find out how to fix the problem affecting them both. It is worth noting that at one point we had a much lighter (only about 20MB or so I think) DMG however that broke icon support. There is definitely space for optimisation here in terms of what we're including in the .app bundle. If we end up going down the QRC route (which also has the advantage of improving speed as it eliminates the large number of small files) then we can also ensure the icons are kept compressed on disk. > >> >Again, I think this is not an either-or situation. Both centralized >> >and standalone app installations are affected by this problem, unless >> >relying on .qrc files, exclusively. >> >> There *is* an either-or situation: work with the existing >> QStandardPaths or don't. Kate.app shows that the former is not >> impossible, and it has the big advantage that you do everything >> in-house without modifying Qt. > > QRC, sing it with me. That's the whole secret, as far as I am aware: > Circumventing the problem by not installing files. Since I found out > about this, I sleep a lot better as maintainer of RKWard, but it still > bugs me, severly, for KF5 as a whole. > >> Another thing: it seems that in your reasoning you have only >> considered the search algorithms for finding readable locations. For >> the writable locations you cannot simply append a "compatibility" >> location, and in turn that means that resources generated at runtime >> will later be found in "native" Mac locations because QSP will return >> the first location that has the requested file. > > Why would that be a problem? If something is written to the writable > location, it will then later be found. Sure, that's the point, isn't it. > Possibly by a different application, or by a different version of the > same application. Also true, but nothing new. > > Regards > Thomas Cheers, Ben
Re: QStandardPaths::GenericDataLocation on MacOS
Hi, I'll just answer a few select points, because, yes I am aware this has a tiny bit of a history, and I don't want to see a repeat. The master plan is actually to get David Faure to champion a solution (whichever one) into Qt. So before getting lost in the detail of the various approaches, I'd like to hear a word on the general direction that seems most promising. On Thu, 03 May 2018 11:24:51 +0200 René J.V. Bertin wrote: [...] > What is often overlooked is that not all of Apple's own (and > 3rd-party) software installs via simple DMGs, notably applications > that depend on or provide shared resources. I would assume that > iTunes still installs via an installer, for instance. True, yet another route would be to simply require KF5-based apps to install to /Library/Application Support. But that solution has obvious limitations that I would like to avoid, and - by analogy to Unix _and_ Windows - are entirely avoidable. > There are individual project efforts that manage to get a working > standalone app bundle using an unpatched Qt and QSP. [...] It's not even hard for individual apps, today. Just compile in all data files as qresources, and the problem is done with. As an application maintainer that is my fallback plan. But it's not really my preferred vision for KF5 as a whole. > But in the end it's the KDE model itself that cannot easily (if at > all) be adapted to desktop systems like Macs and MSWin PCs if you do > not want to provide a central install of the shared resources > including certain external dependencies like DBus. Yes, but "central" is pretty relative, here. I know single app installers are not your vision of KF5 on Mac, and I certainly don't want to push them as _the_ way to go, but to be honest, I'm quite impressed with the .dmg's created on the binary-factory for Mac, today. A fully packaged Kate is available at 88MB download for instance, despite including a whole bunch of frameworks. Yet that seems mostly orthogonal to the issue at hand to me. Both monolithic and single app installation are facing the same problem right now, and essentially need (one of) the same solution to be able to fix it. > >Not sure about the MacPorts-take on doing something like this, but > >for > > MacPorts (and Fink, which still exists!) use a single prefix in which > everything is installed like Gentoo Prefix or pkgsrc would do it. I'm absolutely aware of that. The question is simply: Would MacPorts tolaterate a symlink from /opt/local/Resources to /opt/local/share (/opt/local being the installation prefix)? If so, the simple patch I presented should immediately allow a clean way forward for KF5 on MacPorts (if upstreamed). > Patch size is not a problem at least if the patch is potentially > upstreamable. But it could be, if the responsible reviewer is a bit uninterested... > The problem I see with qt.conf is that it is > application-specific; each app bundle will have to provide its own > copy. True, but so what, if it can be generated, automatically. > The alternative would be to convince Christoph Cullmann and > like-minded people to work on a way to mould their respective > approaches for building standalone app bundles into something that > can be added to the ECM. Again, I think this is not an either-or situation. Both centralized and standalone app installations are affected by this problem, unless relying on .qrc files, exclusively. Regards Thomas pgpao32bL4V_q.pgp Description: OpenPGP digital signature
Re: QStandardPaths::GenericDataLocation on MacOS
Hi, On Thu, 03 May 2018 16:01:46 +0200 René J.V. Bertin wrote: > On Thursday May 03 2018 15:19:33 Thomas Friedrichsmeier wrote: > >plan is actually to get David Faure to champion a solution (whichever > >one) into Qt. > > I can be wrong, but I don't see that happening beyond his general > support. well, let's hear him on this. > I don't see what Unix does here. /Library/Application Support is just > the OS X equivalent of $prefix/share and maybe parts of > $prefix/libexec; both are central, shared location that have a > per-user equivalent (~/Library/Application Support and > ~/.local/share). The limitations I see are purely and only related to > the way software has to be installed. Yes, it's about the installation options. It is _primarily_ about the fact that the one theorically possible installation option on Mac matches with _none_ of the major available solutions (macports, homebrew, craft). It is also about the fact that this installation option causes problems, e.g. if users are to mix MacPorts and a single-app-bundled KF5 app, or two versions of a KF5 app, or many other not-so-unlikely scenarios. Unix does support installing software to any prefix (because it provides a way to customize lookup). Windows supports installing software to any prefix (because it includes an installation-relative path in the list of paths for GenericDataLocation). Mac does not, because it lacks both (even though installation-relative paths are available in other QStandardPaths). > >It's not even hard for individual apps, today. Just compile in all > >data files as qresources, and the problem is done with. > > That may be true for resources that contain no executable native > code, but it probably won't work for libraries and plugins. Those can be handled in qt.conf (and macdeployqt auto-generates an appropriate qt.conf, for instance). That may or may not be our favorite solution, but it is an already available option (and if there is a concern that this could cause problems for monolithic (macports-style) installations, qt.conf can even be provided as a per-app qresource, which in principle, should be possible to generate, and compile into each binary, automatically. > >A fully packaged Kate is available at 88MB download for instance, > >despite including a whole bunch of frameworks. > > But with that you get a version of the application that just drops > the functionality [...] I just don't want to get into that discussion, at all, here. I absolutely see use-cases for both "monolithic" and "single app" installations. So both should be supported. Currently neither is. So the issue at hand isn't to discuss which of the two is superior, but to find out how to fix the problem affecting them both. > >Again, I think this is not an either-or situation. Both centralized > >and standalone app installations are affected by this problem, unless > >relying on .qrc files, exclusively. > > There *is* an either-or situation: work with the existing > QStandardPaths or don't. Kate.app shows that the former is not > impossible, and it has the big advantage that you do everything > in-house without modifying Qt. QRC, sing it with me. That's the whole secret, as far as I am aware: Circumventing the problem by not installing files. Since I found out about this, I sleep a lot better as maintainer of RKWard, but it still bugs me, severly, for KF5 as a whole. > Another thing: it seems that in your reasoning you have only > considered the search algorithms for finding readable locations. For > the writable locations you cannot simply append a "compatibility" > location, and in turn that means that resources generated at runtime > will later be found in "native" Mac locations because QSP will return > the first location that has the requested file. Why would that be a problem? If something is written to the writable location, it will then later be found. Sure, that's the point, isn't it. Possibly by a different application, or by a different version of the same application. Also true, but nothing new. Regards Thomas pgp30EqsMkE4W.pgp Description: OpenPGP digital signature
Re: QStandardPaths::GenericDataLocation on MacOS
On Thursday May 03 2018 15:19:33 Thomas Friedrichsmeier wrote: Hi, >plan is actually to get David Faure to champion a solution (whichever >one) into Qt. I can be wrong, but I don't see that happening beyond his general support. Also, what are the chances that this will be overhauled anyway for Qt6? >install to /Library/Application Support. But that solution has obvious >limitations that I would like to avoid, and - by analogy to Unix >_and_ Windows - are entirely avoidable. I don't see what Unix does here. /Library/Application Support is just the OS X equivalent of $prefix/share and maybe parts of $prefix/libexec; both are central, shared location that have a per-user equivalent (~/Library/Application Support and ~/.local/share). The limitations I see are purely and only related to the way software has to be installed. >It's not even hard for individual apps, today. Just compile in all >data files as qresources, and the problem is done with. That may be true for resources that contain no executable native code, but it probably won't work for libraries and plugins. >A fully packaged Kate is available at 88MB download for instance, >despite including a whole bunch of frameworks. But with that you get a version of the application that just drops the functionality it gets through DBus, uses the alien-looking Breeze icons and default colour palette, plus the "native" Mac widget style which makes it look like an application for the visually impaired (everything way too big). It works, but IMHO no longer has any advantages over well-established truly native competitors like BBEdit (which clocks in at 32Mb *uncompressed*). And: supposing those 88Mb are the uncompressed app bundle size, how much of that has to be duplicated in each and every app that uses the Kate kpart? >I'm absolutely aware of that. The question is simply: Would MacPorts >tolaterate a symlink from /opt/local/Resources to /opt/local/share They probably wouldn't like it, but if the dislike is strong enough it will lead to a small patch that undoes or modifies the change. Careful though, should probably be something like QLibraryInfo::prefixPath and not the method from QApplication which will point to the MacOS directory inside the app bundle. >Again, I think this is not an either-or situation. Both centralized and >standalone app installations are affected by this problem, unless >relying on .qrc files, exclusively. There *is* an either-or situation: work with the existing QStandardPaths or don't. Kate.app shows that the former is not impossible, and it has the big advantage that you do everything in-house without modifying Qt. Another thing: it seems that in your reasoning you have only considered the search algorithms for finding readable locations. For the writable locations you cannot simply append a "compatibility" location, and in turn that means that resources generated at runtime will later be found in "native" Mac locations because QSP will return the first location that has the requested file. Which reminds me that I've already suggested several times that it would be useful to rethink the build system so that it determines the install locations based on QSP (using qtpaths or a dedicated small utility). Cheers, R.
Re: QStandardPaths::GenericDataLocation on MacOS
On Thursday May 03 2018 22:22:30 Ben Cooksley wrote:
Hi,
> Last time around the Qt Core maintainer vetoed our attempts to get
> this changed, and forced us to liase with a Qt Company employee whose
> responsiveness was lacking (to say the least). Suffice to say, that
> effectively killed the effort last time round.
A Qt Core maintainer who isn't a Qt employee?
IIRC, David Faure is the original author of the QSP class, but maybe not of the
actual Mac implementation. David was constructive enough in the past in his
feedback on my patch, though we did have some disagreements and we never got
around to sitting down to get it sorted out once and for all.
Note that I wouldn't really mind upstreaming only a minimal change to QSP, as
long as that doesn't make achieving my patch more complicated (or worse,
requires a serious overhaul of all my current KF5 ports).
I really am quite happy with how my current patch works; it allows all KF5
software that doesn't do Mac-incompatible things to work (almost) "out of the
box". It also allows me to focus on function over form so to speak ("form"
being the way how/where things are installed/bundled).
R.
Re: QStandardPaths::GenericDataLocation on MacOS
On Thu, May 3, 2018 at 8:02 PM, Thomas Friedrichsmeier
wrote:
> Hi!
Hi all,
>
> Quick summary, first:
> KF5 relies on storing data in QStandardPaths::GenericDataLocation, but
> that location is problematic on Mac because it refers to absolute,
> non-customizable system paths, only. We'd like to avoid patching Qt, so
> we are looking to come up with a solution that has a chance of getting
> upstreamed.
>
> The problem in a lot more detail:
> Feel free to skip over any sections that you are already familiar with.
>
> -- Usage of GenericDataLocation in KF5 --
> Many KF5 frameworks store data or look for data in
> QStandardPaths::GenericDataLocation . KXmlGui and Syntax-Highlighting
> are just two such examples, which, importantly, are designed such that
> users (people or applications) can supply additional data files in
> specific directories, such as returned by
> QStandardPaths::locateAll(QStandardPaths::GenericDataLocation,
> QStringLiteral("kxmlgui5/"))
> Using QStandardPaths::GenericDataLocation has also been suggested
> in places such as
> https://community.kde.org/Frameworks/Porting_Notes/KStandardDirs .
>
> Nowadays compiled in qrc resources are used instead of installed files,
> increasingly, and this removes any worries about file system paths.
> However, due to the above we should assume that use of
> QStandardPaths::GenericDataLocation is wide-spread in KF5-based
> applications, and must remain supported.
>
> -- The situation on Linux --
> On Unix systems (not Mac), the paths returned for
> QStandardPaths::GenericDataLocation correspond to those of the
> XDG_DATA_DIRS environment variable. These will usually be system wide
> paths including /usr/share and /usr/local/share. Since the typical
> application deployment happens to one of those directories, installed
> files will be accessible to frameworks out of the box. In case of
> non-standard installation paths, modifying XDG_DATA_DIRS provides a
> straight-forward mechanism to adjust lookup.
>
> -- The situation on Windows --
> On Windows, GenericDataLocation is mapped to the system specific
> "FOLDERID_LocalAppData", which includes some system wide paths, but
> also "", and "/data", where is the location of
> the binary. While these paths are not directly controllable, having the
> relative path in them allows
> - installing a group of applications sharing the same file or
> - installing a single application (with supporting frameworks)
> both to arbitrary installation paths. The only thing that is not -
> easily - possible compared to Unix, is having two Applications
> installed to two separate installation root directories, and still
> sharing the same data files.
>
> The expected filesystem layout is somewhat different from Unix (data
> instead of share), but those differences can be handled (and are
> handled) by ECM:
> https://github.com/KDE/extra-cmake-modules/blob/master/kde-modules/KDEInstallDirs.cmake#L529
> . ("/data" is the path chosen from the available options).
>
> -- The situation on MacOS --
> On MacOS, GenericDataLocation is mapped to the system specific
> "NSApplicationSupportDirectory", which contains the paths
> "/Library/Application Support/" and "~/Library/Application Support/".
> Notably, any installation-relative path is missing, as are any options
> for customization.
>
> As an interesting aside, the handling is different for AppDataLocation
> and AppLocalDataLocation, which add the relative path
> "/../Resources/" (and also append "" to all paths).
> This matches with the notion that data either belongs to exactly one
> application (AppDataLocation) or to _all_ applications on the system
> (GenericDataLocation), but it does not support the notion that there
> may be a group/bundle of applications sharing their data, or that a
> single application may look up installation-relative data in any other
> than "/../Resources/". (Importantly not in
> "/../Resources/kxmlgui5", for intance).
>
> We'll talk about this in a bit, but for now, back to the
> current definition on GenericDataLocation on Mac:
>
> This setup severely limits the viable deployment options for KF5-based
> applications on Mac to basically just one: They must be installed (or
> at least their data must be installed) to a fixed central system
> folder. Unfortunately, this conflicts with all major existing
> approaches:
>
> - MacPorts: Uses a centralized installation, _but_ inside an own
> installation root (/opt/local/ by default), and is designed not to
> touch any paths outside that installation root. MacPorts _still_ does
> not have any official KF5 packages, and the ones provided by our own
> kde-mac people rely on a patched Qt (details in a bit).
> - Homebrew: Also a centralized installation, also inside its own
> installation root (typically in a user's home directory). Can cope
> better because it is not afraid to create symlinks outside its
> installation root (we'll discuss that, too).
> - Craft ("norm
QStandardPaths::GenericDataLocation on MacOS
Hi!
Quick summary, first:
KF5 relies on storing data in QStandardPaths::GenericDataLocation, but
that location is problematic on Mac because it refers to absolute,
non-customizable system paths, only. We'd like to avoid patching Qt, so
we are looking to come up with a solution that has a chance of getting
upstreamed.
The problem in a lot more detail:
Feel free to skip over any sections that you are already familiar with.
-- Usage of GenericDataLocation in KF5 --
Many KF5 frameworks store data or look for data in
QStandardPaths::GenericDataLocation . KXmlGui and Syntax-Highlighting
are just two such examples, which, importantly, are designed such that
users (people or applications) can supply additional data files in
specific directories, such as returned by
QStandardPaths::locateAll(QStandardPaths::GenericDataLocation,
QStringLiteral("kxmlgui5/"))
Using QStandardPaths::GenericDataLocation has also been suggested
in places such as
https://community.kde.org/Frameworks/Porting_Notes/KStandardDirs .
Nowadays compiled in qrc resources are used instead of installed files,
increasingly, and this removes any worries about file system paths.
However, due to the above we should assume that use of
QStandardPaths::GenericDataLocation is wide-spread in KF5-based
applications, and must remain supported.
-- The situation on Linux --
On Unix systems (not Mac), the paths returned for
QStandardPaths::GenericDataLocation correspond to those of the
XDG_DATA_DIRS environment variable. These will usually be system wide
paths including /usr/share and /usr/local/share. Since the typical
application deployment happens to one of those directories, installed
files will be accessible to frameworks out of the box. In case of
non-standard installation paths, modifying XDG_DATA_DIRS provides a
straight-forward mechanism to adjust lookup.
-- The situation on Windows --
On Windows, GenericDataLocation is mapped to the system specific
"FOLDERID_LocalAppData", which includes some system wide paths, but
also "", and "/data", where is the location of
the binary. While these paths are not directly controllable, having the
relative path in them allows
- installing a group of applications sharing the same file or
- installing a single application (with supporting frameworks)
both to arbitrary installation paths. The only thing that is not -
easily - possible compared to Unix, is having two Applications
installed to two separate installation root directories, and still
sharing the same data files.
The expected filesystem layout is somewhat different from Unix (data
instead of share), but those differences can be handled (and are
handled) by ECM:
https://github.com/KDE/extra-cmake-modules/blob/master/kde-modules/KDEInstallDirs.cmake#L529
. ("/data" is the path chosen from the available options).
-- The situation on MacOS --
On MacOS, GenericDataLocation is mapped to the system specific
"NSApplicationSupportDirectory", which contains the paths
"/Library/Application Support/" and "~/Library/Application Support/".
Notably, any installation-relative path is missing, as are any options
for customization.
As an interesting aside, the handling is different for AppDataLocation
and AppLocalDataLocation, which add the relative path
"/../Resources/" (and also append "" to all paths).
This matches with the notion that data either belongs to exactly one
application (AppDataLocation) or to _all_ applications on the system
(GenericDataLocation), but it does not support the notion that there
may be a group/bundle of applications sharing their data, or that a
single application may look up installation-relative data in any other
than "/../Resources/". (Importantly not in
"/../Resources/kxmlgui5", for intance).
We'll talk about this in a bit, but for now, back to the
current definition on GenericDataLocation on Mac:
This setup severely limits the viable deployment options for KF5-based
applications on Mac to basically just one: They must be installed (or
at least their data must be installed) to a fixed central system
folder. Unfortunately, this conflicts with all major existing
approaches:
- MacPorts: Uses a centralized installation, _but_ inside an own
installation root (/opt/local/ by default), and is designed not to
touch any paths outside that installation root. MacPorts _still_ does
not have any official KF5 packages, and the ones provided by our own
kde-mac people rely on a patched Qt (details in a bit).
- Homebrew: Also a centralized installation, also inside its own
installation root (typically in a user's home directory). Can cope
better because it is not afraid to create symlinks outside its
installation root (we'll discuss that, too).
- Craft ("normal installation"): Again, a centralized installation,
typically inside a user's home directory. No real solution, to the
problem, yet (the reason for this mail).
- Craft ("application bundle images"): Craft can also create .dmg
packages from singl
Re: QStandardPaths::GenericDataLocation on MacOS
On Thursday May 03 2018 10:02:35 Thomas Friedrichsmeier wrote: Hi, I have mostly skimmed over your long email but allow me to warn you firsthand that I've tried to get feedback and constructive collaboration like this before and never got very far because the prevailing attitude is 1) don't patch Qt and 2) be as Mac'ish as possible (without necessarily having a lot of experience with what that actually means). What is often overlooked is that not all of Apple's own (and 3rd-party) software installs via simple DMGs, notably applications that depend on or provide shared resources. I would assume that iTunes still installs via an installer, for instance. There are individual project efforts that manage to get a working standalone app bundle using an unpatched Qt and QSP. It is rather evident that this should be possible because Qt designed QSP for this purpose AND to allow using Qt for apps offered via Apple's app store. This works well enough for applications that don't don't depend on others and don't need DBus to advertise their services. But in the end it's the KDE model itself that cannot easily (if at all) be adapted to desktop systems like Macs and MSWin PCs if you do not want to provide a central install of the shared resources including certain external dependencies like DBus. On Mac the "native" way to do that would be a single framework in /Library/Frameworks which bundles all KF5 frameworks like certain of Apple's frameworks contain multiple sub-frameworks, plus a corresponding store in /Library/Application Support where other KF5 libraries, plugins etc. can install their support files. That would of course represent a huge effort that should have been started immediately with KF5 5.0.0 . A couple of years into "supporting" KF5 on Mac and trying to get traction for it on/in MacPorts I have come to the conclusion that there just is no demand beyond possibly the few applications that are sufficiently self-sufficient to be offered "as an icon on a DMG"; Kate, probably KDevelop, Krita and possibly the other Calligra members and maybe a handful of others. digiKam has a Mac installer, apparently its target audience is expected to have enough computer literacy to know what to do with one of those :) About MacPorts and official KF5 ports: unlikely to happen unless someone else is willing to recreate ports so you get a sensible experience with the unpatched Qt5 provided by the official Qt5 port (or adapt my KF5 ports minimally to that effect). The hurdle here is that I've never been able to arrive at a constructive collaboration with the Qt5 port maintainer; without that there will be no official alternative Qt5 port for use with KDE and we'll remain stuck with the need for a custom ports tree. >1. It requires "smart" installers and uninstallers, will not work with >plain .dmg's. >2. Once installed, an application cannot trivially be moved, any more, >as that would also require adjusting the symlinks. >3. It only really works for a single centralized installation. However, >if installing two KF5-based applications AppA and AppB to separate >installation roots, where would - for instance - ~/Library/Application >Support/kxmlgui5 point to? A directory inside AppA's installation >root, or inside AppB's installation root? It cannot be both. > >appropriate. The drawbacks I can think of: >1. A rather large patch It is largish because designed with upstreamability in mind and I thin it also still contains echoes of earlier approaches that could be removed. A patch for a Qt version only to be used with KF5 code can be a lot leaner. The stale ticket you refer to is probably the one on which it was said that the Qt devs are open for this kind of patch, provided it is well justified (argued) and it is transparent by default so that AppStore compatibility isn't jeopardised. My idea has always been to test and tune my patch in the wild and then present it again with community back-up. There has also been an exchange with someone from the MSWin universe about linking this patch to to-be-developed qt.conf functionality, e.g. to replace the XDG_DATA_DIRS env. variable or possibly even the option to define all (alternative) QSP locations via qt.conf . >2. There may be a concern about side-effects, as the behavior of Qt >will depend on an environment variable where it did not before, with >_potential_ interaction between separate installations. The env. variable is not required, it was added to conform with the behaviour on other Unix variants but I never use it myself. XDG_DATA_DIRS is not an env. variable that is used by other Mac software, but it shouldn't be that hard to find an alternative for it should that be necessary. >3. Will also need some "cooperation" from the calling app in order to >work for all installation scenarios (XDG_DATA_DIRS will have to be >set/adjusted, and an extra setup call may be needed to request >"Unix-behavior" from QStandardPaths). No
