Re: KDE CI: Frameworks kservice kf5-qt5 WindowsMSVCQt5.10 - Build # 41 - Still unstable!
On Sun, Aug 26, 2018 at 12:00 AM David Faure wrote: > > On vendredi 24 août 2018 14:00:12 CEST Ben Cooksley wrote: > > > > I guess this is something the CI tooling will need to be handling? > > > > > > I'm confused, because surely something has been done already, no? > > > Any lookup done by lib code has to find the files installed by that lib. > > > I know we moved some stuff to qrc files, but afaik not everything. > > > [though maybe the missing stuff isn't covered by unittests] > > > > We unpack everything needed into the install prefix yes. > > CMake and the rest of the tools are more than happy to use the > > resources in there as we can point them in that direction using PATH, > > CMAKE_PREFIX_PATH, etc. > > OK. I guess using the install prefix for this made sense when we installed > before running tests ;-) > Now with the new logic it seems like a bit of a strange location (since the > stuff isn't installed there), but why not, it makes no difference on Unix > (it's the "future install prefix" now). Yeah. To a certain extent it does still make sense as we need to capture the stuff which is installed for later builds. > > > Qt of course when it comes to QStandardPaths can't be given that > > direction, hence the test failures. > > Right. On Windows the only solution I can think of, is to copy that stuff into > bin/data in the builddir. I've now arranged for that to start happening. Unfortunately it looks like KService is still unhappy, just with different failure messages :) > > Indeed for a developer trying to work in a custom prefix on Windows, the setup > isn't ideal. We still need a solution for that in QStandardPaths, but I don't > know what it would be (env var? config file?). It's been hard to get this > changed in Qt because any of this would not be "standard", breaking the name > of the class :-) I'd suggest an environment variable named something along the lines of QT_DATA_PATH. I know historically we've faced quite a bit of resistance from both module maintainers as well as Qt Company employees on this whole thing though. > > AFAICS a developer would have to either install/copy dependencies into > build/bin/data or to install them into the actual standard paths. Yes. Installing dependencies into build/bin/data/ isn't really workable unfortunately as if you're building multiple things they'll get mixed up with each other so it'll end up being copying in practice I think. > > At runtime for deployed applications I think the KDE-windows people are fine > with everything in bin/data? >From my understanding that works perfectly yes. > > > > > My only concern about that though is it will make running tests > > > > locally for devs harder as they'll need to know to do this. > > > > > > Devs don't install each framework into its own install prefix, so their > > > situation is actually much easier than the CI situation :-) > > > > Ah, sorry it seems i've not explained what the CI system does here > > completely. On Linux and FreeBSD we use the same install prefix for > > everything - only populating it with what's needed for each build. > > Oh indeed. I remembered wrong. > > So /home/jenkins/install-prefix/share has the stuff needed by the current job > (and only one job at a time can run). OK. Yep. > > > On Windows we follow the same concept, with the difference that the > > install prefix in question is always within the Jenkins workspace for > > that build. > > I'm curious: why? Why not a "fixed" path like on Unix? I set it up like this because on Windows it is extremely foreseeable that software gets installed to a path which is wildly different compared to what we build it with. If we were to use a single fixed path then any code which relies on this single fixed path would pass fine on the CI system, and then end up failing on end-users systems which is not ideal. > > > In the case of a KService build for instance, it will find kdoctools, > > ki18n, etc all within C:\CI\workspace\Frameworks kservice kf5-qt5 > > WindowsMSVCQt5.11/install-prefix/ > > > > This should (theoretically at least) be reasonably close to a developers > > system. > > Not on Windows, since QStanrdPaths won't find stuff in a random custom prefix > :-) > > KDE-Windows people, any input? > > Otherwise, short term, if the CI could use build/bin instead as install prefix > for the dependencies, it seems to me that it would help. I've put code in place now to copy everything over, which should make this stuff available to tests for now at least. Ideally, we would have a solution like XDG_DATA_DIRS for Windows and macOS (which is the proper solution to this) > > -- > David Faure, fa...@kde.org, http://www.davidfaure.fr > Working on KDE Frameworks 5 > > > Cheers, Ben
Re: KDE CI: Frameworks kservice kf5-qt5 WindowsMSVCQt5.10 - Build # 41 - Still unstable!
On vendredi 24 août 2018 14:00:12 CEST Ben Cooksley wrote: > > > I guess this is something the CI tooling will need to be handling? > > > > I'm confused, because surely something has been done already, no? > > Any lookup done by lib code has to find the files installed by that lib. > > I know we moved some stuff to qrc files, but afaik not everything. > > [though maybe the missing stuff isn't covered by unittests] > > We unpack everything needed into the install prefix yes. > CMake and the rest of the tools are more than happy to use the > resources in there as we can point them in that direction using PATH, > CMAKE_PREFIX_PATH, etc. OK. I guess using the install prefix for this made sense when we installed before running tests ;-) Now with the new logic it seems like a bit of a strange location (since the stuff isn't installed there), but why not, it makes no difference on Unix (it's the "future install prefix" now). > Qt of course when it comes to QStandardPaths can't be given that > direction, hence the test failures. Right. On Windows the only solution I can think of, is to copy that stuff into bin/data in the builddir. Indeed for a developer trying to work in a custom prefix on Windows, the setup isn't ideal. We still need a solution for that in QStandardPaths, but I don't know what it would be (env var? config file?). It's been hard to get this changed in Qt because any of this would not be "standard", breaking the name of the class :-) AFAICS a developer would have to either install/copy dependencies into build/bin/data or to install them into the actual standard paths. At runtime for deployed applications I think the KDE-windows people are fine with everything in bin/data? > > > My only concern about that though is it will make running tests > > > locally for devs harder as they'll need to know to do this. > > > > Devs don't install each framework into its own install prefix, so their > > situation is actually much easier than the CI situation :-) > > Ah, sorry it seems i've not explained what the CI system does here > completely. On Linux and FreeBSD we use the same install prefix for > everything - only populating it with what's needed for each build. Oh indeed. I remembered wrong. So /home/jenkins/install-prefix/share has the stuff needed by the current job (and only one job at a time can run). OK. > On Windows we follow the same concept, with the difference that the > install prefix in question is always within the Jenkins workspace for > that build. I'm curious: why? Why not a "fixed" path like on Unix? > In the case of a KService build for instance, it will find kdoctools, > ki18n, etc all within C:\CI\workspace\Frameworks kservice kf5-qt5 > WindowsMSVCQt5.11/install-prefix/ > > This should (theoretically at least) be reasonably close to a developers > system. Not on Windows, since QStanrdPaths won't find stuff in a random custom prefix :-) KDE-Windows people, any input? Otherwise, short term, if the CI could use build/bin instead as install prefix for the dependencies, it seems to me that it would help. -- David Faure, fa...@kde.org, http://www.davidfaure.fr Working on KDE Frameworks 5
Re: KDE CI: Frameworks kservice kf5-qt5 WindowsMSVCQt5.10 - Build # 41 - Still unstable!
On Fri, Aug 24, 2018 at 11:37 PM David Faure wrote: > > On vendredi 24 août 2018 11:21:19 CEST Ben Cooksley wrote: > > > > My guess is that instead of copying it to the install prefix it needs > > > > to go to $WORKSPACE\build\bin\data\? > > > > > > That sounds right. > > > > > > While running tests for kservice, the install prefix of kservice shouldn't > > > be visible at all > > > > > > > (I thought the ECM stuff handled this though) > > > > > > I'm not sure what you're referring to. Which ECM stuff, handling what? > > > (certainly there's nothing copying dependencies into build/bin/data) > > > > Hmm Okay. I was under the impression there was code in ECM that helped > > setup for tests like this by doing the copying. > > I am really not the best person to be giving advice on current or future > solutions for CI on Windows, since I don't know what was done about all this. > > Are you referring to what I did in ECM so that the tests and executables > *created by the current repository* go into build/bin? Sure, that's done by > ECM. But here we're talking about copying the stuff from the *dependencies*, > aren't we? Sorry, that was the ECM code I was referring to previously yes. I can confirm we're talking about dependencies. > > How did this work until now, for everything else than mimetypes? > PATH surely, for libs and executables. But what about things looked up via > QStandardPaths, which isn't extensible with env vars on Windows? For tests, I suspect it simply hasn't worked. Actual applications are fine once installed of course. > > > I guess this is something the CI tooling will need to be handling? > > I'm confused, because surely something has been done already, no? > Any lookup done by lib code has to find the files installed by that lib. > I know we moved some stuff to qrc files, but afaik not everything. > [though maybe the missing stuff isn't covered by unittests] We unpack everything needed into the install prefix yes. CMake and the rest of the tools are more than happy to use the resources in there as we can point them in that direction using PATH, CMAKE_PREFIX_PATH, etc. Qt of course when it comes to QStandardPaths can't be given that direction, hence the test failures. > > > My only concern about that though is it will make running tests > > locally for devs harder as they'll need to know to do this. > > Devs don't install each framework into its own install prefix, so their > situation is actually much easier than the CI situation :-) Ah, sorry it seems i've not explained what the CI system does here completely. On Linux and FreeBSD we use the same install prefix for everything - only populating it with what's needed for each build. On Windows we follow the same concept, with the difference that the install prefix in question is always within the Jenkins workspace for that build. In the case of a KService build for instance, it will find kdoctools, ki18n, etc all within C:\CI\workspace\Frameworks kservice kf5-qt5 WindowsMSVCQt5.11/install-prefix/ This should (theoretically at least) be reasonably close to a developers system. Cheers, Ben > > But I'm not a Windows dev, so I might be missing something. > > -- > David Faure, fa...@kde.org, http://www.davidfaure.fr > Working on KDE Frameworks 5 > > >
Re: KDE CI: Frameworks kservice kf5-qt5 WindowsMSVCQt5.10 - Build # 41 - Still unstable!
On vendredi 24 août 2018 11:21:19 CEST Ben Cooksley wrote: > > > My guess is that instead of copying it to the install prefix it needs > > > to go to $WORKSPACE\build\bin\data\? > > > > That sounds right. > > > > While running tests for kservice, the install prefix of kservice shouldn't > > be visible at all > > > > > (I thought the ECM stuff handled this though) > > > > I'm not sure what you're referring to. Which ECM stuff, handling what? > > (certainly there's nothing copying dependencies into build/bin/data) > > Hmm Okay. I was under the impression there was code in ECM that helped > setup for tests like this by doing the copying. I am really not the best person to be giving advice on current or future solutions for CI on Windows, since I don't know what was done about all this. Are you referring to what I did in ECM so that the tests and executables *created by the current repository* go into build/bin? Sure, that's done by ECM. But here we're talking about copying the stuff from the *dependencies*, aren't we? How did this work until now, for everything else than mimetypes? PATH surely, for libs and executables. But what about things looked up via QStandardPaths, which isn't extensible with env vars on Windows? > I guess this is something the CI tooling will need to be handling? I'm confused, because surely something has been done already, no? Any lookup done by lib code has to find the files installed by that lib. I know we moved some stuff to qrc files, but afaik not everything. [though maybe the missing stuff isn't covered by unittests] > My only concern about that though is it will make running tests > locally for devs harder as they'll need to know to do this. Devs don't install each framework into its own install prefix, so their situation is actually much easier than the CI situation :-) But I'm not a Windows dev, so I might be missing something. -- David Faure, fa...@kde.org, http://www.davidfaure.fr Working on KDE Frameworks 5
Re: KDE CI: Frameworks kservice kf5-qt5 WindowsMSVCQt5.10 - Build # 41 - Still unstable!
On Fri, Aug 24, 2018 at 7:22 PM David Faure wrote: > > On dimanche 19 août 2018 18:11:32 CEST Ben Cooksley wrote: > > On Sun, Aug 19, 2018 at 10:29 PM David Faure wrote: > > > On lundi 13 août 2018 11:08:24 CEST Ben Cooksley wrote: > > > > Hi David, > > > > > > > > Unfortunately it doesn't look like the fix we put in place on Saturday > > > > did > > > > the trick. > > > > > > > > I have a feeling that QStandardPaths on Windows at least will also > > > > consider > > > > the Qt install prefix in addition to the local to the executable paths > > > > when > > > > searching for mime information. That would explain why the other tests > > > > are > > > > working and why making that change didn't fix it. > > > > > > Damn, by now I forgot which change we made for that particular test ;-) > > > > > > The thing is, Qt has its own builtin copy of freedesktop.org.xml so > > > QMimeDatabase just works. (So it's not even about the Qt install prefix, > > > Qt > > > doesn't install any of this, it's a qrc built into QtCore). > > > > > > > Any ideas why this still fails? > > > > > > This test is trying to actually find the mimetype xml files on disk in > > > order to build the sycoca database (mime->apps mapping). For this, the > > > XML embedded into QtCore is completely useless, we need the actual > > > shared-mime-info installed somewhere where we can find it with > > > QStandardPaths. > > > > > > How is this usually solved on Windows? > > > > > > Ah, now I think I remember the change we made: copying the mime stuff to > > > data/mime. Well, given the above, it still sounds like the right fix, but > > > I > > > wonder what we're actually copying. Is there a data/mime/text/plain.xml > > > for instance? > > > > > > Hmm, actually this fails finding the dir in the first place. > > > QStandardPaths::locateAll(QStandardPaths::GenericDataLocation, "mime", > > > QStandardPaths::LocateDirectory); > > > should really find \data\mime on Windows, there's no doubt > > > about that, so I'm wondering if the copying of data\mime happens correctly > > > and ends up in the right place? > > > > The copying is currently grabbing it from the system install provided > > by Craft (at C:\Craft\CI\windows-msvc2017_64-cl-debug\bin\data) and > > copying it to the local application install directory (at > > $WORKSPACE\install-prefix\bin\data). The copy is done for the whole of > > bin\data\ so it should grab everything. > > > > My guess is that instead of copying it to the install prefix it needs > > to go to $WORKSPACE\build\bin\data\? > > That sounds right. > > While running tests for kservice, the install prefix of kservice shouldn't be > visible at all > > > (I thought the ECM stuff handled this though) > > I'm not sure what you're referring to. Which ECM stuff, handling what? > (certainly there's nothing copying dependencies into build/bin/data) Hmm Okay. I was under the impression there was code in ECM that helped setup for tests like this by doing the copying. I guess this is something the CI tooling will need to be handling? My only concern about that though is it will make running tests locally for devs harder as they'll need to know to do this. Thoughts? > > -- > David Faure, fa...@kde.org, http://www.davidfaure.fr > Working on KDE Frameworks 5 > > > Cheers, Ben
Re: KDE CI: Frameworks kservice kf5-qt5 WindowsMSVCQt5.10 - Build # 41 - Still unstable!
On dimanche 19 août 2018 18:11:32 CEST Ben Cooksley wrote: > On Sun, Aug 19, 2018 at 10:29 PM David Faure wrote: > > On lundi 13 août 2018 11:08:24 CEST Ben Cooksley wrote: > > > Hi David, > > > > > > Unfortunately it doesn't look like the fix we put in place on Saturday > > > did > > > the trick. > > > > > > I have a feeling that QStandardPaths on Windows at least will also > > > consider > > > the Qt install prefix in addition to the local to the executable paths > > > when > > > searching for mime information. That would explain why the other tests > > > are > > > working and why making that change didn't fix it. > > > > Damn, by now I forgot which change we made for that particular test ;-) > > > > The thing is, Qt has its own builtin copy of freedesktop.org.xml so > > QMimeDatabase just works. (So it's not even about the Qt install prefix, > > Qt > > doesn't install any of this, it's a qrc built into QtCore). > > > > > Any ideas why this still fails? > > > > This test is trying to actually find the mimetype xml files on disk in > > order to build the sycoca database (mime->apps mapping). For this, the > > XML embedded into QtCore is completely useless, we need the actual > > shared-mime-info installed somewhere where we can find it with > > QStandardPaths. > > > > How is this usually solved on Windows? > > > > Ah, now I think I remember the change we made: copying the mime stuff to > > data/mime. Well, given the above, it still sounds like the right fix, but > > I > > wonder what we're actually copying. Is there a data/mime/text/plain.xml > > for instance? > > > > Hmm, actually this fails finding the dir in the first place. > > QStandardPaths::locateAll(QStandardPaths::GenericDataLocation, "mime", > > QStandardPaths::LocateDirectory); > > should really find \data\mime on Windows, there's no doubt > > about that, so I'm wondering if the copying of data\mime happens correctly > > and ends up in the right place? > > The copying is currently grabbing it from the system install provided > by Craft (at C:\Craft\CI\windows-msvc2017_64-cl-debug\bin\data) and > copying it to the local application install directory (at > $WORKSPACE\install-prefix\bin\data). The copy is done for the whole of > bin\data\ so it should grab everything. > > My guess is that instead of copying it to the install prefix it needs > to go to $WORKSPACE\build\bin\data\? That sounds right. While running tests for kservice, the install prefix of kservice shouldn't be visible at all > (I thought the ECM stuff handled this though) I'm not sure what you're referring to. Which ECM stuff, handling what? (certainly there's nothing copying dependencies into build/bin/data) -- David Faure, fa...@kde.org, http://www.davidfaure.fr Working on KDE Frameworks 5
Re: KDE CI: Frameworks kservice kf5-qt5 WindowsMSVCQt5.10 - Build # 41 - Still unstable!
On Sun, Aug 19, 2018 at 10:29 PM David Faure wrote: > > On lundi 13 août 2018 11:08:24 CEST Ben Cooksley wrote: > > Hi David, > > > > Unfortunately it doesn't look like the fix we put in place on Saturday did > > the trick. > > > > I have a feeling that QStandardPaths on Windows at least will also consider > > the Qt install prefix in addition to the local to the executable paths when > > searching for mime information. That would explain why the other tests are > > working and why making that change didn't fix it. > > Damn, by now I forgot which change we made for that particular test ;-) > > The thing is, Qt has its own builtin copy of freedesktop.org.xml so > QMimeDatabase just works. (So it's not even about the Qt install prefix, Qt > doesn't install any of this, it's a qrc built into QtCore). > > > Any ideas why this still fails? > > This test is trying to actually find the mimetype xml files on disk in order > to build the sycoca database (mime->apps mapping). For this, the XML embedded > into QtCore is completely useless, we need the actual shared-mime-info > installed somewhere where we can find it with QStandardPaths. > > How is this usually solved on Windows? > > Ah, now I think I remember the change we made: copying the mime stuff to > data/mime. Well, given the above, it still sounds like the right fix, but I > wonder what we're actually copying. Is there a data/mime/text/plain.xml > for instance? > > Hmm, actually this fails finding the dir in the first place. > QStandardPaths::locateAll(QStandardPaths::GenericDataLocation, "mime", > QStandardPaths::LocateDirectory); > should really find \data\mime on Windows, there's no doubt > about that, so I'm wondering if the copying of data\mime happens correctly and > ends up in the right place? The copying is currently grabbing it from the system install provided by Craft (at C:\Craft\CI\windows-msvc2017_64-cl-debug\bin\data) and copying it to the local application install directory (at $WORKSPACE\install-prefix\bin\data). The copy is done for the whole of bin\data\ so it should grab everything. My guess is that instead of copying it to the install prefix it needs to go to $WORKSPACE\build\bin\data\? (I thought the ECM stuff handled this though) > > -- > David Faure, fa...@kde.org, http://www.davidfaure.fr > Working on KDE Frameworks 5 > > > Cheers, Ben
Re: KDE CI: Frameworks kservice kf5-qt5 WindowsMSVCQt5.10 - Build # 41 - Still unstable!
On lundi 13 août 2018 11:08:24 CEST Ben Cooksley wrote: > Hi David, > > Unfortunately it doesn't look like the fix we put in place on Saturday did > the trick. > > I have a feeling that QStandardPaths on Windows at least will also consider > the Qt install prefix in addition to the local to the executable paths when > searching for mime information. That would explain why the other tests are > working and why making that change didn't fix it. Damn, by now I forgot which change we made for that particular test ;-) The thing is, Qt has its own builtin copy of freedesktop.org.xml so QMimeDatabase just works. (So it's not even about the Qt install prefix, Qt doesn't install any of this, it's a qrc built into QtCore). > Any ideas why this still fails? This test is trying to actually find the mimetype xml files on disk in order to build the sycoca database (mime->apps mapping). For this, the XML embedded into QtCore is completely useless, we need the actual shared-mime-info installed somewhere where we can find it with QStandardPaths. How is this usually solved on Windows? Ah, now I think I remember the change we made: copying the mime stuff to data/mime. Well, given the above, it still sounds like the right fix, but I wonder what we're actually copying. Is there a data/mime/text/plain.xml for instance? Hmm, actually this fails finding the dir in the first place. QStandardPaths::locateAll(QStandardPaths::GenericDataLocation, "mime", QStandardPaths::LocateDirectory); should really find \data\mime on Windows, there's no doubt about that, so I'm wondering if the copying of data\mime happens correctly and ends up in the right place? -- David Faure, fa...@kde.org, http://www.davidfaure.fr Working on KDE Frameworks 5
Re: KDE CI: Frameworks kservice kf5-qt5 WindowsMSVCQt5.10 - Build # 41 - Still unstable!
Hi David, Unfortunately it doesn't look like the fix we put in place on Saturday did the trick. I have a feeling that QStandardPaths on Windows at least will also consider the Qt install prefix in addition to the local to the executable paths when searching for mime information. That would explain why the other tests are working and why making that change didn't fix it. Any ideas why this still fails? Cheers, Ben On Sun, Aug 12, 2018 at 3:32 AM CI System wrote: > *BUILD UNSTABLE* > Build URL > https://build.kde.org/job/Frameworks%20kservice%20kf5-qt5%20WindowsMSVCQt5.10/41/ > Project: Frameworks kservice kf5-qt5 WindowsMSVCQt5.10 > Date of build: Sat, 11 Aug 2018 15:19:21 + > Build duration: 12 min and counting > * JUnit Tests * > Name: (root) Failed: 1 test(s), Passed: 9 test(s), Skipped: 0 test(s), > Total: 10 test(s) > > > * - Failed: TestSuite.kmimeassociationstest* >
KDE CI: Frameworks kservice kf5-qt5 WindowsMSVCQt5.10 - Build # 41 - Still unstable!
BUILD UNSTABLE Build URL https://build.kde.org/job/Frameworks%20kservice%20kf5-qt5%20WindowsMSVCQt5.10/41/ Project: Frameworks kservice kf5-qt5 WindowsMSVCQt5.10 Date of build: Sat, 11 Aug 2018 15:19:21 + Build duration: 12 min and counting JUnit Tests Name: (root) Failed: 1 test(s), Passed: 9 test(s), Skipped: 0 test(s), Total: 10 test(s)Failed: TestSuite.kmimeassociationstest
