[RFC] QEMU as Xcode project on macOS
I've recently been thinking about how feasible a stripped down Xcode project for QEMU would be, i.e. you just get the QEMU sources, click on qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, nothing. I've done this before for gtk(mm), which you might know, depends on approx. 24 individual libraries (glib, jpeg, png, pixman, atk, gdk, cairo, pixman, graphene, sigcpp, ... gtk, gtkmm) that you would usually all need to download and ./configure && make & make install each individually on macOS. Or right, you could alternatively "just install" them from Homebrew, MacPorts, Fink. But no matter which solution you choose, it easily ends up in a mess (conflicts, misbehaviours) on macOS to install libs and apps globally. And I think that's the problem why there are currently relatively little contribution for QEMU coming from devs on macOS. Because you don't want to install things globally on a macOS system, it's simply not working well there as it does with Linux distros. And the other thing is: I've tested the waters with Apple and filed a QEMU related macOS bug with them. The response was like expected; they basically said 'if there's no Xcode project, then we don't investigate it'. The question is, and I don't have the big picture of QEMU yet to judge that, how much is auto generated for QEMU i.e. with custom scripts that would probably destroy this plan? There are these trace calls that are auto generated, is there more like the TCG part for instance? What I could imagine: a hand crafted Xcode project as a starting point, and if that works out, switching to auto generating that Xcode project from the Meson inftrastructure to avoid multiplication of maintenance effort. Best regards, Christian Schoenebeck
Re: [RFC] QEMU as Xcode project on macOS
On Wed, 9 Sep 2020 at 13:56, Christian Schoenebeck wrote: > I've recently been thinking about how feasible a stripped down Xcode project > for QEMU would be, i.e. you just get the QEMU sources, click on > qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, nothing. How would this work? Can you have an XCode project that just says "to build this program we will run configure and make" ? We definitely don't want to have two parallel mechanisms for specifying how to build QEMU, because they'll just get out of sync. > each individually on macOS. Or right, you could alternatively "just install" > them from Homebrew, MacPorts, Fink. But no matter which solution you choose, > it easily ends up in a mess (conflicts, misbehaviours) on macOS to install > libs and apps globally. And I think that's the problem why there are currently > relatively little contribution for QEMU coming from devs on macOS. Because you > don't want to install things globally on a macOS system, it's simply not > working well there as it does with Linux distros. My experience with homebrew has been pretty good overall. If there's a better way to handle OSX hosts that doesn't require us to carry around all our dependencies (which is impractical) that would be interesting to investigate. > The question is, and I don't have the big picture of QEMU yet to judge that, > how much is auto generated for QEMU i.e. with custom scripts that would > probably destroy this plan? There are these trace calls that are auto > generated, is there more like the TCG part for instance? Lots of stuff is autogenerated, yes. thanks -- PMM
Re: [RFC] QEMU as Xcode project on macOS
On Wed, Sep 09, 2020 at 02:56:46PM +0200, Christian Schoenebeck wrote: > I've recently been thinking about how feasible a stripped down Xcode project > for QEMU would be, i.e. you just get the QEMU sources, click on > qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, nothing. > > I've done this before for gtk(mm), which you might know, depends on approx. > 24 > individual libraries (glib, jpeg, png, pixman, atk, gdk, cairo, pixman, > graphene, sigcpp, ... gtk, gtkmm) that you would usually all need to download > and > > ./configure && make & make install > > each individually on macOS. Or right, you could alternatively "just install" > them from Homebrew, MacPorts, Fink. But no matter which solution you choose, > it easily ends up in a mess (conflicts, misbehaviours) on macOS to install > libs and apps globally. And I think that's the problem why there are > currently > relatively little contribution for QEMU coming from devs on macOS. Because > you > don't want to install things globally on a macOS system, it's simply not > working well there as it does with Linux distros. > > And the other thing is: I've tested the waters with Apple and filed a QEMU > related macOS bug with them. The response was like expected; they basically > said 'if there's no Xcode project, then we don't investigate it'. > > The question is, and I don't have the big picture of QEMU yet to judge that, > how much is auto generated for QEMU i.e. with custom scripts that would > probably destroy this plan? There are these trace calls that are auto > generated, is there more like the TCG part for instance? > > What I could imagine: a hand crafted Xcode project as a starting point, and > if > that works out, switching to auto generating that Xcode project from the > Meson > inftrastructure to avoid multiplication of maintenance effort. The current way we generate a makefile from ninja output is not our long term desired approach. Eventually the intent is that we should be able to use meson + ninja exclusively. The ninja -> make convertor we currently rely on introduces maint problems of its own. So I don't think we want to introduce a ninja -> Xcode converter, as that is still effectively giving us 1 + 1/2 different build systems, so is a new maint burden. Ideally any xcode setup would just invoke whatever our standard build tools are IMHO, so it has zero possibility of introducing new build problems. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [RFC] QEMU as Xcode project on macOS
On 09/09/2020 14.56, Christian Schoenebeck wrote: > I've recently been thinking about how feasible a stripped down Xcode project > for QEMU would be, i.e. you just get the QEMU sources, click on > qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, nothing. Meson seems to have some exporter for Xcode according to https://mesonbuild.com/IDE-integration.html ... maybe you can harness that feature somehow? > The question is, and I don't have the big picture of QEMU yet to judge that, > how much is auto generated for QEMU i.e. with custom scripts that would > probably destroy this plan? There are these trace calls that are auto > generated, is there more like the TCG part for instance? Yes, I think we generate code in a couple of places, e.g. the code in target/s390x/ uses a "gen-features" helper to generate some code. So implementing a separate Xcode project that does not use the main build files does not sound very appealing. Thomas
Re: [RFC] QEMU as Xcode project on macOS
> On 9 Sep 2020, at 16:30, Peter Maydell wrote: > > My experience with homebrew has been pretty good overall. Homebrew is fine for use cases that don't mind it is a rolling distribution. :-( For complex builds it is safer to lock the environment to certain known to work versions, and homebrew cannot do this. You can disable the auto-update feature, and keep it locked to a set of versions, but usually only within a macOS version. After you update the system, usually you have to update homebrew. For my production xPack builds, I simply could not use homebrew, I had to recreate a build environment in a separate folder, with the desired versions of the tools. Regards, Liviu
Re: [RFC] QEMU as Xcode project on macOS
On Wed, Sep 09, 2020 at 03:41:02PM +0200, Thomas Huth wrote: > On 09/09/2020 14.56, Christian Schoenebeck wrote: > > I've recently been thinking about how feasible a stripped down Xcode > > project > > for QEMU would be, i.e. you just get the QEMU sources, click on > > qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, nothing. > > Meson seems to have some exporter for Xcode according to > https://mesonbuild.com/IDE-integration.html ... maybe you can harness > that feature somehow? I feel like using the Meson XCode generator still ends up giving us 1 + 1/2 distinct build systems, as I very much doubt that there is a guarantee of behavioural parity between its XCode output and Ninja output. > > > The question is, and I don't have the big picture of QEMU yet to judge > > that, > > how much is auto generated for QEMU i.e. with custom scripts that would > > probably destroy this plan? There are these trace calls that are auto > > generated, is there more like the TCG part for instance? > > Yes, I think we generate code in a couple of places, e.g. the code in > target/s390x/ uses a "gen-features" helper to generate some code. So > implementing a separate Xcode project that does not use the main build > files does not sound very appealing. > > Thomas > > Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [RFC] QEMU as Xcode project on macOS
> On Sep 9, 2020, at 8:56 AM, Christian Schoenebeck > wrote: > > I've recently been thinking about how feasible a stripped down Xcode project > for QEMU would be, i.e. you just get the QEMU sources, click on > qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, nothing. > > I've done this before for gtk(mm), which you might know, depends on approx. > 24 > individual libraries (glib, jpeg, png, pixman, atk, gdk, cairo, pixman, > graphene, sigcpp, ... gtk, gtkmm) that you would usually all need to download > and > > ./configure && make & make install > > each individually on macOS. Or right, you could alternatively "just install" > them from Homebrew, MacPorts, Fink. But no matter which solution you choose, > it easily ends up in a mess (conflicts, misbehaviours) on macOS to install > libs and apps globally. And I think that's the problem why there are > currently > relatively little contribution for QEMU coming from devs on macOS. Because > you > don't want to install things globally on a macOS system, it's simply not > working well there as it does with Linux distros. > > And the other thing is: I've tested the waters with Apple and filed a QEMU > related macOS bug with them. The response was like expected; they basically > said 'if there's no Xcode project, then we don't investigate it'. > > The question is, and I don't have the big picture of QEMU yet to judge that, > how much is auto generated for QEMU i.e. with custom scripts that would > probably destroy this plan? There are these trace calls that are auto > generated, is there more like the TCG part for instance? > > What I could imagine: a hand crafted Xcode project as a starting point, and > if > that works out, switching to auto generating that Xcode project from the > Meson > inftrastructure to avoid multiplication of maintenance effort. > > Best regards, > Christian Schoenebeck > I think the solution to this problem is to switch over the CMake (https://en.wikipedia.org/wiki/CMake). It is a build system that lets you specify how you want to build software. There are many targets available including 'make' and an Xcode project file.
Re: [RFC] QEMU as Xcode project on macOS
On 09/09/20 15:41, Thomas Huth wrote: >> The question is, and I don't have the big picture of QEMU yet to judge that, >> how much is auto generated for QEMU i.e. with custom scripts that would >> probably destroy this plan? There are these trace calls that are auto >> generated, is there more like the TCG part for instance? > Yes, I think we generate code in a couple of places, e.g. the code in > target/s390x/ uses a "gen-features" helper to generate some code. So > implementing a separate Xcode project that does not use the main build > files does not sound very appealing. More than a couple places---all of QAPI and tracing code is autogenerated. Perhaps Meson's xcodeproj backend can be improved to support everything that QEMU requires, I'm not sure if there are any limitations in Xcode that prevent that. Paolo
Re: [RFC] QEMU as Xcode project on macOS
On Mittwoch, 9. September 2020 15:30:09 CEST Peter Maydell wrote: > On Wed, 9 Sep 2020 at 13:56, Christian Schoenebeck > > wrote: > > I've recently been thinking about how feasible a stripped down Xcode > > project for QEMU would be, i.e. you just get the QEMU sources, click on > > qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, > > nothing. > How would this work? Can you have an XCode project that just > says "to build this program we will run configure and make" ? You can run shell scripts. There are two main mechanisms in Xcode for that, one is "Build phases" where you can define shell code being executed for an individual "target" (e.g. a lib/app being built), which shell to be used (e.g. sh, zsh, bash, python), the actual shell code to be executed, and optionally a list of output files (e.g. .c files) said to be generated by the shell code which then in turn are compiled by Xcode according to their file type: https://www.mokacoding.com/blog/better-build-phase-scripts/ Limitation: the precise list of generated output files needs to be specified to the Xcode project if they need to be compiled. So you cannot let the shell script dynamically tell Xcode "here are the source files I generated: [...] compile them please". For that reason there is a 2nd mechanism in Xcode called "Build Rules", which basically allow you defining custom source file types, so e.g. Xcode would run a shell script for every source file that ends with "*.foo", the shell script would then generate for each one of them e.g. a .c file, which then would in turn be compiled by Xcode with clang. In this example somebody auto generates .c files from .css files (sorry, dusty screenshots, but it's still the same thing): https://www.cocoanetics.com/2012/02/xcode-build-rules/ > We definitely don't want to have two parallel mechanisms for > specifying how to build QEMU, because they'll just get out of sync. Sure, the Xcode project file should be auto generated for that reason, otherwise it's DOA. > > each individually on macOS. Or right, you could alternatively "just > > install" them from Homebrew, MacPorts, Fink. But no matter which solution > > you choose, it easily ends up in a mess (conflicts, misbehaviours) on > > macOS to install libs and apps globally. And I think that's the problem > > why there are currently relatively little contribution for QEMU coming > > from devs on macOS. Because you don't want to install things globally on > > a macOS system, it's simply not working well there as it does with Linux > > distros. > > My experience with homebrew has been pretty good overall. > If there's a better way to handle OSX hosts that doesn't > require us to carry around all our dependencies (which is > impractical) that would be interesting to investigate. It probably depends on how many and which FOSS projects you compiled on Mac. I no longer install from Homebrew and co, nor do I "pip install" things for the same reasons. And AFAICS other devs on Mac made similar experiences. Plus I am used to FOSS projects having more issues on Mac then on Linux. If it's an Xcode generated app that crashes, then Xcode automatically opens the precise crashed lib's source file in the editor, along with the stacktrace, you fix the bug, click on "Run" and that's it. Without Xcode that would be like 45 mins just for recompiling the relevant dependencies. On Mittwoch, 9. September 2020 15:40:05 CEST Daniel P. Berrangé wrote: > Ideally any xcode setup would just invoke whatever our standard > build tools are IMHO, so it has zero possibility of introducing > new build problems. Then you would not win anything on Mac. The problematic on macOS is that Apple froze many standard FOSS tools that switched to GPL3. So by default you just have e.g. GNU make 3.81 (2006), Bash 3.2.57 (2007), ... unless you start to manually install them (e.g. from Homebrew & Co). And being forced to use Meson on Mac with all its Python dependencies does not make it easier. IMO the Xcode project file would ideally be auto generated from the existing build infrastructure (e.g. Meson) and the scripts that are still required to be run by Xcode should be able to work with the old versions of build tools installed on a standard macOS system. On Mittwoch, 9. September 2020 15:41:02 CEST Thomas Huth wrote: > On 09/09/2020 14.56, Christian Schoenebeck wrote: > > I've recently been thinking about how feasible a stripped down Xcode > > project for QEMU would be, i.e. you just get the QEMU sources, click on > > qemu.xcodeproj, Cmd + B, done. No extra installation, no configure, > > nothing. > Meson seems to have some exporter for Xcode according to > https://mesonbuild.com/IDE-integration.html ... maybe you can harness > that feature somehow? Good to know, thanks! I'll give it a try. > > The question is, and I don't have the big picture of QEMU yet to judge > > that, how much is auto generated for QEMU i.e. with custom scripts that
Re: [RFC] QEMU as Xcode project on macOS
> On 9 Sep 2020, at 20:32, Christian Schoenebeck wrote: > > And being forced to use Meson > on Mac with all its Python dependencies does not make it easier. FYI, I'm working on a cross-platform standalone meson package, a .tar.gz that you can install in any location and includes the python run-time. I already have the build scripts functional for Windows (which is generally the most difficult platform); the build scripts for Linux/macOS will follow soon. https://github.com/xpack-dev-tools/meson-build-xpack/blob/xpack-develop/scripts/common-apps-functions-source.sh Regards, Liviu
Re: [RFC] QEMU as Xcode project on macOS
> On Mittwoch, 9. September 2020 15:40:05 CEST Daniel P. Berrangé wrote: > > Ideally any xcode setup would just invoke whatever our standard > > build tools are IMHO, so it has zero possibility of introducing > > new build problems. > > Then you would not win anything on Mac. The problematic on macOS is that > Apple > froze many standard FOSS tools that switched to GPL3. So by default you just > have e.g. GNU make 3.81 (2006), Bash 3.2.57 (2007), ... unless you start to > manually install them (e.g. from Homebrew & Co). And being forced to use > Meson > on Mac with all its Python dependencies does not make it easier. It is going to need to use homebrew (or an equivalent) to bring in various 3rd party libraries QEMU depends on at build time, not least glib. So avoiding homebrew for the build system doesn't appear like an obvious win to me. I don't think QEMU wants to restrict itself to merely what is available in the basic macOS install. We expect users to have to pull in stuff from homebrew or an equivalent to get functionality that Apple has no intention of ever shipping in their base OS install. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [RFC] QEMU as Xcode project on macOS
On Mittwoch, 9. September 2020 20:13:55 CEST Daniel P. Berrangé wrote: > > On Mittwoch, 9. September 2020 15:40:05 CEST Daniel P. Berrangé wrote: > > > Ideally any xcode setup would just invoke whatever our standard > > > build tools are IMHO, so it has zero possibility of introducing > > > new build problems. > > > > Then you would not win anything on Mac. The problematic on macOS is that > > Apple froze many standard FOSS tools that switched to GPL3. So by default > > you just have e.g. GNU make 3.81 (2006), Bash 3.2.57 (2007), ... unless > > you start to manually install them (e.g. from Homebrew & Co). And being > > forced to use Meson on Mac with all its Python dependencies does not make > > it easier. > > It is going to need to use homebrew (or an equivalent) to bring in various > 3rd party libraries QEMU depends on at build time, not least glib. External libraries are not the problem. AFAICS that's just libffi, glib and pixman. These could be added as submodules. The more relevant ones would be the build tools that those QEMU scripts are running to auto generate source files. > So avoiding homebrew for the build system doesn't appear like an obvious win > to me. I would see several benefits: 1. High potential of attracting much more devs on macOS if it's just click and run. 2. Increased quality on Mac due to more devs. 3. Being able to delegate QEMU related macOS bugs to Apple. 4. *Much* more efficient work-flow when developing with the Xcode IDE (-> both increased development speed & quality). On Mittwoch, 9. September 2020 19:45:04 CEST Liviu Ionescu wrote: > FYI, I'm working on a cross-platform standalone meson package, a .tar.gz > that you can install in any location and includes the python run-time. > > I already have the build scripts functional for Windows (which is generally > the most difficult platform); the build scripts for Linux/macOS will follow > soon. > > https://github.com/xpack-dev-tools/meson-build-xpack/blob/xpack-develop/scri > pts/common-apps-functions-source.sh Interesting! That would defuse the homebrew problematic. However you would still be opted out from (4.) all those Xcode IDE features that make development such efficient. Best regards, Christian Schoenebeck
Re: [RFC] QEMU as Xcode project on macOS
> On 9 Sep 2020, at 21:56, Christian Schoenebeck wrote: > > ... However you would > still be opted out from (4.) all those Xcode IDE features that make > development such efficient. Could you elaborate? I thought that a standalone meson can be used with any environment, it just reduces the dependency on Python and possibly other system libraries, otherwise it provides the same functionality. You unpack the archive anywhere, place that location in the path and use it as usual. Why would it be opted out for Xcode? Thank you, Liviu
Re: [RFC] QEMU as Xcode project on macOS
On Wed, 9 Sep 2020 at 19:56, Christian Schoenebeck wrote: > Interesting! That would defuse the homebrew problematic. You still need a Python 3.5 or better for the other scripts QEMU has as part of its build process, so you will still need a python other than the Apple /usr/bin/python to build even if you have a standalone meson with its own python. thanks -- PMM
Re: [RFC] QEMU as Xcode project on macOS
On Mittwoch, 9. September 2020 21:03:32 CEST Liviu Ionescu wrote: > > On 9 Sep 2020, at 21:56, Christian Schoenebeck > > wrote: > > > > ... However you would > > still be opted out from (4.) all those Xcode IDE features that make > > development such efficient. > > Could you elaborate? > > I thought that a standalone meson can be used with any environment, it just > reduces the dependency on Python and possibly other system libraries, > otherwise it provides the same functionality. You unpack the archive > anywhere, place that location in the path and use it as usual. > > Why would it be opted out for Xcode? You would still not have a .xcodeproj file, which is a separate thing, as that's the only binding point for really making use of Xcode in an efficient way. Otherwise it does not have any knowledge about which source files belong to the project, what are the compiler flags, what preprocessor statements have to be applied and in which order. So you could still open source files in Xcode of course, but it would just be an ordinary plain text editor. So this is about having or not having features like: auto completion, refactoring, one-click auto fixes of trivial issues, click to jump to definition/declaration (which might be in an external lib), built-in debugger, graphical static analysis, graphical sanitizers and much more ... Best regards, Christian Schoenebeck
Re: [RFC] QEMU as Xcode project on macOS
> On 9 Sep 2020, at 22:16, Peter Maydell wrote: > > ... You still need a Python 3.5 or better for the other scripts > QEMU has as part of its build process, so you will still > need a python other than the Apple /usr/bin/python to build > even if you have a standalone meson with its own python. Ah, sure, if the build script uses Python for other purposes you'll need one, but the embedded Python will allow the standalone meson to run properly regardless of what Python you install. One less dependency to worry about is usually a good thing. Regards, Liviu
Re: [RFC] QEMU as Xcode project on macOS
On Wed, Sep 09, 2020 at 08:56:48PM +0200, Christian Schoenebeck wrote: > On Mittwoch, 9. September 2020 20:13:55 CEST Daniel P. Berrangé wrote: > > > On Mittwoch, 9. September 2020 15:40:05 CEST Daniel P. Berrangé wrote: > > > > Ideally any xcode setup would just invoke whatever our standard > > > > build tools are IMHO, so it has zero possibility of introducing > > > > new build problems. > > > > > > Then you would not win anything on Mac. The problematic on macOS is that > > > Apple froze many standard FOSS tools that switched to GPL3. So by default > > > you just have e.g. GNU make 3.81 (2006), Bash 3.2.57 (2007), ... unless > > > you start to manually install them (e.g. from Homebrew & Co). And being > > > forced to use Meson on Mac with all its Python dependencies does not make > > > it easier. > > > > It is going to need to use homebrew (or an equivalent) to bring in various > > 3rd party libraries QEMU depends on at build time, not least glib. > > External libraries are not the problem. AFAICS that's just libffi, glib and > pixman. These could be added as submodules. > > The more relevant ones would be the build tools that those QEMU scripts are > running to auto generate source files. I don't think we want to be adding more 3rd party deps as submodules, quite the opposite, we want to remove more submodules we currently have. Bundling every important dep we use as a submodule and providing build rules, means we're effectively re-inventing Homebrew / MacPorts and this is not a sane use of our time. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [RFC] QEMU as Xcode project on macOS
On 09/09/20 22:13, Liviu Ionescu wrote: >> ... You still need a Python 3.5 or better for the other scripts >> QEMU has as part of its build process, so you will still need a >> python other than the Apple /usr/bin/python to build even if you >> have a standalone meson with its own python. > > Ah, sure, if the build script uses Python for other purposes you'll > need one, but the embedded Python will allow the standalone meson to > run properly regardless of what Python you install. Note that if you use the version of Meson that is bundled with QEMU, you can use --python=/path/to/python3 --meson=internal and the build will automatically use that same Python executable to run the scripts. Paolo
Re: [RFC] QEMU as Xcode project on macOS
On Donnerstag, 10. September 2020 09:37:10 CEST Daniel P. Berrangé wrote: > I don't think we want to be adding more 3rd party deps as submodules, quite > the opposite, we want to remove more submodules we currently have. > > Bundling every important dep we use as a submodule and providing build > rules, means we're effectively re-inventing Homebrew / MacPorts and this is > not a sane use of our time. Well, that's actually the whole point of this thread: saving developers' time. And I think the submodule solution is the most sane one. Frankly if you compile FOSS software on Mac that asks you to "just" install dependencies with Homebrew and co, it feels like you have 2 jobs: a software developer, and a distribution maintainer. The difference to the submodule though: a much larger amount of developers have to do that maintainer job (manually resolving conflicts & crashes, rollbacks, etc.). Best regards, Christian Schoenebeck
Re: [RFC] QEMU as Xcode project on macOS
On Thu, Sep 10, 2020 at 11:32:35AM +0200, Christian Schoenebeck wrote: > On Donnerstag, 10. September 2020 09:37:10 CEST Daniel P. Berrangé wrote: > > I don't think we want to be adding more 3rd party deps as submodules, quite > > the opposite, we want to remove more submodules we currently have. > > > > Bundling every important dep we use as a submodule and providing build > > rules, means we're effectively re-inventing Homebrew / MacPorts and this is > > not a sane use of our time. > > Well, that's actually the whole point of this thread: saving developers' > time. > And I think the submodule solution is the most sane one. > > Frankly if you compile FOSS software on Mac that asks you to "just" install > dependencies with Homebrew and co, it feels like you have 2 jobs: a software > developer, and a distribution maintainer. The difference to the submodule > though: a much larger amount of developers have to do that maintainer job > (manually resolving conflicts & crashes, rollbacks, etc.). I don't think it saves time. If you look at the bigger picture across multiple project it costs time. Every project that depends on glib or gtk or gnutls or etc reinvents the wheel figuring out a suitable recipe for bundling & building these dependencies. Worse is that they will all do it slightly differently, or use a variety of versions, and so users get a worse experiance too with different features available and different things broken. Projects like HomeBrew exist precisely to save developer time because these build steps can be figured out once, and every project can now just rely on the well maintained HomeBrew packages. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [RFC] QEMU as Xcode project on macOS
On Donnerstag, 10. September 2020 11:39:10 CEST Daniel P. Berrangé wrote: > > Well, that's actually the whole point of this thread: saving developers' > > time. And I think the submodule solution is the most sane one. > > > > Frankly if you compile FOSS software on Mac that asks you to "just" > > install > > dependencies with Homebrew and co, it feels like you have 2 jobs: a > > software developer, and a distribution maintainer. The difference to the > > submodule though: a much larger amount of developers have to do that > > maintainer job (manually resolving conflicts & crashes, rollbacks, etc.). > > I don't think it saves time. If you look at the bigger picture across > multiple project it costs time. Every project that depends on glib or > gtk or gnutls or etc reinvents the wheel figuring out a suitable > recipe for bundling & building these dependencies. Worse is that they > will all do it slightly differently, or use a variety of versions, and > so users get a worse experiance too with different features available > and different things broken. Projects like HomeBrew exist precisely to > save developer time because these build steps can be figured out once, > and every project can now just rely on the well maintained HomeBrew > packages. That only works for consumers at most. For developers it is actually the complete opposite on Mac: you start to install things from somewhere, then you need to install something from somewhere else, manually build & install stuff, and you end up in conflicts and misbehaviours all over the place. The way to go for devs on Mac is: 3rd party libs should not be installed into global space, rather be built & linked either as dynamic frameworks (including assets) or as static libs. Then apps always run with the precise version and flags of libs they were tested with and never conflict with another app's version/config of libs. Best regards, Christian Schoenebeck
Re: [RFC] QEMU as Xcode project on macOS
> On 10 Sep 2020, at 12:20, Paolo Bonzini wrote: > > --python=/path/to/python3 --meson=internal > > and the build will automatically use that same Python executable to run > the scripts. Ok, good to know. My standalone meson is intended for integration into Eclipse and Visual Studio Code, and should also run on CI servers, so it has a different audience. Regards, Liviu
Re: [RFC] QEMU as Xcode project on macOS
> On 10 Sep 2020, at 13:14, Christian Schoenebeck > wrote: > > The way to go for devs on Mac is: 3rd party libs should not be installed into > global space, rather be built & linked either as dynamic frameworks > (including > assets) or as static libs. Then apps always run with the precise version and > flags of libs they were tested with and never conflict with another app's > version/config of libs. Fully agree. Just that I use the same concept for all my builds (Linux & Windows), not only the Mac builds. Liviu
Re: [RFC] QEMU as Xcode project on macOS
On Thu, 10 Sep 2020 at 11:14, Christian Schoenebeck wrote: > For developers it is actually the complete opposite on Mac: you start to > install things from somewhere, then you need to install something from > somewhere else, manually build & install stuff, and you end up in conflicts > and misbehaviours all over the place. This has not been my experience -- homebrew has everything, and it doesn't have problems like this. > The way to go for devs on Mac is: 3rd party libs should not be installed into > global space, rather be built & linked either as dynamic frameworks (including > assets) or as static libs. Then apps always run with the precise version and > flags of libs they were tested with and never conflict with another app's > version/config of libs. Does Apple (or anybody else) provide a framework for doing this so that developers of individual applications can just say "my app needs libs X, Y, Z" and they don't have to mess around finding, figuring out how to compile, and shipping the sources of libs X, Y, Z? If there's a better mechanism than Homebrew for this that's great, but at the moment what you seem to be saying is "you should do a lot more work to manually set something up where you ship the sources to all your dependencies and then build them all". There's no way we're ever going to do that, it is just way too much work for very little gain. thanks -- PMM
Re: [RFC] QEMU as Xcode project on macOS
Il gio 10 set 2020, 11:32 Christian Schoenebeck ha scritto: > On Donnerstag, 10. September 2020 09:37:10 CEST Daniel P. Berrangé wrote: > > I don't think we want to be adding more 3rd party deps as submodules, > quite > > the opposite, we want to remove more submodules we currently have. > > > > Bundling every important dep we use as a submodule and providing build > > rules, means we're effectively re-inventing Homebrew / MacPorts and this > is > > not a sane use of our time. > > Well, that's actually the whole point of this thread: saving developers' > time. > And I think the submodule solution is the most sane one. > You're very welcome to do this, but what you're doing is effectively a *distribution of QEMU* that targets the "macOS build from source" scenario, for people that don't want to use Homebrew. It is *not* going to be used by QEMU developers, because QEMU developers have conflicting requirements: * Not getting in the business of maintaining a distribution of all their dependencies (glib, pixman, etc.) * Not wanting to maintain multiple build systems both of which are non-negotiable. What you might do is use the configure script and the ninja backend to build all generated sources (produced by either configure, meson or make); then copy those generated file over to a new build directory, invoke meson again with the xcode backend, and distribute the result so that it is ready to build from source using xcode. The resulting distribution is not appropriate to develop QEMU, but it would be okay to install it and even for some simple debugging. The above approach still doesn't solve the problem of building glib and friends as part of the same xcode project. Perhaps this can be fixed by patching the xcodeproj that Meson generated. The scripts needed to do so should be relatively stable and can certainly be included in the upstream QEMU sources. You can't expect that other people will test them unless you also include them somehow in our CI, but just having the scripts would be a start. Thanks, Paolo > Frankly if you compile FOSS software on Mac that asks you to "just" > install > dependencies with Homebrew and co, it feels like you have 2 jobs: a > software > developer, and a distribution maintainer. The difference to the submodule > though: a much larger amount of developers have to do that maintainer job > (manually resolving conflicts & crashes, rollbacks, etc.). > > Best regards, > Christian Schoenebeck > > >
Re: [RFC] QEMU as Xcode project on macOS
On Thu, Sep 10, 2020 at 11:35:46AM +0100, Peter Maydell wrote: > On Thu, 10 Sep 2020 at 11:14, Christian Schoenebeck > wrote: > > For developers it is actually the complete opposite on Mac: you start to > > install things from somewhere, then you need to install something from > > somewhere else, manually build & install stuff, and you end up in conflicts > > and misbehaviours all over the place. > > This has not been my experience -- homebrew has everything, and > it doesn't have problems like this. > > > The way to go for devs on Mac is: 3rd party libs should not be installed > > into > > global space, rather be built & linked either as dynamic frameworks > > (including > > assets) or as static libs. Then apps always run with the precise version and > > flags of libs they were tested with and never conflict with another app's > > version/config of libs. > > Does Apple (or anybody else) provide a framework for doing this > so that developers of individual applications can just say "my > app needs libs X, Y, Z" and they don't have to mess around > finding, figuring out how to compile, and shipping the sources of > libs X, Y, Z? If there's a better mechanism than Homebrew for this > that's great, but at the moment what you seem to be saying is > "you should do a lot more work to manually set something up where > you ship the sources to all your dependencies and then build them > all". There's no way we're ever going to do that, it is just > way too much work for very little gain. AFAICT both MacPorts and HomeBrew can be installed into custom locations, at least if you do a "from source" install of them, rather than using the pre-built packages. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [RFC] QEMU as Xcode project on macOS
On Thu, Sep 10, 2020 at 12:41:45PM +0200, Paolo Bonzini wrote: > Il gio 10 set 2020, 11:32 Christian Schoenebeck ha > scritto: > > > On Donnerstag, 10. September 2020 09:37:10 CEST Daniel P. Berrangé wrote: > > > I don't think we want to be adding more 3rd party deps as submodules, > > quite > > > the opposite, we want to remove more submodules we currently have. > > > > > > Bundling every important dep we use as a submodule and providing build > > > rules, means we're effectively re-inventing Homebrew / MacPorts and this > > is > > > not a sane use of our time. > > > > Well, that's actually the whole point of this thread: saving developers' > > time. > > And I think the submodule solution is the most sane one. > > > > You're very welcome to do this, but what you're doing is effectively a > *distribution of QEMU* that targets the "macOS build from source" scenario, > for people that don't want to use Homebrew. It is *not* going to be used by > QEMU developers, because QEMU developers have conflicting requirements: > > * Not getting in the business of maintaining a distribution of all their > dependencies (glib, pixman, etc.) > > * Not wanting to maintain multiple build systems > > both of which are non-negotiable. > > What you might do is use the configure script and the ninja backend to > build all generated sources (produced by either configure, meson or make); > then copy those generated file over to a new build directory, invoke meson > again with the xcode backend, and distribute the result so that it is ready > to build from source using xcode. The resulting distribution is not > appropriate to develop QEMU, but it would be okay to install it and even > for some simple debugging. > > The above approach still doesn't solve the problem of building glib and > friends as part of the same xcode project. Perhaps this can be fixed by > patching the xcodeproj that Meson generated. > > The scripts needed to do so should be relatively stable and can certainly > be included in the upstream QEMU sources. You can't expect that other > people will test them unless you also include them somehow in our CI, but > just having the scripts would be a start. I'm not convinced we want to take what amounts to distribution packaging into the main QEMU repo, as QEMU has generally stayed out of this area leaving it to be done as downstream users/projects desire. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [RFC] QEMU as Xcode project on macOS
> On 10 Sep 2020, at 13:45, Daniel P. Berrangé wrote: > > AFAICT both MacPorts and HomeBrew can be installed into custom locations, > at least if you do a "from source" install of them, rather than using the > pre-built packages. Yes, I use this for HomeBrew, but generally you cannot select the individual versions for libraries and tools, the Ruby code used to build packages is in a Git, you can checkout a moment in time, but you get the selection of versions at that moment. If you try using individual Ruby files for different packages, you generally run into dependency issues. Regards, Liviu
Re: [RFC] QEMU as Xcode project on macOS
On Donnerstag, 10. September 2020 12:35:46 CEST Peter Maydell wrote: > Does Apple (or anybody else) provide a framework for doing this > so that developers of individual applications can just say "my > app needs libs X, Y, Z" and they don't have to mess around > finding, figuring out how to compile, and shipping the sources of > libs X, Y, Z? If there's a better mechanism than Homebrew for this > that's great, but at the moment what you seem to be saying is > "you should do a lot more work to manually set something up where > you ship the sources to all your dependencies and then build them > all". There's no way we're ever going to do that, it is just > way too much work for very little gain. On Mac I have 'framework'ed & Xcoded the FOSS libs I need. And when I need to compile a new FOSS lib/app that requires libs x,y,z then I just drag those already existing Xcode projects as referenced subprojects into that new Xcode project: https://developer.apple.com/library/archive/documentation/ToolsLanguages/ Conceptual/Xcode_Overview/WorkingonRelatedProjects.html For instance if I have to compile some new app that needs gtk(mm), all I need to do is pulling in gtkmm.xcodeproj (e.g. per drag & drop) which in turn already references all its own dependencies (subsubsub..projects) like glib, cairo, etc. So I just click 'Run' and that's it. And if the app triggers a crash in some of the subprojects, Xcode stops at the precise lib's source location, you fix it, click 'Run' and you're done in seconds. With the regular 'make' approach you would spend hours for that. So that's a highly modular end-to-end-from-source approach. The precise location of the individual xcode projects does not matter BTW as Xcode references subprojects by UDID. You can move them around as you want and it still works. No need to install them at global space, predefined structures, or even hard coded locations, and no duplication of sources either. I know, all this sounds very unorthodox what I am doing at first glance. But if you think about it, all it would take is if individual FOSS projects would start maintaining an .xcodeproj file, which in turn could then be referenced by UUID from other FOSS project's .xcodeproj file as dependency and you would get a low cost end-to-end-by-source solution, already on upstream level, i.e. without the need of having dedicated projects managing "distributions". On Donnerstag, 10. September 2020 12:41:45 CEST Paolo Bonzini wrote: > You're very welcome to do this, but what you're doing is effectively a > *distribution of QEMU* that targets the "macOS build from source" scenario, > for people that don't want to use Homebrew. It is *not* going to be used by > QEMU developers, because QEMU developers have conflicting requirements: Okay, no problem. I will see where to go from here. I am not even sure yet whether all those source generating scripts of QEMU would be a show stopper for what I have planned. I mean (counting): ... yes, still just 2 hands here. If it turns out to work fine, then maybe I just push a branch of QEMU for Mac developers somewhere for a while, as I think it would lower the entry level for new developers on Mac tremendously. Apparently there is still more changes with Meson ahead anyway (getting rid of the Makefiles or whatever else is planned). So we'll see ... Best regards, Christian Schoenebeck
Re: [RFC] QEMU as Xcode project on macOS
On 10/09/20 16:40, Christian Schoenebeck wrote: > If it turns out to work fine, then maybe I just push a branch of QEMU for Mac > developers somewhere for a while, as I think it would lower the entry level > for new developers on Mac tremendously. Apparently there is still more > changes > with Meson ahead anyway (getting rid of the Makefiles or whatever else is > planned). So we'll see ... If there are no submodule dependencies (capstone/libfdt/slirp), effectively you can already behave as if the Makefiles did not exist and build QEMU with ninja and "meson test". So you could build your scripts from there. Paolo
Re: [RFC] QEMU as Xcode project on macOS
On Freitag, 11. September 2020 19:19:05 CEST Paolo Bonzini wrote: > On 10/09/20 16:40, Christian Schoenebeck wrote: > > If it turns out to work fine, then maybe I just push a branch of QEMU for > > Mac developers somewhere for a while, as I think it would lower the entry > > level for new developers on Mac tremendously. Apparently there is still > > more changes with Meson ahead anyway (getting rid of the Makefiles or > > whatever else is planned). So we'll see ... > > If there are no submodule dependencies (capstone/libfdt/slirp), > effectively you can already behave as if the Makefiles did not exist and > build QEMU with ninja and "meson test". So you could build your scripts > from there. > > Paolo Good to know, thanks! Best regards, Christian Schoenebeck