Provisional snap for DUB (D language package/build manager)
Hello folks, Following my attempt to create a snap for LDC (LLVM-based compiler for the D programming language), I thought I'd have a go at another D-related project and snap DUB. This is a package/build manager that's popular in the D community, and so having it available as a snap could be very useful. First caveats: this is a command-line developer tool, so some of the same limitations are going to apply as were identified with the LDC snap (access to system tools, linking against host-system development libraries, etc.). I also had to take some temporary shortcuts to ensure that the packaged DUB had a D compiler available. The draft package is available here: https://github.com/WebDrake/dub.snap/pull/1/files A few things on what went into it, and corresponding requests for feedback. First: DUB can be built two ways; either by calling a shell script, `build.sh`, or by DUB itself via an existing install. I couldn't identify an obvious way to handle the former, so (given that DUB is packaged in Ubuntu 16.04) I opted to create a `dub.py` snapcraft plugin. I'm not a Pythonist myself, so any feedback on that code would be welcome. But I have a couple of other questions: * Is there any way for the plugin to ask for a `dub` instance to be available? Currently I'm just specifying `dub` as a build dependency in addition to the plugin. * Assuming I'd wanted to go the `build.sh` route, is there any way I could have achieved that with existing plugins? DUB doesn't have an `install` option, only a `build` one, which creates some problems in terms of determining what files go into the snap. I compromised on a short term workaround where the plugin copies everything in the `build/` dir of the part being built, and the user is expected to manually specify what bits of that should actually wind up in the final snap. If there is a target path where the built files are placed, the plugin can handle that, too. Two questions there, too: * Is there any way to filter that stuff out already at the staging area or earlier? I tried using `filesets:` but didn't have much luck. * Is there any way to detect what extra files have been created after the build completes and just use those? Since DUB needs a D compiler to be able to build anything, I first tried to just use the existing Ubuntu `ldc` package, and then reverted to copying the entirety of my LDC snap setup because it proved simpler to get working. That's very much a temporary measure, I hope, but again, questions here: * Is there any way for me to give the DUB snap access to the ldc2 and/or ldmd2 installed in my LDC snap package, other than explicitly defining a `d-compiler` interface and submitting it to `snapd`? * Is there any way of defining/using/testing an interface short of building my own local snapd? The last issue is a bit weird: DUB installed by the current snap package can build D projects, but exits with an error: "Bad System Call". It's been difficult to track down what was going on here (`strace dub build` for example was not very edifying, presumably because of the containerization) so any advice on how to identify what's wrong? At a guess, it's related to the step where the built binaries are made executable, because they get created, but are _not_ executable at the end of the process. This is a bit of a surprise, because the standalone LDC snap doesn't have this issue (things get made executable just fine). I did try making ldc2 and ldmd2 apps of the DUB snap too, but that seemed to make no difference. OTOH that might be related to how the snap-packaged DUB would invoke the D compiler (because presumably you'd need to call `dub.ldc2` in order to ensure that you get the `home` plug working properly, while DUB probably calls `ldc2` directly). Anyway, any thoughts on what could be going on here? Thanks in advance for any advice or thoughts, and best wishes, -- Joe -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
Le 27/10/2016 à 01:13, Joseph Rushton Wakeling a écrit : > Hello folks, Hey Joseph, It would be great to have D support! Thanks for working on that :) I'm going to try to answer to some of your questions, but I'm sure Sergio (CCed) would be able to give deeper insights on some parts as being the snapcraft upstream. > > Following my attempt to create a snap for LDC (LLVM-based compiler for > the D programming language), I thought I'd have a go at another > D-related project and snap DUB. This is a package/build manager > that's popular in the D community, and so having it available as a > snap could be very useful. > > First caveats: this is a command-line developer tool, so some of the > same limitations are going to apply as were identified with the LDC > snap (access to system tools, linking against host-system development > libraries, etc.). I also had to take some temporary shortcuts to > ensure that the packaged DUB had a D compiler available. > > The draft package is available here: > https://github.com/WebDrake/dub.snap/pull/1/files > > A few things on what went into it, and corresponding requests for > feedback. > > First: DUB can be built two ways; either by calling a shell script, > `build.sh`, or by DUB itself via an existing install. I couldn't > identify an obvious way to handle the former, so (given that DUB is > packaged in Ubuntu 16.04) I opted to create a `dub.py` snapcraft plugin. > > I'm not a Pythonist myself, so any feedback on that code would be > welcome. But I have a couple of other questions: > > * Is there any way for the plugin to ask for a `dub` instance to be > available? > Currently I'm just specifying `dub` as a build dependency in > addition to the > plugin. > > * Assuming I'd wanted to go the `build.sh` route, is there any way I > could > have achieved that with existing plugins? I would really advise going the build.sh route. Doing a custom snapcraft plugin as you started is definitively the right path. If you take other plugins having similar problematic, like Go, here is how they work: - Golang binaries (compiler tools and standard library) installs itself in parts//go. This logic is from the golang plugin itself. You will need in your case to ensure LDC is relocatable and can be downloaded and extracted this way. You can then build without depending on any package or distribution. - Then, the plugin run go build, go install, changing the paths. The plugin can define optional parameters that it will use then. In you case, I think the plugin will setup environment variables to refers to local LDC and standard D libraries. Then, you can have one option being "entry-point", which will refers in your DUB snap case to build.sh. The plugin would execute it. > > DUB doesn't have an `install` option, only a `build` one, which > creates some problems in terms of determining what files go into the > snap. I compromised on a short term workaround where the plugin > copies everything in the `build/` dir of the part being built, and the > user is expected to manually specify what bits of that should actually > wind up in the final snap. If there is a target path where the built > files are placed, the plugin can handle that, too. > > Two questions there, too: > > * Is there any way to filter that stuff out already at the staging area > or earlier? I tried using `filesets:` but didn't have much luck. > > * Is there any way to detect what extra files have been created > after the > build completes and just use those? >From your questions and the statement that there is no explicit install steps, I think you mean that D programs just build in tree (snapcraft tries to encourage builds out of trees) and then, the install step for people is to copy the resulting binaries and assets manually? If so, I can see 2 paths here: - either you expect people to explicitely list assets and binaries that should be installed (this could again be a custom option of your plugin) - either you try to be starter, as you told, and detect extra files being created in the parts//build. There is no built-in features in snapcraft for this as far as I know, but, it would be quite easy to build it as part of your plugin: list at the start of the build phase files in parts//build and redo the same thing after the build. Then, in the install phase, use that list (that you store on disk) to copy newly created content in parts//install. Then, snapcraft will pick it up from there. Note that I think you still need a manual options for your users to list eventual assets (or they can use the dump plugin in another part). Just pick the flow which would be the more natural to the D plugin users. If you prefer to copy everything and have your users filtering from the stage area to the prime area, you can use the "snap:" stenza (don't use filesets for now if you don't plan to repeat the file list name). > > Since DUB needs a D compiler to be able to build anything, I first > tried
Re: Provisional snap for DUB (D language package/build manager)
Le 27/10/2016 à 08:37, Didier Roche a écrit : > Le 27/10/2016 à 01:13, Joseph Rushton Wakeling a écrit : >> Hello folks, > Hey Joseph, > > It would be great to have D support! Thanks for working on that :) > I'm going to try to answer to some of your questions, but I'm sure > Sergio (CCed) would be able to give deeper insights on some parts as > being the snapcraft upstream. >> Following my attempt to create a snap for LDC (LLVM-based compiler for >> the D programming language), I thought I'd have a go at another >> D-related project and snap DUB. This is a package/build manager >> that's popular in the D community, and so having it available as a >> snap could be very useful. >> >> First caveats: this is a command-line developer tool, so some of the >> same limitations are going to apply as were identified with the LDC >> snap (access to system tools, linking against host-system development >> libraries, etc.). I also had to take some temporary shortcuts to >> ensure that the packaged DUB had a D compiler available. >> >> The draft package is available here: >> https://github.com/WebDrake/dub.snap/pull/1/files >> >> A few things on what went into it, and corresponding requests for >> feedback. >> >> First: DUB can be built two ways; either by calling a shell script, >> `build.sh`, or by DUB itself via an existing install. I couldn't >> identify an obvious way to handle the former, so (given that DUB is >> packaged in Ubuntu 16.04) I opted to create a `dub.py` snapcraft plugin. >> >> I'm not a Pythonist myself, so any feedback on that code would be >> welcome. But I have a couple of other questions: >> >> * Is there any way for the plugin to ask for a `dub` instance to be >> available? >> Currently I'm just specifying `dub` as a build dependency in >> addition to the >> plugin. >> >> * Assuming I'd wanted to go the `build.sh` route, is there any way I >> could >> have achieved that with existing plugins? > I would really advise going the build.sh route. Doing a custom snapcraft > plugin as you started is definitively the right path. > If you take other plugins having similar problematic, like Go, here is > how they work: > - Golang binaries (compiler tools and standard library) installs itself > in parts//go. This logic is from the golang plugin itself. > You will need in your case to ensure LDC is relocatable and can be > downloaded and extracted this way. You can then build without depending > on any package or distribution. > - Then, the plugin run go build, go install, changing the paths. The > plugin can define optional parameters that it will use then. In you > case, I think the plugin will setup environment variables to refers to > local LDC and standard D libraries. Then, you can have one option being > "entry-point", which will refers in your DUB snap case to build.sh. The > plugin would execute it. Just a note: after a quick double checking , the snapcraft go plugin relies on ubuntu packages (I think it should rather download latest stable go and use GOROOT to use it). This isn't the case for other plugins like nodejs which was the one I had in mind and I still think you should prefer that road. Didier -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
On 27/10/16 08:37, Didier Roche wrote: It would be great to have D support! Thanks for working on that :) Great to hear :-) Just to be clear, it's already in principle possible to package D applications as snaps; DUB support isn't needed, but is very helpful, because a lot of D projects do use it. I'm going to try to answer to some of your questions, but I'm sure Sergio (CCed) would be able to give deeper insights on some parts as being the snapcraft upstream. Thanks very much. I'll respond to some of your suggestions below ... I would really advise going the build.sh route. Doing a custom snapcraft plugin as you started is definitively the right path. Yes, agreed. I was curious about the `build.sh` option not because I think it's a good idea, but because I was thinking: "What if it was the only build option for this project?" I'd like to know how to handle that situation if I ever encounter it in another project. From your questions and the statement that there is no explicit install steps, I think you mean that D programs just build in tree (snapcraft tries to encourage builds out of trees) and then, the install step for people is to copy the resulting binaries and assets manually? Yes, exactly. It's not _entirely_ as bad as it sounds because each project's `dub.json` config file can define a `targetPath` variable where built programs will be placed. The plugin can use that if available. If so, I can see 2 paths here: - either you expect people to explicitely list assets and binaries that should be installed (this could again be a custom option of your plugin) This is pretty much the direction I've taken for now. - either you try to be starter, as you told, and detect extra files being created in the parts//build. There is no built-in features in snapcraft for this as far as I know, but, it would be quite easy to build it as part of your plugin: list at the start of the build phase files in parts//build and redo the same thing after the build. Then, in the install phase, use that list (that you store on disk) to copy newly created content in parts//install. Then, snapcraft will pick it up from there. Note that I think you still need a manual options for your users to list eventual assets (or they can use the dump plugin in another part). Nice thought. I'll try to look into that. There is a 3rd option which could be for the plugin to read the `dub.json` config file of the project being built, and parse that to discover the targets that will be built, but honestly that feels like duplication of effort, and it would probably be better to encourage the DUB developers to provide an `install` option. If you prefer to copy everything and have your users filtering from the stage area to the prime area, you can use the "snap:" stenza (don't use filesets for now if you don't plan to repeat the file list name). Yup, this is the option I use now. I would advise in a first step to consider the D compiler used by DUB to build other things to be part of your snap. You want to control the version in it and consider it as part of your dependencies you need in your snap. We can discuss later about an interface, but I would really go that way as a first step. Yup, bundling the compiler into the snap is the way I have gone as a first step. However, the real issue is that DUB is supposed to provide the user with the option to choose the D compiler to use. It's tolerable as a first step to just give one option bundled, but ideally the user would be able to invoke DUB and specify to it any compiler they like -- whether a snap-packaged one, or one installed on the host system. I was assuming that allowing the user to (optionally) specify a compiler of choice via an interface would be easier to achieve, but I don't know if that's true. There is no way right now on definining new interfaces apart from building them in snapd, and so, use your own local version. (That's the reason why I advise you to not bother with this right now, let's take an easier path first, figure the rest, and then, we can rediscuss) Well, I already took the easier path, so that's why I raised the discussion of alternatives ... :-) I would look at /var/log/syslogs. Apparmor and seccomp denials are listed there. Note that if you didn't already, you should really start developping your snap in devmode. That way, it will get confinment out of the equasion to get your relocatable code and dependencies working. Then, we can turn on confinement and figure out those issues to be able to publish in the stable channel. Yea, I probably should have started with devmode. Thanks for the advice about syslogs; I'll check it out and see what I can find. Don't make your snap calling other snap endpoints. It's better from your own snap code to ensure that your local ldc2 is in your snap $PATH, and call ldc2 directly (it will then use the parent process confinment profile, of course). I think that
Re: Provisional snap for DUB (D language package/build manager)
El 27/10/16 a las 05:27, Didier Roche escribió: Just a note: after a quick double checking , the snapcraft go plugin relies on ubuntu packages (I think it should rather download latest stable go and use GOROOT to use it). This isn't the case for other plugins like nodejs which was the one I had in mind and I still think you should prefer that road. It used to be the other way around, this is going to be solved with a combination of: - using upstream go releases when on the supported arch - enable cross compilation for go (when no cgo is used initially) - allow parts to 'provide' things for plugins (this solves much more than just the go story, but would also make the ros, python, java, maven, ant and other stories much nicer). -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
El 27/10/16 a las 17:13, Joseph Rushton Wakeling escribió: On 27/10/16 08:37, Didier Roche wrote: It would be great to have D support! Thanks for working on that :) Great to hear :-) Just to be clear, it's already in principle possible to package D applications as snaps; DUB support isn't needed, but is very helpful, because a lot of D projects do use it. I'm going to try to answer to some of your questions, but I'm sure Sergio (CCed) would be able to give deeper insights on some parts as being the snapcraft upstream. Thanks very much. I'll respond to some of your suggestions below ... I would really advise going the build.sh route. Doing a custom snapcraft plugin as you started is definitively the right path. Yes, agreed. I was curious about the `build.sh` option not because I think it's a good idea, but because I was thinking: "What if it was the only build option for this project?" I'd like to know how to handle that situation if I ever encounter it in another project. From your questions and the statement that there is no explicit install steps, I think you mean that D programs just build in tree (snapcraft tries to encourage builds out of trees) and then, the install step for people is to copy the resulting binaries and assets manually? Yes, exactly. It's not _entirely_ as bad as it sounds because each project's `dub.json` config file can define a `targetPath` variable where built programs will be placed. The plugin can use that if available. If so, I can see 2 paths here: - either you expect people to explicitely list assets and binaries that should be installed (this could again be a custom option of your plugin) This is pretty much the direction I've taken for now. The make plugin has an `artifacts` option (if you inherit from it you get it for free). If necessary we can move this to the base plugin or even as a core capability. Used for projects that are older or do not follow GNU conventions of having an `install` target. - either you try to be starter, as you told, and detect extra files being created in the parts//build. There is no built-in features in snapcraft for this as far as I know, but, it would be quite easy to build it as part of your plugin: list at the start of the build phase files in parts//build and redo the same thing after the build. Then, in the install phase, use that list (that you store on disk) to copy newly created content in parts//install. Then, snapcraft will pick it up from there. Note that I think you still need a manual options for your users to list eventual assets (or they can use the dump plugin in another part). Nice thought. I'll try to look into that. There is a 3rd option which could be for the plugin to read the `dub.json` config file of the project being built, and parse that to discover the targets that will be built, but honestly that feels like duplication of effort, and it would probably be better to encourage the DUB developers to provide an `install` option. If you prefer to copy everything and have your users filtering from the stage area to the prime area, you can use the "snap:" stenza (don't use filesets for now if you don't plan to repeat the file list name). Yup, this is the option I use now. I would advise in a first step to consider the D compiler used by DUB to build other things to be part of your snap. You want to control the version in it and consider it as part of your dependencies you need in your snap. We can discuss later about an interface, but I would really go that way as a first step. Yup, bundling the compiler into the snap is the way I have gone as a first step. However, the real issue is that DUB is supposed to provide the user with the option to choose the D compiler to use. It's tolerable as a first step to just give one option bundled, but ideally the user would be able to invoke DUB and specify to it any compiler they like -- whether a snap-packaged one, or one installed on the host system. I was assuming that allowing the user to (optionally) specify a compiler of choice via an interface would be easier to achieve, but I don't know if that's true. There is no way right now on definining new interfaces apart from building them in snapd, and so, use your own local version. (That's the reason why I advise you to not bother with this right now, let's take an easier path first, figure the rest, and then, we can rediscuss) Well, I already took the easier path, so that's why I raised the discussion of alternatives ... :-) I would look at /var/log/syslogs. Apparmor and seccomp denials are listed there. Note that if you didn't already, you should really start developping your snap in devmode. That way, it will get confinment out of the equasion to get your relocatable code and dependencies working. Then, we can turn on confinement and figure out those issues to be able to publish in the stable channel. Yea, I probably should have started with d
Re: Provisional snap for DUB (D language package/build manager)
On 27/10/16 10:27, Didier Roche wrote: Just a note: after a quick double checking , the snapcraft go plugin relies on ubuntu packages (I think it should rather download latest stable go and use GOROOT to use it). This isn't the case for other plugins like nodejs which was the one I had in mind and I still think you should prefer that road. In principle that should be possible, as there are precompiled downloads of dub available here: https://code.dlang.org/download Using these could have some advantages, including allowing the user of the plugin to specify the exact version of dub they want to build with. Since there are different Linux downloads for x86, x86-64 and ARM, is there any way for the snapcract plugin to detect what the host system is and download accordingly ... ? -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
On 10/29/2016 08:14 AM, Joseph Rushton Wakeling wrote: > On 27/10/16 10:27, Didier Roche wrote: >> Just a note: after a quick double checking , the snapcraft go plugin >> relies on ubuntu packages (I think it should rather download latest >> stable go and use GOROOT to use it). This isn't the case for other >> plugins like nodejs which was the one I had in mind and I still think >> you should prefer that road. > > In principle that should be possible, as there are precompiled downloads > of dub available here: https://code.dlang.org/download Using these > could have some advantages, including allowing the user of the plugin to > specify the exact version of dub they want to build with. > > Since there are different Linux downloads for x86, x86-64 and ARM, is > there any way for the snapcract plugin to detect what the host system is > and download accordingly ... ? Indeed, you can utilize the `self.project.arch_triplet` property for that. -- Kyle Fazzari (kyrofa) Software Engineer Canonical Ltd. k...@canonical.com signature.asc Description: OpenPGP digital signature -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
El 31 oct. 2016 6:25 PM, "Kyle Fazzari" escribió: > > > > On 10/29/2016 08:14 AM, Joseph Rushton Wakeling wrote: > > On 27/10/16 10:27, Didier Roche wrote: > >> Just a note: after a quick double checking , the snapcraft go plugin > >> relies on ubuntu packages (I think it should rather download latest > >> stable go and use GOROOT to use it). This isn't the case for other > >> plugins like nodejs which was the one I had in mind and I still think > >> you should prefer that road. > > > > In principle that should be possible, as there are precompiled downloads > > of dub available here: https://code.dlang.org/download Using these > > could have some advantages, including allowing the user of the plugin to > > specify the exact version of dub they want to build with. > > > > Since there are different Linux downloads for x86, x86-64 and ARM, is > > there any way for the snapcract plugin to detect what the host system is > > and download accordingly ... ? > > Indeed, you can utilize the `self.project.arch_triplet` property for that. The nodejs plugin actually used this construct if you need a reference. -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
On 27/10/16 22:13, Joseph Rushton Wakeling wrote: On 27/10/16 08:37, Didier Roche wrote: I would look at /var/log/syslogs. Apparmor and seccomp denials are listed there. Note that if you didn't already, you should really start developping your snap in devmode. That way, it will get confinment out of the equasion to get your relocatable code and dependencies working. Then, we can turn on confinement and figure out those issues to be able to publish in the stable channel. Yea, I probably should have started with devmode. Thanks for the advice about syslogs; I'll check it out and see what I can find. OK, so it looks like apparmor was indeed responsible. The loglines in question: Oct 30 17:50:50 computername kernel: [ 9532.992875] audit: type=1400 audit(1477846250.853:43): apparmor="DENIED" operation="link" profile="snap.dub.dub" name="/home/username/code/D/dgraph/build/dgraph_graphtest" pid=22464 comm="dub" requested_mask="l" denied_mask="l" fsuid=1000 ouid=1000 target="/home/username/code/D/dgraph/.dub/build/application-debug-linux.posix-x86_64-ldc_0-B7AFC7F4AA486AA98C5445F91F5653DB/dgraph_graphtest" Oct 30 17:50:50 computername kernel: [ 9533.035303] audit: type=1326 audit(1477846250.897:44): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=22464 comm="dub" exe="/snap/dub/x1/bin/dub" sig=31 arch=c03e syscall=92 compat=0 ip=0x7f9b72d13717 code=0x0 I'm not experienced with apparmor, so could someone explain exactly what this means? (I get the general idea, but the specifics would be useful to understand precisely.) In particular, is there an obvious reason why this might be showing up with the dub snap, when the earlier ldc2 snap didn't have this problem? I would guess because the ldc2 instance used by the snap-packaged dub is internal to the snap and does not benefit from the home-directory interface that dub itself gets? Setting the containment to devmode removes the problem, but it would be nice to be able to have strict confinement earlier rather than later. -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
El 01/11/16 a las 23:31, Joseph Rushton Wakeling escribió: On 27/10/16 22:13, Joseph Rushton Wakeling wrote: On 27/10/16 08:37, Didier Roche wrote: I would look at /var/log/syslogs. Apparmor and seccomp denials are listed there. Note that if you didn't already, you should really start developping your snap in devmode. That way, it will get confinment out of the equasion to get your relocatable code and dependencies working. Then, we can turn on confinement and figure out those issues to be able to publish in the stable channel. Yea, I probably should have started with devmode. Thanks for the advice about syslogs; I'll check it out and see what I can find. OK, so it looks like apparmor was indeed responsible. The loglines in question: Oct 30 17:50:50 computername kernel: [ 9532.992875] audit: type=1400 audit(1477846250.853:43): apparmor="DENIED" operation="link" profile="snap.dub.dub" name="/home/username/code/D/dgraph/build/dgraph_graphtest" pid=22464 comm="dub" requested_mask="l" denied_mask="l" fsuid=1000 ouid=1000 target="/home/username/code/D/dgraph/.dub/build/application-debug-linux.posix-x86_64-ldc_0-B7AFC7F4AA486AA98C5445F91F5653DB/dgraph_graphtest" Oct 30 17:50:50 computername kernel: [ 9533.035303] audit: type=1326 audit(1477846250.897:44): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=22464 comm="dub" exe="/snap/dub/x1/bin/dub" sig=31 arch=c03e syscall=92 compat=0 ip=0x7f9b72d13717 code=0x0 I'm not experienced with apparmor, so could someone explain exactly what this means? (I get the general idea, but the specifics would be useful to understand precisely.) If this is x86_64, everything is aligned with the world, syscall 92 is chown. A useful tool here can help you out, and luckily there is one, run `snap install snappy-debug` and it will do some nice things to figure out what is going on wth these apparmor and seccomp blockers. In particular, is there an obvious reason why this might be showing up with the dub snap, when the earlier ldc2 snap didn't have this problem? I would guess because the ldc2 instance used by the snap-packaged dub is internal to the snap and does not benefit from the home-directory interface that dub itself gets? It seems to be just a dub problem. Setting the containment to devmode removes the problem, but it would be nice to be able to have strict confinement earlier rather than later. If this is the problem and you can patch the software then removing the chown could work, I am CCing Jamie for other ideas that could come up. -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
On 01/11/16 22:43, Sergio Schvezov wrote: If this is x86_64, everything is aligned with the world, syscall 92 is chown. A useful tool here can help you out, and luckily there is one, run `snap install snappy-debug` and it will do some nice things to figure out what is going on wth these apparmor and seccomp blockers. Cheers, I'll try that out. Will probably be a little while before I follow up -- I'm going to be away from computer for the next couple of weeks. If this is the problem and you can patch the software then removing the chown could work, I am CCing Jamie for other ideas that could come up. In principle might be possible. Thanks very much for the help & advice :-) -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
Regards, Jamie. > On 1 Nov 2016, at 23:43, Sergio Schvezov > wrote: > > > >> El 01/11/16 a las 23:31, Joseph Rushton Wakeling escribió: >>> On 27/10/16 22:13, Joseph Rushton Wakeling wrote: On 27/10/16 08:37, Didier Roche wrote: I would look at /var/log/syslogs. Apparmor and seccomp denials are listed there. Note that if you didn't already, you should really start developping your snap in devmode. That way, it will get confinment out of the equasion to get your relocatable code and dependencies working. Then, we can turn on confinement and figure out those issues to be able to publish in the stable channel. >>> >>> Yea, I probably should have started with devmode. Thanks for the advice >>> about >>> syslogs; I'll check it out and see what I can find. >> >> OK, so it looks like apparmor was indeed responsible. The loglines in >> question: >> >> Oct 30 17:50:50 computername kernel: [ 9532.992875] audit: type=1400 >> audit(1477846250.853:43): apparmor="DENIED" operation="link" >> profile="snap.dub.dub" >> name="/home/username/code/D/dgraph/build/dgraph_graphtest" pid=22464 >> comm="dub" requested_mask="l" denied_mask="l" fsuid=1000 ouid=1000 >> target="/home/username/code/D/dgraph/.dub/build/application-debug-linux.posix-x86_64-ldc_0-B7AFC7F4AA486AA98C5445F91F5653DB/dgraph_graphtest" >> Oct 30 17:50:50 computername kernel: [ 9533.035303] audit: type=1326 >> audit(1477846250.897:44): auid=4294967295 uid=1000 gid=1000 ses=4294967295 >> pid=22464 comm="dub" exe="/snap/dub/x1/bin/dub" sig=31 arch=c03e >> syscall=92 compat=0 ip=0x7f9b72d13717 code=0x0 >> >> I'm not experienced with apparmor, so could someone explain exactly what >> this means? (I get the general idea, but the specifics would be useful to >> understand precisely.) > > If this is x86_64, everything is aligned with the world, syscall 92 is chown. > A useful tool here can help you out, and luckily there is one, run `snap > install snappy-debug` and it will do some nice things to figure out what is > going on wth these apparmor and seccomp blockers. > >> >> In particular, is there an obvious reason why this might be showing up with >> the dub snap, when the earlier ldc2 snap didn't have this problem? I would >> guess because the ldc2 instance used by the snap-packaged dub is internal to >> the snap and does not benefit from the home-directory interface that dub >> itself gets? > It seems to be just a dub problem. > >> >> Setting the containment to devmode removes the problem, but it would be nice >> to be able to have strict confinement earlier rather than later. >> > If this is the problem and you can patch the software then removing the chown > could work, I am CCing Jamie for other ideas that could come up. > > -- > Snapcraft mailing list > Snapcraft@lists.snapcraft.io > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/snapcraft -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
On 03/11/16 11:49, Joseph Rushton Wakeling wrote: On 01/11/16 22:43, Sergio Schvezov wrote: If this is x86_64, everything is aligned with the world, syscall 92 is chown. A useful tool here can help you out, and luckily there is one, run `snap install snappy-debug` and it will do some nice things to figure out what is going on wth these apparmor and seccomp blockers. Cheers, I'll try that out. Will probably be a little while before I follow up -- I'm going to be away from computer for the next couple of weeks. Sorry for delayed response; I was on vacation and very deliberately AFK ;-) If this is the problem and you can patch the software then removing the chown could work, I am CCing Jamie for other ideas that could come up. Looking at the dub source code, it seems that the actual build -- i.e. the compiler call that generates the binary -- writes to a temporary .dub/ directory created in the project tree, and the generated files are then copied to the user-perceived output location, with chown and chmod calls to preserve the permissions: https://github.com/dlang/dub/blob/v1.0.0/source/dub/internal/vibecompat/core/file.d#L126-L128 That seems a reasonable enough approach in itself, but is it possible to preserve that basic operational principle without hitting the block on chown? The `copy` function called earlier on L115 actually has an optional `preserve` parameter that allows file attributes to be maintained during the copy; it might be worth trying this out to see if that doesn't hit the blocks from the container, although it still does a chmod: https://github.com/dlang/phobos/blob/60cd8d2aa70e1c9cfd7c58fa42bce7345fd00b77/std/file.d#L3393 ... so whether that would work would presumably hinge on whether chmod is also forbidden by containerization, or just chown. It's not obvious to me why the author of the dub code has instead created a POSIX-specific control block to set permissions manually, so I'll check out that with the dub folks. -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
On 03/11/16 12:08, Jamie Bennett wrote: Regards, Jamie. ... I guess an email got lost there? :-) Thanks in any case for taking the time to look at my questions. -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
On 13/11/16 10:11, Joseph Rushton Wakeling wrote: On 03/11/16 11:49, Joseph Rushton Wakeling wrote: On 01/11/16 22:43, Sergio Schvezov wrote: If this is the problem and you can patch the software then removing the chown could work, I am CCing Jamie for other ideas that could come up. Looking at the dub source code, it seems that the actual build -- i.e. the compiler call that generates the binary -- writes to a temporary .dub/ directory created in the project tree, and the generated files are then copied to the user-perceived output location, with chown and chmod calls to preserve the permissions: https://github.com/dlang/dub/blob/v1.0.0/source/dub/internal/vibecompat/core/file.d#L126-L128 OK, upstream accepted my patch to deal with this. The current state of my external snap package, described here: https://github.com/WebDrake/dub.snap/pull/1 ... now works with some essential basics: * it can compile a program that has no dependencies; * it can download dub packages from https://code.dlang.org/ and incorporate them into a project. The major TODOs would be: * ensure the dub plugin downloads an upstream dub rather than relying on ubuntu packages; * separate the D compiler from the snap, allowing dub to use any compiler available on the host system (whether installed as a system package or as a snap package); * find a way for it to access system libraries so as to build dub packages that have these as dependencies. The latter two I presume come down to the known issue about how to make host resources available to a snap in a safe manner, but I'd be interested in any thoughts on whether the D compiler issue might be achieved any more easily. -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
> Wiadomość napisana przez Joseph Rushton Wakeling > w dniu 01.11.2016, o godz. 22:31: > > On 27/10/16 22:13, Joseph Rushton Wakeling wrote: >> On 27/10/16 08:37, Didier Roche wrote: >>> I would look at /var/log/syslogs. Apparmor and seccomp denials are >>> listed there. Note that if you didn't already, you should really start >>> developping your snap in devmode. That way, it will get confinment out >>> of the equasion to get your relocatable code and dependencies working. >>> Then, we can turn on confinement and figure out those issues to be able >>> to publish in the stable channel. >> >> Yea, I probably should have started with devmode. Thanks for the advice >> about >> syslogs; I'll check it out and see what I can find. > > OK, so it looks like apparmor was indeed responsible. The loglines in > question: > > Oct 30 17:50:50 computername kernel: [ 9532.992875] audit: type=1400 > audit(1477846250.853:43): apparmor="DENIED" operation="link" > profile="snap.dub.dub" > name="/home/username/code/D/dgraph/build/dgraph_graphtest" pid=22464 > comm="dub" requested_mask="l" denied_mask="l" fsuid=1000 ouid=1000 > target="/home/username/code/D/dgraph/.dub/build/application-debug-linux.posix-x86_64-ldc_0-B7AFC7F4AA486AA98C5445F91F5653DB/dgraph_graphtest” This is a link (or symlink, not sure) operation. > Oct 30 17:50:50 computername kernel: [ 9533.035303] audit: type=1326 > audit(1477846250.897:44): auid=4294967295 uid=1000 gid=1000 ses=4294967295 > pid=22464 comm="dub" exe="/snap/dub/x1/bin/dub" sig=31 arch=c03e > syscall=92 compat=0 ip=0x7f9b72d13717 code=0x0 This seems to be chown(1) > > I'm not experienced with apparmor, so could someone explain exactly what this > means? (I get the general idea, but the specifics would be useful to > understand precisely.) > > In particular, is there an obvious reason why this might be showing up with > the dub snap, when the earlier ldc2 snap didn't have this problem? I would > guess because the ldc2 instance used by the snap-packaged dub is internal to > the snap and does not benefit from the home-directory interface that dub > itself gets? > > Setting the containment to devmode removes the problem, but it would be nice > to be able to have strict confinement earlier rather than later. > > -- > Snapcraft mailing list > Snapcraft@lists.snapcraft.io > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/snapcraft -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
Le 13/11/2016 à 18:09, Joseph Rushton Wakeling a écrit : > On 13/11/16 10:11, Joseph Rushton Wakeling wrote: >> On 03/11/16 11:49, Joseph Rushton Wakeling wrote: >>> On 01/11/16 22:43, Sergio Schvezov wrote: If this is the problem and you can patch the software then removing the chown could work, I am CCing Jamie for other ideas that could come up. >> >> Looking at the dub source code, it seems that the actual build -- >> i.e. the >> compiler call that generates the binary -- writes to a temporary >> .dub/ directory >> created in the project tree, and the generated files are then copied >> to the >> user-perceived output location, with chown and chmod calls to >> preserve the >> permissions: >> https://github.com/dlang/dub/blob/v1.0.0/source/dub/internal/vibecompat/core/file.d#L126-L128 >> > > OK, upstream accepted my patch to deal with this. The current state > of my external snap package, described here: > https://github.com/WebDrake/dub.snap/pull/1 > > ... now works with some essential basics: > > * it can compile a program that has no dependencies; > > * it can download dub packages from https://code.dlang.org/ and > incorporate > them into a project. > > The major TODOs would be: > > * ensure the dub plugin downloads an upstream dub rather than > relying on > ubuntu packages; > > * separate the D compiler from the snap, allowing dub to use any > compiler > available on the host system (whether installed as a system > package or > as a snap package); > > * find a way for it to access system libraries so as to build dub > packages > that have these as dependencies. > > The latter two I presume come down to the known issue about how to > make host resources available to a snap in a safe manner, but I'd be > interested in any thoughts on whether the D compiler issue might be > achieved any more easily. > I guess for now, the compiler part (and access to system libraries) should be part of an interface. I'm CCing Jamie to see if he has any thoughts on that. Didier -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Provisional snap for DUB (D language package/build manager)
On 14/11/16 10:27, Didier Roche wrote: I guess for now, the compiler part (and access to system libraries) should be part of an interface. I'm CCing Jamie to see if he has any thoughts on that. Yes, that makes sense -- probably two different interfaces ... ? I know that access to system libraries is a known issue in terms of defining an interface (this was discussed earlier when I was attempting to snap-package LDC, the LLVM-based D compiler). I'm happy to have a go at defining an interface that would allow access to a D compiler, if someone can offer basic guidance as to what would be involved. (Ideally, such an interface would offer access both to snap-packaged D compilers and any D compilers installed on the host system.) Thanks & best wishes, -- Joe -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
