Re: Snapcraft's state tracking improvements
On 14 December 2016 at 04:57, Kyle Fazzari wrote: > Hey everyone. > > I feel it coming on... this is going to be a tome. tl;dr... snapcraft > could be smarter than it is. But would that lead to its doing more for > you than you want? We'd like to find out. > > > *Option 1*: Error out and make you be specific. This is the current > behavior. > > *Option 2*: Automatically take care of everything. If you modify a part > with dependencies, snapcraft will rebuild those dependencies as it sees > fit without your needing to say so. Similarly, if you clean a part with > dependencies, snapcraft will clean those dependencies as it sees fit > without your needing to say so. > I would prefer a reliable Option 2. If it can't be reliable, then Option 1. *Option 3*: Prompt. "You've modified part A, but part B depends upon it. > Would you like to rebuild it as well? (Y/n)" and the like. > No thanks. There seems little purpose to answering 'no', apart from blowing off your foot. And useless if you are driving this from a Makefile or similar (think generating your snapcraft.yaml). > *Option 4*: Add the ability to configure this behavior between options > 1-3. Perhaps the sensible default would be option 3. > > > Our goal here is to do the least surprising thing, which is why we're > asking for input from our users. Please let your voice be heard! I found it annoying that I would need to run three or more commands to build my snap when iterating, and would certainly like it if cleaning and rebuilding happened automatically if I have touched relevant source code. People might need to separate out their parts into different directories to make this reliable. A problematic case I would have tripped over is a slow building autotools project, in which I've added a snapcraft.yaml and some Python wrappers. I would not want to trigger a rebuild of the autotools part because I'm fixing a bug in my wrappers (a second part, but stored in the same tree). Maybe you could handle this by monitoring the files opened by the build process somehow? -- Stuart Bishop -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Snapcraft's state tracking improvements
Hey Kyle, Le 13/12/2016 à 22:57, Kyle Fazzari a écrit : > and force you to say exactly what you wanted. For example, if you simply > said `snapcraft clean part B` we error out, saying something like "Hey > you're trying to clean part B, but part A depends on it. If you intend > for both to be cleaned, please say so." That sounds fine, you have an option that let the user overwrites the warning though right (if cleaning only part B was really what was wanted)? > *Option 2*: Automatically take care of everything. If you modify a part > with dependencies, snapcraft will rebuild those dependencies as it sees > fit without your needing to say so. Similarly, if you clean a part with > dependencies, snapcraft will clean those dependencies as it sees fit > without your needing to say so. > I think I would go for this one but only if you provide an option to disable the smart behaviour and let the developers who know what they are doing be in control. If you don't provide the option, or until you do, I would prefer for the tools to error out asking me to be more specific than wrong guessing what I might be trying to do (and e.g cleaning out that webkit part that takes another day to rebuild) Cheers, Sebastien Bacher -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Snapcraft's state tracking improvements
On 14/12/2016 09:02, Didier Roche wrote: > I would think a good way is always to not try to be too smart. This is > kicking back in some way in the future. I would personally go with > option 1 + a suggestion of command to run to autofix it. Instead of > prompting (option3), add an option like --auto (or --force). +1 Ciao, Alberto -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Snapcraft's state tracking improvements
El 14/12/16 a las 03:02, Didier Roche escribió: Le 13/12/2016 à 22:57, Kyle Fazzari a écrit : Hey everyone. Hey Kyle I feel it coming on... this is going to be a tome. tl;dr... snapcraft could be smarter than it is. But would that lead to its doing more for you than you want? We'd like to find out. I've spoken to a few of you individually about this, and I want to open this topic up for wider conversation. We have a number of bugs logged against snapcraft for its state tracking shortcomings. For example, a few of you may have run into these issues: - You add a stage package to a part in an already-built snap, and run `snapcraft` again. It happily says all the steps have already run, and generates a snap that doesn't contain your stage package. - You add (or modify) a file in the local source for a part of an already-built snap, and run `snapcraft` again`. It again reports that everything has already run, and generates a snap that doesn't contain your modifications. There is another one: there is a remote source which changes. I think snapcraft can be smarter about that and store the hash of the git/bzr commit id, and check if new remote hash is the same or not. This is a little bit more problematic for other kind of assets like tarballs, as you can only create and check a checksum (if not published) after downloading… defeating the caching purpose :) For remote sources, I would only do this on an explicit (re)pull. We don't want to talk to the network (unless a plugin is wired to do so where we strive not to) unless the user really wants to. -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Snapcraft's state tracking improvements
El 13/12/16 a las 21:27, Seth Arnold escribió: On Tue, Dec 13, 2016 at 01:57:20PM -0800, Kyle Fazzari wrote: *Option 2*: Automatically take care of everything. If you modify a part with dependencies, snapcraft will rebuild those dependencies as it sees fit without your needing to say so. Similarly, if you clean a part with This options feels most likely to lead to reliably reproducable results. I am inclined to go with this option as well. When chaining parts with the `after` keyword, what usually comes """before""" are your dependencies whereas when creating a snap one would work on the leaves (the parts holding the most `after`s). I think of it almost as a `Build-Depends` in `debian/control` -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Snapcraft's state tracking improvements
Le 13/12/2016 à 22:57, Kyle Fazzari a écrit : > Hey everyone. Hey Kyle > > I feel it coming on... this is going to be a tome. tl;dr... snapcraft > could be smarter than it is. But would that lead to its doing more for > you than you want? We'd like to find out. > > I've spoken to a few of you individually about this, and I want to open > this topic up for wider conversation. We have a number of bugs logged > against snapcraft for its state tracking shortcomings. For example, a > few of you may have run into these issues: > > - You add a stage package to a part in an already-built snap, and run > `snapcraft` again. It happily says all the steps have already run, and > generates a snap that doesn't contain your stage package. > > - You add (or modify) a file in the local source for a part of an > already-built snap, and run `snapcraft` again`. It again reports that > everything has already run, and generates a snap that doesn't contain > your modifications. There is another one: there is a remote source which changes. I think snapcraft can be smarter about that and store the hash of the git/bzr commit id, and check if new remote hash is the same or not. This is a little bit more problematic for other kind of assets like tarballs, as you can only create and check a checksum (if not published) after downloading… defeating the caching purpose :) > The point is: snapcraft is/will be smart enough to help a lot more than > it currently does in these types of situations. The question I'm posing > to you all is: how much do you want its help? > > I see four options for how to handle such situations: > > > *Option 1*: Error out and make you be specific. This is the current > behavior. > > *Option 2*: Automatically take care of everything. If you modify a part > with dependencies, snapcraft will rebuild those dependencies as it sees > fit without your needing to say so. Similarly, if you clean a part with > dependencies, snapcraft will clean those dependencies as it sees fit > without your needing to say so. > > *Option 3*: Prompt. "You've modified part A, but part B depends upon it. > Would you like to rebuild it as well? (Y/n)" and the like. > > *Option 4*: Add the ability to configure this behavior between options > 1-3. Perhaps the sensible default would be option 3. > > > Our goal here is to do the least surprising thing, which is why we're > asking for input from our users. Please let your voice be heard! I would think a good way is always to not try to be too smart. This is kicking back in some way in the future. I would personally go with option 1 + a suggestion of command to run to autofix it. Instead of prompting (option3), add an option like --auto (or --force). This gives a sane default behavior, and let then integration tools (like IDE integration or scripts) to use the --auto after detecting a failure. Then, they can decide themselves to do it right away or to prompt the user (which is harder if you implement option 3 as a CLI prompt). Option 4 sounds dangerous to me, because there are mutiple situations which can triggers this, and so, if you set auto as a default for everything, you can go trigger surprising behavior for certain cases. Here is my 2 cents -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Snapcraft's state tracking improvements
On Tue, Dec 13, 2016 at 01:57:20PM -0800, Kyle Fazzari wrote: > *Option 2*: Automatically take care of everything. If you modify a part > with dependencies, snapcraft will rebuild those dependencies as it sees > fit without your needing to say so. Similarly, if you clean a part with This options feels most likely to lead to reliably reproducable results. Thanks signature.asc Description: PGP signature -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Re: Snapcraft's state tracking improvements
Thanks kyle, this is really important! In a hackathon event, when the network was bad and shared by many people, keeping cleaning the project and rebuilding the project took a lot of time to build the whole project. Sometime, it took a few tens minutes to get it done! Best regards, XiaoGuo On Wed, Dec 14, 2016 at 5:57 AM, Kyle Fazzari wrote: > Hey everyone. > > I feel it coming on... this is going to be a tome. tl;dr... snapcraft > could be smarter than it is. But would that lead to its doing more for > you than you want? We'd like to find out. > > I've spoken to a few of you individually about this, and I want to open > this topic up for wider conversation. We have a number of bugs logged > against snapcraft for its state tracking shortcomings. For example, a > few of you may have run into these issues: > > - You add a stage package to a part in an already-built snap, and run > `snapcraft` again. It happily says all the steps have already run, and > generates a snap that doesn't contain your stage package. > > - You add (or modify) a file in the local source for a part of an > already-built snap, and run `snapcraft` again`. It again reports that > everything has already run, and generates a snap that doesn't contain > your modifications. > > We used to have even more of these issues, but snapcraft has slowly been > getting smarter about things like this. You may have noticed that v2.23 > fixed that first problem, for example. Well, sort of. Let me explain. > > Snapcraft's state tracking allows us to do some handy things. For > example, as you probably know, when you run the `stage` or `prime` > steps, you're combining files from multiple parts into a single > directory. Thanks to state tracking, snapcraft is able to disentangle > which files came from which part, and allow you to remove a single > part's files using per-step cleaning. > > However, this is complicated by the ability to have inter-part > dependencies (using the `after` keyword). If part A depends on part B > and part B is being cleaned, then snapcraft knows part A needs to be > cleaned as well. But should it go ahead and clean it for you? What if > you were offline and part A required the internet to build? Or > re-building part A took forever and you didn't want to just wipe it? We > weren't sure when we implemented this ability, so we decided to be safe > and force you to say exactly what you wanted. For example, if you simply > said `snapcraft clean part B` we error out, saying something like "Hey > you're trying to clean part B, but part A depends on it. If you intend > for both to be cleaned, please say so." > > This same mindset can be found elsewhere: > > - If you change something in the YAML that makes a part out of date > (e.g. stage packages) it tells you it's out of date and why, but doesn't > clean/rebuild it for you. > > - Trying to build part A which depends on part B before B itself has > been staged leads to "Hey, part A has unsatisfied dependencies." If you > run `snapcraft stage A B` it'll do the right thing. > > - Not-yet-released features will know when local sources for a part have > changed, and build the updates, but what should it do if that part has > another depending upon it? > > The point is: snapcraft is/will be smart enough to help a lot more than > it currently does in these types of situations. The question I'm posing > to you all is: how much do you want its help? > > I see four options for how to handle such situations: > > > *Option 1*: Error out and make you be specific. This is the current > behavior. > > *Option 2*: Automatically take care of everything. If you modify a part > with dependencies, snapcraft will rebuild those dependencies as it sees > fit without your needing to say so. Similarly, if you clean a part with > dependencies, snapcraft will clean those dependencies as it sees fit > without your needing to say so. > > *Option 3*: Prompt. "You've modified part A, but part B depends upon it. > Would you like to rebuild it as well? (Y/n)" and the like. > > *Option 4*: Add the ability to configure this behavior between options > 1-3. Perhaps the sensible default would be option 3. > > > Our goal here is to do the least surprising thing, which is why we're > asking for input from our users. Please let your voice be heard! > > -- > Kyle Fazzari (kyrofa) > Software Engineer > Canonical Ltd. > k...@canonical.com > > > -- > Snapcraft mailing list > Snapcraft@lists.snapcraft.io > Modify settings or unsubscribe at: https://lists.ubuntu.com/ > mailman/listinfo/snapcraft > > -- XiaoGuo, Liu -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
Snapcraft's state tracking improvements
Hey everyone. I feel it coming on... this is going to be a tome. tl;dr... snapcraft could be smarter than it is. But would that lead to its doing more for you than you want? We'd like to find out. I've spoken to a few of you individually about this, and I want to open this topic up for wider conversation. We have a number of bugs logged against snapcraft for its state tracking shortcomings. For example, a few of you may have run into these issues: - You add a stage package to a part in an already-built snap, and run `snapcraft` again. It happily says all the steps have already run, and generates a snap that doesn't contain your stage package. - You add (or modify) a file in the local source for a part of an already-built snap, and run `snapcraft` again`. It again reports that everything has already run, and generates a snap that doesn't contain your modifications. We used to have even more of these issues, but snapcraft has slowly been getting smarter about things like this. You may have noticed that v2.23 fixed that first problem, for example. Well, sort of. Let me explain. Snapcraft's state tracking allows us to do some handy things. For example, as you probably know, when you run the `stage` or `prime` steps, you're combining files from multiple parts into a single directory. Thanks to state tracking, snapcraft is able to disentangle which files came from which part, and allow you to remove a single part's files using per-step cleaning. However, this is complicated by the ability to have inter-part dependencies (using the `after` keyword). If part A depends on part B and part B is being cleaned, then snapcraft knows part A needs to be cleaned as well. But should it go ahead and clean it for you? What if you were offline and part A required the internet to build? Or re-building part A took forever and you didn't want to just wipe it? We weren't sure when we implemented this ability, so we decided to be safe and force you to say exactly what you wanted. For example, if you simply said `snapcraft clean part B` we error out, saying something like "Hey you're trying to clean part B, but part A depends on it. If you intend for both to be cleaned, please say so." This same mindset can be found elsewhere: - If you change something in the YAML that makes a part out of date (e.g. stage packages) it tells you it's out of date and why, but doesn't clean/rebuild it for you. - Trying to build part A which depends on part B before B itself has been staged leads to "Hey, part A has unsatisfied dependencies." If you run `snapcraft stage A B` it'll do the right thing. - Not-yet-released features will know when local sources for a part have changed, and build the updates, but what should it do if that part has another depending upon it? The point is: snapcraft is/will be smart enough to help a lot more than it currently does in these types of situations. The question I'm posing to you all is: how much do you want its help? I see four options for how to handle such situations: *Option 1*: Error out and make you be specific. This is the current behavior. *Option 2*: Automatically take care of everything. If you modify a part with dependencies, snapcraft will rebuild those dependencies as it sees fit without your needing to say so. Similarly, if you clean a part with dependencies, snapcraft will clean those dependencies as it sees fit without your needing to say so. *Option 3*: Prompt. "You've modified part A, but part B depends upon it. Would you like to rebuild it as well? (Y/n)" and the like. *Option 4*: Add the ability to configure this behavior between options 1-3. Perhaps the sensible default would be option 3. Our goal here is to do the least surprising thing, which is why we're asking for input from our users. Please let your voice be heard! -- Kyle Fazzari (kyrofa) Software Engineer Canonical Ltd. k...@canonical.com signature.asc Description: OpenPGP digital signature -- Snapcraft mailing list Snapcraft@lists.snapcraft.io Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/snapcraft
