Re: Currernt config hook implementation scales very badly

[Oliver Grawert]

hi,
On Di, 2017-02-14 at 09:23 -0300, Sergio Schvezov wrote:
> El martes, 14 de febrero de 2017 07h'08:33 ART, Oliver Grawert 
>  escribió:
> > 
> > hi,
> > Am Dienstag, den 14.02.2017, 17:59 +0800 schrieb XiaoGuo Liu:
> > > 
> > > 
> > > I think the problem could be that the stable channel does not
> > > support
> > > it.
> > > 
> > yes, this is definitely the case, this was not released to stable
> > yet,
> > the next stable release will have it. 
> 
> On the snapcraft side we have started the practice of annotating the 
> version the documented item was introduced in. I guess it would be
> good to 
> update the documentation in the snapd wiki to have this. Might tie in
> with 
> `assumes` rather nicely.
> 
well, the problem here is that such a feature spans across many
different places, while snapd 2.22 has the interface support (which is
actually pointed out in the docs), the additional support bits in the
system might need to land in the different packages first and then as a
last step in the configure hook. 

since we also dont have a generic core revision number for all arches
by design it would become quite a table to maintain for each single
config option. perhaps just a date stamp might work "this feature is
available in the stable core snap released on: ..."

ciao
oli

signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Sergio Schvezov]

El martes, 14 de febrero de 2017 07h'08:33 ART, Oliver Grawert 
 escribió:

hi,
Am Dienstag, den 14.02.2017, 17:59 +0800 schrieb XiaoGuo Liu:


I think the problem could be that the stable channel does not support
it.


yes, this is definitely the case, this was not released to stable yet,
the next stable release will have it. 


On the snapcraft side we have started the practice of annotating the 
version the documented item was introduced in. I guess it would be good to 
update the documentation in the snapd wiki to have this. Might tie in with 
`assumes` rather nicely.


Cheers


--
Enviado con Dekko desde mi dispositivo Ubuntu

--
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Oliver Grawert]

hi,
Am Dienstag, den 14.02.2017, 17:59 +0800 schrieb XiaoGuo Liu:
> 
> I think the problem could be that the stable channel does not support
> it.
> 
yes, this is definitely the case, this was not released to stable yet,
the next stable release will have it. 

ciao
oli

signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[XiaoGuo Liu]

Hi Ciao,

Thanks! I just upgraded it to edge channel, and now it works as expected.

liu-xiao-guo@localhost:~$ snap set core service.ssh.disable=true
liu-xiao-guo@localhost:~$  sudo snap set core service.ssh.disable=true
liu-xiao-guo@localhost:~$  snap set core service.ssh.disable=true
liu-xiao-guo@localhost:~$ snap set core service.ssh.disable=false
liu-xiao-guo@localhost:~$ snap interfaces |grep core
:core-support core

I think the problem could be that the stable channel does not support it.

Best regards,
XiaoGuo

On Tue, Feb 14, 2017 at 5:45 PM, Oliver Grawert  wrote:

> hi,
> Am Dienstag, den 14.02.2017, 10:53 +0800 schrieb XiaoGuo Liu:
> > Hi,
> >
> > Today, I just followed the instructions at:
> >
> > https://github.com/CanonicalLtd/ubuntu-core-docs/blob/master/en/refer
> > ence/core-configuration.md
> >
> > to disable ssh. However, I got the error like:
> >
> > liu-xiao-guo@localhost:~$ sudo snap set core service.ssh.disable=true
> > error: cannot perform the following tasks:
> > - Run configure hook of "core" snap
> > (/snap/core/1177/meta/hooks/configure: 62:
> > /snap/core/1177/meta/hooks/configure: systemctl: Permission denied)
> >
> > Is this a bug? It was tested on Raspberry Pi2 device, and my core
> > system info is like:
>
> hmm, works fine on my pi2 install ...
>
> ogra@localhost:~$ snap set core service.ssh.disable=true
> ogra@localhost:~$ snap set core service.ssh.disable=false
> ogra@localhost:~$
>
> line 62 calls: "systemctl disable ssh.service"
> do you get the same error if you call the above (with sudo) manually ?
>
> could your SD card be full or have errors (and thus be mounted
> readonly) ?
>
> also check if the core-support interface is properly connected (it
> should auto-connect, this would definitely be a bug):
>
> ogra@localhost:~$ snap interfaces |grep core
> :core-support core
> ogra@localhost:~$
>
> ciao
> oli
> --
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/snapcraft
>
>


-- 
XiaoGuo, Liu
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Oliver Grawert]

hi,
Am Dienstag, den 14.02.2017, 10:53 +0800 schrieb XiaoGuo Liu:
> Hi,
> 
> Today, I just followed the instructions at:
> 
> https://github.com/CanonicalLtd/ubuntu-core-docs/blob/master/en/refer
> ence/core-configuration.md
> 
> to disable ssh. However, I got the error like:
> 
> liu-xiao-guo@localhost:~$ sudo snap set core service.ssh.disable=true
> error: cannot perform the following tasks:
> - Run configure hook of "core" snap
> (/snap/core/1177/meta/hooks/configure: 62:
> /snap/core/1177/meta/hooks/configure: systemctl: Permission denied)
> 
> Is this a bug? It was tested on Raspberry Pi2 device, and my core
> system info is like:

hmm, works fine on my pi2 install ...

ogra@localhost:~$ snap set core service.ssh.disable=true
ogra@localhost:~$ snap set core service.ssh.disable=false
ogra@localhost:~$ 

line 62 calls: "systemctl disable ssh.service"
do you get the same error if you call the above (with sudo) manually ?

could your SD card be full or have errors (and thus be mounted
readonly) ?

also check if the core-support interface is properly connected (it
should auto-connect, this would definitely be a bug):

ogra@localhost:~$ snap interfaces |grep core
:core-support core
ogra@localhost:~$ 

ciao
oli

signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[XiaoGuo Liu]

Hi,

Today, I just followed the instructions at:

https://github.com/CanonicalLtd/ubuntu-core-docs/blob/master/en/reference/core-configuration.md

to disable ssh. However, I got the error like:

liu-xiao-guo@localhost:~$ sudo snap set core service.ssh.disable=true
error: cannot perform the following tasks:
- Run configure hook of "core" snap (/snap/core/1177/meta/hooks/configure:
62: /snap/core/1177/meta/hooks/configure: systemctl: Permission denied)

Is this a bug? It was tested on Raspberry Pi2 device, and my core system
info is like:

liu-xiao-guo@localhost:~$ snap version
snap2.23~201702101018.git.979009c~ubuntu16.04.1
snapd   2.23~201702101018.git.979009c~ubuntu16.04.1
series  16

liu-xiao-guo@localhost:~$ snap info core
name:  core
summary:   "snapd runtime environment"
publisher: canonical
description: |
  The core runtime environment for snapd
type:core
tracking:edge
installed:   16.04.1 (1177) 68MB -
refreshed:   2017-02-13 04:42:55 + UTC
channels:
  stable:16.04.1 (893)  67MB -
  candidate: 16.04.1 (1083) 68MB -
  beta:  16.04.1 (1083) 68MB -
  edge:  16.04.1 (1177) 68MB -

Thanks & best regards,
xiaoguo

On Wed, Feb 1, 2017 at 6:31 PM, Oliver Grawert  wrote:

> hi,
>
> after we recently added a config hook [1] to the core snap it is now
> possible to disable ssh if you require [2] ...
>
> there is a long standing request to do the same for syslog for systems
> running from SD card, which is why i looked into trying to extend the
> existing configure script for this, which made me notice some issue...
>
> when you call "snap set core foo=bar", there is no indication at all
> for the script which variable changes a value, neither in the shell
> environment nor in the argument list.
>
> the only way to get your value is to call snapctl for every existing
> variable [3]. since you dont know what specific variable was requested
> to change the only way to reliably write it is to write *all* existing
> variables ...
>
> now, if you package something with a more complex config that you want
> completely handled via snap config this gets ugly very fast... imagine
> something like postfix with can potentially have 100s of config
> options. instead of atomically changing a single option you will
> regenerate the full config from scratch. this takes a lot longer,
> bringing the risk of a completely broken config in case of a power
> loss, beyond resulting an an awful config hook script with a large
> block of "snapctl get" calls at the top.
>
> can we extend the config hook implementation minimally so a "snap set
> core foo=bar" would at least have something like "SNAP_OPTION=foo" in
> the environment that the generating script or binary could read to
> avoid the unnecessary processing or would that break some design
> concept ?
>
> ciao
> oli
>
> [1] http://bazaar.launchpad.net/~snappy-dev/core-snap/trunk/view/head:/
> hooks/configure
> [2] https://github.com/CanonicalLtd/ubuntu-core-docs/blob/master/en/ref
> erence/core-configuration.md
> [3] https://github.com/snapcore/snapd/wiki/hooks
> --
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/snapcraft
>
>


-- 
XiaoGuo, Liu
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Mark Shuttleworth]

On 02/02/17 10:23, Oliver Grawert wrote:
> just as a side note, this works for all basic python stuff
> (wrappers/config etc), but not if you use any complex modules. 
> in that case you should indeed ship python in your snap using a python
> plugin in snapcraft.yaml and include the necessary modules in your
> snap.

Yes, my primary point was that if we are going to offer some standard
helper modules, I would suggest those be in Python3 using only the
pieces available in the core which is always available. That way, with
minimal overhead, config capabilities can be improving constantly for
all snapcrafters.

Mark



signature.asc
Description: OpenPGP digital signature
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Oliver Grawert]

hi,
Am Donnerstag, den 02.02.2017, 10:16 + schrieb Mark Shuttleworth:
> 
> > Really? I thought Python was an excellent choice, and built and
> > staged all my wrappers as a Python part. 'core' already has
> > Python3, so it isn't bloating the snap size. And it makes it a
> > doddle to manipulate json, yaml or ini files, when this sort of
> > thing requires an expert to do correctly in bash or dash.
> > 
> > (but look at 'jq' if you insist on shell scripts - it seemed very
> > helpful for dealing with json)
>  
> I would strongly +1 python3 from the core snap for general hook
> authorship. It's always there, it's perfectly fast for one-time
> operations, it's comfortable for text handling, it's architecture-
> independent with small files.
> 

just as a side note, this works for all basic python stuff
(wrappers/config etc), but not if you use any complex modules. 
in that case you should indeed ship python in your snap using a python
plugin in snapcraft.yaml and include the necessary modules in your
snap.

ciao
oli

signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Mark Shuttleworth]

On 02/02/17 03:16, Stuart Bishop wrote:
>
>
> On 1 February 2017 at 21:48, Michael Hall  > wrote:
>
> On a related note, does anybody have a suggestion on a lightweight way
> of programmatically modifying configs in .ini, XML or JSON?
>
> I have a couple of use-cases (erlang-based nosql databases) where the
> server updates the the local configuration, so I can't just overwrite
> the whole thing every time. Didier's example will work for a very
> basic
> key=value config file, but not something more structured, and I'd hate
> to introduce Python, Perl or Java just for this.
>
>
> Really? I thought Python was an excellent choice, and built and staged
> all my wrappers as a Python part. 'core' already has Python3, so it
> isn't bloating the snap size. And it makes it a doddle to manipulate
> json, yaml or ini files, when this sort of thing requires an expert to
> do correctly in bash or dash.
>
> (but look at 'jq' if you insist on shell scripts - it seemed very
> helpful for dealing with json)

I would strongly +1 python3 from the core snap for general hook
authorship. It's always there, it's perfectly fast for one-time
operations, it's comfortable for text handling, it's
architecture-independent with small files.

Mark
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Stuart Bishop]

On 1 February 2017 at 21:48, Michael Hall  wrote:

> On a related note, does anybody have a suggestion on a lightweight way
> of programmatically modifying configs in .ini, XML or JSON?
>
> I have a couple of use-cases (erlang-based nosql databases) where the
> server updates the the local configuration, so I can't just overwrite
> the whole thing every time. Didier's example will work for a very basic
> key=value config file, but not something more structured, and I'd hate
> to introduce Python, Perl or Java just for this.
>

Really? I thought Python was an excellent choice, and built and staged all
my wrappers as a Python part. 'core' already has Python3, so it isn't
bloating the snap size. And it makes it a doddle to manipulate json, yaml
or ini files, when this sort of thing requires an expert to do correctly in
bash or dash.

(but look at 'jq' if you insist on shell scripts - it seemed very helpful
for dealing with json)

-- 
Stuart Bishop 
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Michael Hall]

If we made some generic config helpers in Go, they could be part of the
core snap instead of being included in each individual one.

However, it's actually very hard to have an arch-independent snap, in my
experience, due to the number of dependencies you typically need to
bundle which are often arch-dependent.

Michael Hall
mhall...@gmail.com

On 02/01/2017 10:08 AM, Didier Roche wrote:
> Le 01/02/2017 à 15:48, Michael Hall a écrit :
>> On a related note, does anybody have a suggestion on a lightweight way
>> of programmatically modifying configs in .ini, XML or JSON?
>>
>> I have a couple of use-cases (erlang-based nosql databases) where the
>> server updates the the local configuration, so I can't just overwrite
>> the whole thing every time. Didier's example will work for a very basic
>> key=value config file, but not something more structured, and I'd hate
>> to introduce Python, Perl or Java just for this.
> 
> That's exactly why I limited for now to simple key=value (and
> sourceable) files, while keeping comments and other formats (so .ini
> file should work if you scaffold it). I wanted to limit the hook to
> shell script to be architecture independant and not adding a dep on some
> runtime language.
> 
> I don't know if it's sane to have more structured one in dash-compatible
> shell. Introducing helpers written in Go means that you have to compile
> your (potential arch-all snap like a nodejs webserver) to be arch-dependant.
> Thoughts?
> 
> Didier
>>
>>
>> Michael Hall
>> mhall...@ubuntu.com
>>
>> On 02/01/2017 08:54 AM, Didier Roche wrote:
>>> Le 01/02/2017 à 13:05, Gustavo Niemeyer a écrit :
 Yes, that's in the plans for configuration support already.  The idea
 is to extend snapctl with the ability to introspect exactly which
 settings have changed since the last successful run of the script, and
 perhaps which values it used to hold before.

 For the time being, I suggest not worrying much about that. Just go
 ahead and implement the semantics you want the user of the snap to
 experience. Inside the configure hook, check to see if the requested
 value is already in place and do nothing in that case. This makes the
 script idempotent, resilient because it ensures the desired state is
 indeed in place, and also cheap to execute in the common case.
  
>>> If someone is interested, I have implemented this kind of idempotent
>>> configure hook example. It can be found at
>>> https://github.com/ubuntu/snow-on-me-snap/blob/master/meta/hooks/configure
>>> as part of the christmas snap demo.
>>> It features:
>>> - support key=value formatting
>>> - ignoring, but not erasing unknown keys, comment lines, empty ones.
>>> - preserving formatting and ordering, even when changing an existing key.
>>> - only snap set arguments will be modified if a value is set. If the
>>> value assigned to it is empty, removing the key line
>>> - new elements not present in existing configuration file are simply
>>> appended.
>>>
>>> Supporting new keys (like here "foo") is just a matter of adding:
>>> update_opt_in_config foo
>>>
>>> at the end of the file.
>>>
>>> I hope this helps! (note that the issue is still "what's the supported
>>> keys for this snap?")
>>> Cheers,
>>> Didier
>>>

 On Wed, Feb 1, 2017 at 8:31 AM, Oliver Grawert >>> > wrote:

 hi,

 after we recently added a config hook [1] to the core snap it is now
 possible to disable ssh if you require [2] ...

 there is a long standing request to do the same for syslog for systems
 running from SD card, which is why i looked into trying to extend the
 existing configure script for this, which made me notice some issue...

 when you call "snap set core foo=bar", there is no indication at all
 for the script which variable changes a value, neither in the shell
 environment nor in the argument list.

 the only way to get your value is to call snapctl for every existing
 variable [3]. since you dont know what specific variable was requested
 to change the only way to reliably write it is to write *all* existing
 variables ... 

 now, if you package something with a more complex config that you want
 completely handled via snap config this gets ugly very fast... imagine
 something like postfix with can potentially have 100s of config
 options. instead of atomically changing a single option you will
 regenerate the full config from scratch. this takes a lot longer,
 bringing the risk of a completely broken config in case of a power
 loss, beyond resulting an an awful config hook script with a large
 block of "snapctl get" calls at the top.

 can we extend the config hook implementation minimally so a "snap set
 core foo=bar" would at least have something like "SNAP_OPTION=foo" in

Re: Currernt config hook implementation scales very badly

[Oliver Grawert]

hi,
Am Mittwoch, den 01.02.2017, 16:50 +0100 schrieb Simon Fels:
> 
> > oh, thanks, i had totally forgotten about that one ! 
> > i guess adding this  alongside is indeed possible (if the core-
> > support
> > interface allows me writing to /etc/rsyslog.d/, i havent checked
> > yet)
> It doesn't. But I am pretty sure we can add that.

ok, let me start with making the dir writable in the image ;)

ciao
oli

signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Simon Fels]

On 01.02.2017 13:53, Oliver Grawert wrote:
> hi,
> On Mi, 2017-02-01 at 06:31 -0600, Jamie Strandboge wrote:
>> On Wed, 2017-02-01 at 11:31 +0100, Oliver Grawert wrote:
>>>
>>> hi,
>>>
>>> after we recently added a config hook [1] to the core snap it is
>>> now
>>> possible to disable ssh if you require [2] ...
>>>
>>> there is a long standing request to do the same for syslog for
>>> systems
>>> running from SD card, which is why i looked into trying to extend
>>> the
>>> existing configure script for this, which made me notice some
>>> issue...
>>>
>> There is also https://bugs.launchpad.net/snappy/+bug/1504645 for
>> remote
>> syslogging. Is this something that might be implemented as part of
>> this work?
>>
>> (sorry to hijack the thread, feel free to comment in the bug).
>>
> 
> oh, thanks, i had totally forgotten about that one ! 
> i guess adding this  alongside is indeed possible (if the core-support
> interface allows me writing to /etc/rsyslog.d/, i havent checked yet)

It doesn't. But I am pretty sure we can add that.

regards,
Simon


-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Didier Roche]

Le 01/02/2017 à 15:48, Michael Hall a écrit :
> On a related note, does anybody have a suggestion on a lightweight way
> of programmatically modifying configs in .ini, XML or JSON?
>
> I have a couple of use-cases (erlang-based nosql databases) where the
> server updates the the local configuration, so I can't just overwrite
> the whole thing every time. Didier's example will work for a very basic
> key=value config file, but not something more structured, and I'd hate
> to introduce Python, Perl or Java just for this.

That's exactly why I limited for now to simple key=value (and
sourceable) files, while keeping comments and other formats (so .ini
file should work if you scaffold it). I wanted to limit the hook to
shell script to be architecture independant and not adding a dep on some
runtime language.

I don't know if it's sane to have more structured one in dash-compatible
shell. Introducing helpers written in Go means that you have to compile
your (potential arch-all snap like a nodejs webserver) to be arch-dependant.
Thoughts?

Didier
>
>
> Michael Hall
> mhall...@ubuntu.com
>
> On 02/01/2017 08:54 AM, Didier Roche wrote:
>> Le 01/02/2017 à 13:05, Gustavo Niemeyer a écrit :
>>> Yes, that's in the plans for configuration support already.  The idea
>>> is to extend snapctl with the ability to introspect exactly which
>>> settings have changed since the last successful run of the script, and
>>> perhaps which values it used to hold before.
>>>
>>> For the time being, I suggest not worrying much about that. Just go
>>> ahead and implement the semantics you want the user of the snap to
>>> experience. Inside the configure hook, check to see if the requested
>>> value is already in place and do nothing in that case. This makes the
>>> script idempotent, resilient because it ensures the desired state is
>>> indeed in place, and also cheap to execute in the common case.
>>>  
>> If someone is interested, I have implemented this kind of idempotent
>> configure hook example. It can be found at
>> https://github.com/ubuntu/snow-on-me-snap/blob/master/meta/hooks/configure
>> as part of the christmas snap demo.
>> It features:
>> - support key=value formatting
>> - ignoring, but not erasing unknown keys, comment lines, empty ones.
>> - preserving formatting and ordering, even when changing an existing key.
>> - only snap set arguments will be modified if a value is set. If the
>> value assigned to it is empty, removing the key line
>> - new elements not present in existing configuration file are simply
>> appended.
>>
>> Supporting new keys (like here "foo") is just a matter of adding:
>> update_opt_in_config foo
>>
>> at the end of the file.
>>
>> I hope this helps! (note that the issue is still "what's the supported
>> keys for this snap?")
>> Cheers,
>> Didier
>>
>>>
>>> On Wed, Feb 1, 2017 at 8:31 AM, Oliver Grawert >> > wrote:
>>>
>>> hi,
>>>
>>> after we recently added a config hook [1] to the core snap it is now
>>> possible to disable ssh if you require [2] ...
>>>
>>> there is a long standing request to do the same for syslog for systems
>>> running from SD card, which is why i looked into trying to extend the
>>> existing configure script for this, which made me notice some issue...
>>>
>>> when you call "snap set core foo=bar", there is no indication at all
>>> for the script which variable changes a value, neither in the shell
>>> environment nor in the argument list.
>>>
>>> the only way to get your value is to call snapctl for every existing
>>> variable [3]. since you dont know what specific variable was requested
>>> to change the only way to reliably write it is to write *all* existing
>>> variables ... 
>>>
>>> now, if you package something with a more complex config that you want
>>> completely handled via snap config this gets ugly very fast... imagine
>>> something like postfix with can potentially have 100s of config
>>> options. instead of atomically changing a single option you will
>>> regenerate the full config from scratch. this takes a lot longer,
>>> bringing the risk of a completely broken config in case of a power
>>> loss, beyond resulting an an awful config hook script with a large
>>> block of "snapctl get" calls at the top.
>>>
>>> can we extend the config hook implementation minimally so a "snap set
>>> core foo=bar" would at least have something like "SNAP_OPTION=foo" in
>>> the environment that the generating script or binary could read to
>>> avoid the unnecessary processing or would that break some design
>>> concept ?
>>>
>>> ciao
>>> oli
>>>
>>> [1] http://bazaar.launchpad.net/~snappy-dev/core-snap/trunk/view/head:/
>>> hooks/configure
>>> 
>>> 
>>> [2] https://github.com/CanonicalLtd/ubuntu-core-docs/blob/master/en/ref
>>>

Re: Currernt config hook implementation scales very badly

[Michael Hall]

On a related note, does anybody have a suggestion on a lightweight way
of programmatically modifying configs in .ini, XML or JSON?

I have a couple of use-cases (erlang-based nosql databases) where the
server updates the the local configuration, so I can't just overwrite
the whole thing every time. Didier's example will work for a very basic
key=value config file, but not something more structured, and I'd hate
to introduce Python, Perl or Java just for this.


Michael Hall
mhall...@ubuntu.com

On 02/01/2017 08:54 AM, Didier Roche wrote:
> Le 01/02/2017 à 13:05, Gustavo Niemeyer a écrit :
>>
>> Yes, that's in the plans for configuration support already.  The idea
>> is to extend snapctl with the ability to introspect exactly which
>> settings have changed since the last successful run of the script, and
>> perhaps which values it used to hold before.
>>
>> For the time being, I suggest not worrying much about that. Just go
>> ahead and implement the semantics you want the user of the snap to
>> experience. Inside the configure hook, check to see if the requested
>> value is already in place and do nothing in that case. This makes the
>> script idempotent, resilient because it ensures the desired state is
>> indeed in place, and also cheap to execute in the common case.
>>  
> 
> If someone is interested, I have implemented this kind of idempotent
> configure hook example. It can be found at
> https://github.com/ubuntu/snow-on-me-snap/blob/master/meta/hooks/configure
> as part of the christmas snap demo.
> It features:
> - support key=value formatting
> - ignoring, but not erasing unknown keys, comment lines, empty ones.
> - preserving formatting and ordering, even when changing an existing key.
> - only snap set arguments will be modified if a value is set. If the
> value assigned to it is empty, removing the key line
> - new elements not present in existing configuration file are simply
> appended.
> 
> Supporting new keys (like here "foo") is just a matter of adding:
> update_opt_in_config foo
> 
> at the end of the file.
> 
> I hope this helps! (note that the issue is still "what's the supported
> keys for this snap?")
> Cheers,
> Didier
> 
>>
>>
>> On Wed, Feb 1, 2017 at 8:31 AM, Oliver Grawert > > wrote:
>>
>> hi,
>>
>> after we recently added a config hook [1] to the core snap it is now
>> possible to disable ssh if you require [2] ...
>>
>> there is a long standing request to do the same for syslog for systems
>> running from SD card, which is why i looked into trying to extend the
>> existing configure script for this, which made me notice some issue...
>>
>> when you call "snap set core foo=bar", there is no indication at all
>> for the script which variable changes a value, neither in the shell
>> environment nor in the argument list.
>>
>> the only way to get your value is to call snapctl for every existing
>> variable [3]. since you dont know what specific variable was requested
>> to change the only way to reliably write it is to write *all* existing
>> variables ... 
>>
>> now, if you package something with a more complex config that you want
>> completely handled via snap config this gets ugly very fast... imagine
>> something like postfix with can potentially have 100s of config
>> options. instead of atomically changing a single option you will
>> regenerate the full config from scratch. this takes a lot longer,
>> bringing the risk of a completely broken config in case of a power
>> loss, beyond resulting an an awful config hook script with a large
>> block of "snapctl get" calls at the top.
>>
>> can we extend the config hook implementation minimally so a "snap set
>> core foo=bar" would at least have something like "SNAP_OPTION=foo" in
>> the environment that the generating script or binary could read to
>> avoid the unnecessary processing or would that break some design
>> concept ?
>>
>> ciao
>> oli
>>
>> [1] http://bazaar.launchpad.net/~snappy-dev/core-snap/trunk/view/head:/
>> hooks/configure
>> 
>> 
>> [2] https://github.com/CanonicalLtd/ubuntu-core-docs/blob/master/en/ref
>> erence/core-configuration.md
>> 
>> 
>> [3] https://github.com/snapcore/snapd/wiki/hooks
>> 
>> --
>> Snapcraft mailing list
>> Snapcraft@lists.snapcraft.io 
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/snapcraft
>> 
>>
>>
>>
>>
>> -- 
>> gustavo @ http://niemeyer.net
>>
>>
> 
> 
> 

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or 

Re: Currernt config hook implementation scales very badly

[Didier Roche]

Le 01/02/2017 à 13:05, Gustavo Niemeyer a écrit :
>
> Yes, that's in the plans for configuration support already.  The idea
> is to extend snapctl with the ability to introspect exactly which
> settings have changed since the last successful run of the script, and
> perhaps which values it used to hold before.
>
> For the time being, I suggest not worrying much about that. Just go
> ahead and implement the semantics you want the user of the snap to
> experience. Inside the configure hook, check to see if the requested
> value is already in place and do nothing in that case. This makes the
> script idempotent, resilient because it ensures the desired state is
> indeed in place, and also cheap to execute in the common case.
>  

If someone is interested, I have implemented this kind of idempotent
configure hook example. It can be found at
https://github.com/ubuntu/snow-on-me-snap/blob/master/meta/hooks/configure
as part of the christmas snap demo.
It features:
- support key=value formatting
- ignoring, but not erasing unknown keys, comment lines, empty ones.
- preserving formatting and ordering, even when changing an existing key.
- only snap set arguments will be modified if a value is set. If the
value assigned to it is empty, removing the key line
- new elements not present in existing configuration file are simply
appended.

Supporting new keys (like here "foo") is just a matter of adding:
update_opt_in_config foo

at the end of the file.

I hope this helps! (note that the issue is still "what's the supported
keys for this snap?")
Cheers,
Didier

>
>
> On Wed, Feb 1, 2017 at 8:31 AM, Oliver Grawert  > wrote:
>
> hi,
>
> after we recently added a config hook [1] to the core snap it is now
> possible to disable ssh if you require [2] ...
>
> there is a long standing request to do the same for syslog for systems
> running from SD card, which is why i looked into trying to extend the
> existing configure script for this, which made me notice some issue...
>
> when you call "snap set core foo=bar", there is no indication at all
> for the script which variable changes a value, neither in the shell
> environment nor in the argument list.
>
> the only way to get your value is to call snapctl for every existing
> variable [3]. since you dont know what specific variable was requested
> to change the only way to reliably write it is to write *all* existing
> variables ... 
>
> now, if you package something with a more complex config that you want
> completely handled via snap config this gets ugly very fast... imagine
> something like postfix with can potentially have 100s of config
> options. instead of atomically changing a single option you will
> regenerate the full config from scratch. this takes a lot longer,
> bringing the risk of a completely broken config in case of a power
> loss, beyond resulting an an awful config hook script with a large
> block of "snapctl get" calls at the top.
>
> can we extend the config hook implementation minimally so a "snap set
> core foo=bar" would at least have something like "SNAP_OPTION=foo" in
> the environment that the generating script or binary could read to
> avoid the unnecessary processing or would that break some design
> concept ?
>
> ciao
> oli
>
> [1] http://bazaar.launchpad.net/~snappy-dev/core-snap/trunk/view/head:/
> hooks/configure
> 
> 
> [2] https://github.com/CanonicalLtd/ubuntu-core-docs/blob/master/en/ref
> erence/core-configuration.md
> 
> 
> [3] https://github.com/snapcore/snapd/wiki/hooks
> 
> --
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io 
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/snapcraft
> 
>
>
>
>
> -- 
> gustavo @ http://niemeyer.net
>
>

-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Oliver Grawert]

hi,
On Mi, 2017-02-01 at 06:31 -0600, Jamie Strandboge wrote:
> On Wed, 2017-02-01 at 11:31 +0100, Oliver Grawert wrote:
> > 
> > hi,
> > 
> > after we recently added a config hook [1] to the core snap it is
> > now
> > possible to disable ssh if you require [2] ...
> > 
> > there is a long standing request to do the same for syslog for
> > systems
> > running from SD card, which is why i looked into trying to extend
> > the
> > existing configure script for this, which made me notice some
> > issue...
> > 
> There is also https://bugs.launchpad.net/snappy/+bug/1504645 for
> remote
> syslogging. Is this something that might be implemented as part of
> this work?
> 
> (sorry to hijack the thread, feel free to comment in the bug).
> 

oh, thanks, i had totally forgotten about that one ! 
i guess adding this  alongside is indeed possible (if the core-support
interface allows me writing to /etc/rsyslog.d/, i havent checked yet)

ciao
oli

signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Jamie Strandboge]

On Wed, 2017-02-01 at 11:31 +0100, Oliver Grawert wrote:
> hi,
> 
> after we recently added a config hook [1] to the core snap it is now
> possible to disable ssh if you require [2] ...
> 
> there is a long standing request to do the same for syslog for systems
> running from SD card, which is why i looked into trying to extend the
> existing configure script for this, which made me notice some issue...
> 
There is also https://bugs.launchpad.net/snappy/+bug/1504645 for remote
syslogging. Is this something that might be implemented as part of this work?

(sorry to hijack the thread, feel free to comment in the bug).

-- 
Jamie Strandboge | http://www.canonical.com



signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Gustavo Niemeyer]

The current behavior is to present the full configuration every time, so
it'll definitely result in a healthy configuration every time no matter
what subset has changed, assuming the whole configuration is actually
valid, because changing 2 or 10 options at once results in the same
behavior on the script end. This was one of the important changes we
pursued during the reimplementation last year. What we need now is adding
some features to help with the complex cases, when there are hundreds of
options and you want to optimize the implementation having in mind exactly
which pieces have changed.

That optimization may actually be implemented completely internally to the
configure script, if we had an urgent use case today, since we can always
cache the last valid configuration and diff it. But it'll be nice to have
that as a standard feature instead.

Also, a good problem to have.


On Wed, Feb 1, 2017 at 9:53 AM, Mark Shuttleworth  wrote:

>
> We definitely want some sort of "dry-run" capability, where you could
> present a snippet of YAML covering a subset of config and learn whether
> that would result in a healthy config, and what changes it would
> actually imply.
>
> Mark
>
>
> --
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
> an/listinfo/snapcraft
>
>


-- 
gustavo @ http://niemeyer.net
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Gustavo Niemeyer]

Yes, that's in the plans for configuration support already.  The idea is to
extend snapctl with the ability to introspect exactly which settings have
changed since the last successful run of the script, and perhaps which
values it used to hold before.

For the time being, I suggest not worrying much about that. Just go ahead
and implement the semantics you want the user of the snap to experience.
Inside the configure hook, check to see if the requested value is already
in place and do nothing in that case. This makes the script idempotent,
resilient because it ensures the desired state is indeed in place, and also
cheap to execute in the common case.



On Wed, Feb 1, 2017 at 8:31 AM, Oliver Grawert  wrote:

> hi,
>
> after we recently added a config hook [1] to the core snap it is now
> possible to disable ssh if you require [2] ...
>
> there is a long standing request to do the same for syslog for systems
> running from SD card, which is why i looked into trying to extend the
> existing configure script for this, which made me notice some issue...
>
> when you call "snap set core foo=bar", there is no indication at all
> for the script which variable changes a value, neither in the shell
> environment nor in the argument list.
>
> the only way to get your value is to call snapctl for every existing
> variable [3]. since you dont know what specific variable was requested
> to change the only way to reliably write it is to write *all* existing
> variables ...
>
> now, if you package something with a more complex config that you want
> completely handled via snap config this gets ugly very fast... imagine
> something like postfix with can potentially have 100s of config
> options. instead of atomically changing a single option you will
> regenerate the full config from scratch. this takes a lot longer,
> bringing the risk of a completely broken config in case of a power
> loss, beyond resulting an an awful config hook script with a large
> block of "snapctl get" calls at the top.
>
> can we extend the config hook implementation minimally so a "snap set
> core foo=bar" would at least have something like "SNAP_OPTION=foo" in
> the environment that the generating script or binary could read to
> avoid the unnecessary processing or would that break some design
> concept ?
>
> ciao
> oli
>
> [1] http://bazaar.launchpad.net/~snappy-dev/core-snap/trunk/view/head:/
> hooks/configure
> [2] https://github.com/CanonicalLtd/ubuntu-core-docs/blob/master/en/ref
> erence/core-configuration.md
> [3] https://github.com/snapcore/snapd/wiki/hooks
> --
> Snapcraft mailing list
> Snapcraft@lists.snapcraft.io
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/snapcraft
>
>


-- 
gustavo @ http://niemeyer.net
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Re: Currernt config hook implementation scales very badly

[Mark Shuttleworth]

We definitely want some sort of "dry-run" capability, where you could
present a snippet of YAML covering a subset of config and learn whether
that would result in a healthy config, and what changes it would
actually imply.

Mark



signature.asc
Description: OpenPGP digital signature
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft

Currernt config hook implementation scales very badly

[Oliver Grawert]

hi,

after we recently added a config hook [1] to the core snap it is now
possible to disable ssh if you require [2] ...

there is a long standing request to do the same for syslog for systems
running from SD card, which is why i looked into trying to extend the
existing configure script for this, which made me notice some issue...

when you call "snap set core foo=bar", there is no indication at all
for the script which variable changes a value, neither in the shell
environment nor in the argument list.

the only way to get your value is to call snapctl for every existing
variable [3]. since you dont know what specific variable was requested
to change the only way to reliably write it is to write *all* existing
variables ... 

now, if you package something with a more complex config that you want
completely handled via snap config this gets ugly very fast... imagine
something like postfix with can potentially have 100s of config
options. instead of atomically changing a single option you will
regenerate the full config from scratch. this takes a lot longer,
bringing the risk of a completely broken config in case of a power
loss, beyond resulting an an awful config hook script with a large
block of "snapctl get" calls at the top.

can we extend the config hook implementation minimally so a "snap set
core foo=bar" would at least have something like "SNAP_OPTION=foo" in
the environment that the generating script or binary could read to
avoid the unnecessary processing or would that break some design
concept ?

ciao
oli

[1] http://bazaar.launchpad.net/~snappy-dev/core-snap/trunk/view/head:/
hooks/configure
[2] https://github.com/CanonicalLtd/ubuntu-core-docs/blob/master/en/ref
erence/core-configuration.md
[3] https://github.com/snapcore/snapd/wiki/hooks

signature.asc
Description: This is a digitally signed message part
-- 
Snapcraft mailing list
Snapcraft@lists.snapcraft.io
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/snapcraft