Re: Sending binaries over relations

[John Meinel]

It does feel like a good fit for resources, with the one caveat that he
wants to maintain a lock-step version of the resource across services.
There is slightly more work with the current designs for resources, in that
each charm will think about its version of the resource independently. But
we will have the fingerprint information to allow for users to compare and
be confident that both services are using the same resource.

John
=:->

On Wed, Jan 20, 2016 at 8:53 PM, Mark Shuttleworth  wrote:

> On 20/01/16 14:24, Merlijn Sebrechts wrote:
> > So my question is: Is there a way to send large binary files between
> > Charms? Or is this problem better solved by using a subordinate
> > kafka-plugin Charm like the Hadoop Charms do?
>
> It sounds like you want the new "Resources" capability coming in Juju 2.0
> :)
>
> For shared large blobs (like a JVM or a big ball of libraries) the charm
> can ask the state server to cache the blob and distribute it to all the
> units. There are mechanisms for users to supply the blob if needed, too.
>
> Mark
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Juju stable 1.25.3 is proposed for release

[Curtis Hovey-Canonical]

# juju-core 1.25.3

A new proposed stable release of Juju, juju-core 1.25.3, is now available.
This release may replace version 1.25.0 on Thursday January 21.


## Getting Juju

juju-core 1.25.3 is available for Xenial and backported to earlier
series in the following PPA:

https://launchpad.net/~juju/+archive/proposed

Windows, Centos, and OS X users will find installers at:

https://launchpad.net/juju-core/+milestone/1.25.3

Proposed releases use the "proposed" simple-streams. You must configure
the `agent-stream` option in your environments.yaml to use the matching
juju agents.


## Notable Changes

This releases addresses a defect discovered in proposed 1.25.2


## Resolved issues

  * Unit loses network connectivity during bootstrap: juju 1.25.2 +
maas 1.9
Lp 1534795

 * "cannot allocate memory" when running "juju run"
Lp 1382556

  * Bootstrap with the vsphere provider fails to log into the virtual
machine
Lp 1511138

  * Add-machine with vsphere triggers machine-0: panic: juju home
hasn't been initialized
Lp 1513492

  * Using maas 1.9 as provider using dhcp nic will prevent juju
bootstrap
Lp 1512371

  * Worker/storageprovisioner: machine agents attempting to attach
environ-scoped volumes
Lp 1483492

  * Restore: agent old password not found in configuration
Lp 1452082

  * "ignore-machine-addresses" broken for containers
Lp 1509292

  * Deploying a service to a space which has no subnets causes the
agent to panic
Lp 1499426

  * /var/lib/juju gone after 1.18->1.20 upgrade and manual edit of
agent.conf
Lp 1444912

  * Juju bootstrap fails to successfully configure the bridge juju-br0
when deploying with wily 4.2 kernel
Lp 1496972

  * Incompatible cookie format change
Lp 1511717

  * Error environment destruction failed: destroying storage: listing
volumes: get https://x.x.x.x:8776/v2//volumes/detail: local
error: record overflow
Lp 1512399

  * Replica set emptyconfig maas bootstrap
Lp 1412621

  * Juju can't find daily image streams from cloud-
images.ubuntu.com/daily
Lp 1513982

  * Rsyslog certificate fails when using ipv6/4 dual stack with
prefer-ipv6: true
Lp 1478943

  * Improper address:port joining
Lp 1518128

  * Juju status  broken
Lp 1516989

  * 1.25.1 with maas 1.8: devices dns allocation uses non-unique
hostname
Lp 1525280

  * Increment minimum juju version for 2.0 upgrade to 1.25.3
Lp 1533751

  * Make assignment of units to machines use a worker
Lp 1497312

  * `juju environments` fails due to missing ~/.juju/current-
environment
Lp 1506680

  * Juju 1.25 misconfigures juju-br0 when using maas 1.9 bonded
interface
Lp 1516891

  * Destroy-environment on an unbootstrapped maas environment can
release all my nodes
Lp 1490865

  * On juju upgrade the security group lost ports for the exposed
services
Lp 1506649

  * Support centos and windows image metadata
Lp 1523693

  * Upgrade-juju shows available tools and best version but did not
output what it decided to do
Lp 1403655

  * Invalid binary version, version "1.23.3--amd64" or "1.23.3--armhf"
Lp 1459033

  * Add xenial to supported series
Lp 1533262


Finally

We encourage everyone to subscribe the mailing list at
juju-...@lists.canonical.com, or join us on #juju-dev on freenode.


-- 
Curtis Hovey
Canonical Cloud Development and Operations
http://launchpad.net/~sinzui

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Juju stable 1.25.3 is proposed for release

[Curtis Hovey-Canonical]

# juju-core 1.25.3

A new proposed stable release of Juju, juju-core 1.25.3, is now available.
This release may replace version 1.25.0 on Thursday January 21.


## Getting Juju

juju-core 1.25.3 is available for Xenial and backported to earlier
series in the following PPA:

https://launchpad.net/~juju/+archive/proposed

Windows, Centos, and OS X users will find installers at:

https://launchpad.net/juju-core/+milestone/1.25.3

Proposed releases use the "proposed" simple-streams. You must configure
the `agent-stream` option in your environments.yaml to use the matching
juju agents.


## Notable Changes

This releases addresses a defect discovered in proposed 1.25.2


## Resolved issues

  * Unit loses network connectivity during bootstrap: juju 1.25.2 +
maas 1.9
Lp 1534795

 * "cannot allocate memory" when running "juju run"
Lp 1382556

  * Bootstrap with the vsphere provider fails to log into the virtual
machine
Lp 1511138

  * Add-machine with vsphere triggers machine-0: panic: juju home
hasn't been initialized
Lp 1513492

  * Using maas 1.9 as provider using dhcp nic will prevent juju
bootstrap
Lp 1512371

  * Worker/storageprovisioner: machine agents attempting to attach
environ-scoped volumes
Lp 1483492

  * Restore: agent old password not found in configuration
Lp 1452082

  * "ignore-machine-addresses" broken for containers
Lp 1509292

  * Deploying a service to a space which has no subnets causes the
agent to panic
Lp 1499426

  * /var/lib/juju gone after 1.18->1.20 upgrade and manual edit of
agent.conf
Lp 1444912

  * Juju bootstrap fails to successfully configure the bridge juju-br0
when deploying with wily 4.2 kernel
Lp 1496972

  * Incompatible cookie format change
Lp 1511717

  * Error environment destruction failed: destroying storage: listing
volumes: get https://x.x.x.x:8776/v2//volumes/detail: local
error: record overflow
Lp 1512399

  * Replica set emptyconfig maas bootstrap
Lp 1412621

  * Juju can't find daily image streams from cloud-
images.ubuntu.com/daily
Lp 1513982

  * Rsyslog certificate fails when using ipv6/4 dual stack with
prefer-ipv6: true
Lp 1478943

  * Improper address:port joining
Lp 1518128

  * Juju status  broken
Lp 1516989

  * 1.25.1 with maas 1.8: devices dns allocation uses non-unique
hostname
Lp 1525280

  * Increment minimum juju version for 2.0 upgrade to 1.25.3
Lp 1533751

  * Make assignment of units to machines use a worker
Lp 1497312

  * `juju environments` fails due to missing ~/.juju/current-
environment
Lp 1506680

  * Juju 1.25 misconfigures juju-br0 when using maas 1.9 bonded
interface
Lp 1516891

  * Destroy-environment on an unbootstrapped maas environment can
release all my nodes
Lp 1490865

  * On juju upgrade the security group lost ports for the exposed
services
Lp 1506649

  * Support centos and windows image metadata
Lp 1523693

  * Upgrade-juju shows available tools and best version but did not
output what it decided to do
Lp 1403655

  * Invalid binary version, version "1.23.3--amd64" or "1.23.3--armhf"
Lp 1459033

  * Add xenial to supported series
Lp 1533262


Finally

We encourage everyone to subscribe the mailing list at
juju-...@lists.canonical.com, or join us on #juju-dev on freenode.


-- 
Curtis Hovey
Canonical Cloud Development and Operations
http://launchpad.net/~sinzui

-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Sending binaries over relations

[Merlijn Sebrechts]

I like the idea of rsync. Is there a way to restrict access to a single
file on the rsync server?

2016-01-20 16:16 GMT+01:00 Marco Ceppi :

> I don't think sending the binary via relation is a good idea. Either
> spinning up a web service or using rsync would be a better bet
>
> On Wed, Jan 20, 2016, 10:10 AM Matthew Williams <
> matthew.willi...@canonical.com> wrote:
>
>> Would it not be better for the charm to have a path the client can `wget`
>> the libraries from - this path can be sent via the relation as a string
>>
>> Matty
>>
>> On Wed, Jan 20, 2016 at 2:30 PM, José Antonio Rey 
>> wrote:
>>
>>> Hey,
>>>
>>> One of the options would be to cat the file as a string and pass that
>>> string over the connection, finally echoing that string to foo.binary.
>>>
>>> What do others think?
>>>
>>> --
>>> José Antonio Rey
>>>
>>> On Wed, Jan 20, 2016, 08:25 Merlijn Sebrechts <
>>> merlijn.sebrec...@gmail.com> wrote:
>>>
 Hi


 I have a question I'd like to discuss, if you guys aren't to busy
 prepping for Ubucon.. :)

 I've found a number of Java projects where, in order to communicate for
 example with Kafka, they require the Kafka Java libraries for that specific
 version. For the moment, I solve this by downloading the libraries from a
 deployed Kafka installation and include them in the Charm. However, this
 has the disadvantage that everytime the Kafka charm version changes, I have
 to update the libraries in all the charms that connect to Kafka. It would
 be better if there was a way to send these libraries over the connection.
 This way, a Charm that can connect to one version of Kafka has a very high
 chance of being able to connect to the next version.

 So my question is: Is there a way to send large binary files between
 Charms? Or is this problem better solved by using a subordinate
 kafka-plugin Charm like the Hadoop Charms do?



 Kind regards
 Merlijn Sebrechts
 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju

>>>
>>> --
>>> Juju mailing list
>>> Juju@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>
>>>
>> --
>> Juju mailing list
>> Juju@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>>
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Sending binaries over relations

[Merlijn Sebrechts]

@José: I worry that this will become a big strain to the state server. I
don't know much about the internal architecture to be certain of this,
though..

@Matthew: This would side-step jujuresources. It might be feasible if
jujuresources could provide a way for the Charm to Share that resource,
though... Even then, we might get into trouble if the binaries aren't
included as a resource but are compiled during installation...

2016-01-20 16:10 GMT+01:00 Matthew Williams 
:

> Would it not be better for the charm to have a path the client can `wget`
> the libraries from - this path can be sent via the relation as a string
>
> Matty
>
> On Wed, Jan 20, 2016 at 2:30 PM, José Antonio Rey  wrote:
>
>> Hey,
>>
>> One of the options would be to cat the file as a string and pass that
>> string over the connection, finally echoing that string to foo.binary.
>>
>> What do others think?
>>
>> --
>> José Antonio Rey
>>
>> On Wed, Jan 20, 2016, 08:25 Merlijn Sebrechts <
>> merlijn.sebrec...@gmail.com> wrote:
>>
>>> Hi
>>>
>>>
>>> I have a question I'd like to discuss, if you guys aren't to busy
>>> prepping for Ubucon.. :)
>>>
>>> I've found a number of Java projects where, in order to communicate for
>>> example with Kafka, they require the Kafka Java libraries for that specific
>>> version. For the moment, I solve this by downloading the libraries from a
>>> deployed Kafka installation and include them in the Charm. However, this
>>> has the disadvantage that everytime the Kafka charm version changes, I have
>>> to update the libraries in all the charms that connect to Kafka. It would
>>> be better if there was a way to send these libraries over the connection.
>>> This way, a Charm that can connect to one version of Kafka has a very high
>>> chance of being able to connect to the next version.
>>>
>>> So my question is: Is there a way to send large binary files between
>>> Charms? Or is this problem better solved by using a subordinate
>>> kafka-plugin Charm like the Hadoop Charms do?
>>>
>>>
>>>
>>> Kind regards
>>> Merlijn Sebrechts
>>> --
>>> Juju mailing list
>>> Juju@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>
>>
>> --
>> Juju mailing list
>> Juju@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>>
>>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Master is unblocked

[Cheryl Jennings]

Hi Everyone,

Yesterday we had a blessed CI run on master which we will be releasing as
our 2.0-alpha1.  The final touches will be put on the release notes and we
will turn the crank to get an official release shortly.

In the meantime, I've unblocked master.  So, let the thundering herd of
merges begin!

Thanks again for your patience while we worked towards a stable 2.0-alpha1!
-Cheryl
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Automatic retries of hooks

[Gabriel Samfira]

On Mi, 2016-01-20 at 10:39 -0500, Aaron Bentley wrote:
> On 2016-01-20 10:30 AM, Gabriel Samfira wrote:
> > The auto-retry thing was created to overcome situations in which
> > the machine is rebooted, or chashes during a hook run
> > (independently of juju). In this case, the charm would not be able
> > to recover automatically from a transient situation.
> 
> If the intent was to handle reboots, couldn't it be written to
> restart
> any pending hooks after a reboot, rather than when the hooks fail?

The original intent was to re-run a hook in case of external
intervention outside of juju. This includes but is not limited to:

* automatic reboots
* OOM situation
* power outage
* killall -9 jujud (chaos monkey/gremlins/postal sysadmin)

This has grown to automatically retry on any failure. While retrying
 once at agent startup is enough for *some* needs, it may not be enough
for other charms. I would not remove the current behavior. I would
simply make it configurable in case the current behavior does not suit
everyone. The auto retry on all errors is a safe bet for charms that do
not implement retry logic, and as William stated, retrying an operation
inside a hook, will block all other hooks form running.

Just my 2 cents.

> Even re-running just at agent-startup would be a lot clearer.
> 
> Aaron
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Automatic retries of hooks

[William Reade]

On Wed, Jan 20, 2016 at 2:42 PM, Dean Henrichsmeyer 
wrote:

> Hi,
>
> It seems the original point James was making is getting missed. No one is
> arguing over the value of being able to retry and/or idempotent hooks.
> Yes, you should be able to retry them and yes nothing should break if you
> run them over and over.
>
> The point made is that Juju shouldn't be automatically retrying them. The
> argument of "no one knows what went wrong so Juju automatically retrying
> them is a better experience" doesn't work. The intelligence of the stack in
> question, regardless of what it is, goes in the charms. If you start
> conflating and mixing up where the intelligence goes then creating,
> running, and debugging those distributed systems will be a nightmare.
>

Hook errors *will* happen, and often for transient reasons. In handling
this, we can choose between "users retry without understanding the details"
and "juju retries without understanding the details" [0]. I'd be happy to
make the behaviour configurable, for the rare cases when the user *does*
understand the details and wants full and detailed control, but I don't
think that's the common case.

The magic should only be in Juju's ability to effectively drive the models
> and intelligence encoded in the charms. It shouldn't make assumptions about
> what that intelligence is or what those models require.
>

Stopping on hook error can only *prevent* those charms from applying their
intelligence. No more hooks to be run => no more opportunity to react. If a
charm wants to be smart about errors, it needs to detect the errors it
*knows* about, and react to those by setting status; and to move on
*without* failing the hook, thereby giving subsequent hooks an opportunity
to be smart.

Ultimately, it comes down to the fact that there's *always* another error
case you haven't considered. If you depend on the charmer to implement
retries for specific errors, that's essentially a whitelist, and they're
stuck playing whack-a-mole forever [1]. But if the charmer can depend on
external retries, they only have to worry about maintaining a
definitely-fatal blacklist and reporting those conditions in status.

Am I making any sense here?

Cheers
William


[0] or "the system stays broken forever", I suppose :).
[1] I imagine the rational approach there is to give up, and start
whitelisting by operation rather than by error; i.e. to accept that most
errors are unknown/transient and should be dumbly retried. And given that,
why should every charmer have to roll their own retries?
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Automatic retries of hooks

[Aaron Bentley]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256


On 2016-01-20 10:30 AM, Gabriel Samfira wrote:
> The auto-retry thing was created to overcome situations in which
> the machine is rebooted, or chashes during a hook run
> (independently of juju). In this case, the charm would not be able
> to recover automatically from a transient situation.

If the intent was to handle reboots, couldn't it be written to restart
any pending hooks after a reboot, rather than when the hooks fail?

Even re-running just at agent-startup would be a lot clearer.

Aaron
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQEcBAEBCAAGBQJWn6pLAAoJEK84cMOcf+9h3mIIAMbumuMlehhMELNAlxMN2bnn
1rYUIZ7P/n2CagdMnjzysZXeUkRSHOjdklE4XKJUzhxzaknRgJXNZ8Ab5R7XMU1F
f4GnOXhskmw4mAae9beve5I4vF2WINxUQcxRaRen6Ov6VRQqRxVnMnZ6S85o4tPY
lMQRh+WP40JTzDkUWcCyKpQ5JgBqP9IQwn21y9v/LiXAfbkzrzqR04hvk7HrMM5W
lRBnTUldj3GHiI8Gjq6TVx6Th76PalfPUHoBlF7cmqEEVXydmuOjzr1C3fZR8VO5
JeXif92z5sR6z4TjoxnT7ixyfoz1Rvu6pKhIPJbi1cptXjDv5wU43MJsNqT6KpQ=
=Igdi
-END PGP SIGNATURE-

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Sending binaries over relations

[Matthew Williams]

Would it not be better for the charm to have a path the client can `wget`
the libraries from - this path can be sent via the relation as a string

Matty

On Wed, Jan 20, 2016 at 2:30 PM, José Antonio Rey  wrote:

> Hey,
>
> One of the options would be to cat the file as a string and pass that
> string over the connection, finally echoing that string to foo.binary.
>
> What do others think?
>
> --
> José Antonio Rey
>
> On Wed, Jan 20, 2016, 08:25 Merlijn Sebrechts 
> wrote:
>
>> Hi
>>
>>
>> I have a question I'd like to discuss, if you guys aren't to busy
>> prepping for Ubucon.. :)
>>
>> I've found a number of Java projects where, in order to communicate for
>> example with Kafka, they require the Kafka Java libraries for that specific
>> version. For the moment, I solve this by downloading the libraries from a
>> deployed Kafka installation and include them in the Charm. However, this
>> has the disadvantage that everytime the Kafka charm version changes, I have
>> to update the libraries in all the charms that connect to Kafka. It would
>> be better if there was a way to send these libraries over the connection.
>> This way, a Charm that can connect to one version of Kafka has a very high
>> chance of being able to connect to the next version.
>>
>> So my question is: Is there a way to send large binary files between
>> Charms? Or is this problem better solved by using a subordinate
>> kafka-plugin Charm like the Hadoop Charms do?
>>
>>
>>
>> Kind regards
>> Merlijn Sebrechts
>> --
>> Juju mailing list
>> Juju@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>>
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Automatic retries of hooks

[Gabriel Samfira]

The auto-retry thing was created to overcome situations in which the machine is 
rebooted, or chashes during a hook run (independently of juju). In this case, 
the charm would not be able to recover automatically from a transient situation.

This scenario is more evident in Windows workloads, where some features like 
Hyper-V require a reboot. This is fine, because you can install the feature 
with a -NoReboot flag, and use the juju-reboot --now tool to safely reboot.

After the machine comes back up, Windows still needs to configure  the new 
feature. While it is configuring the feature, the services start (including 
juju), and hook execution starts up (as its supposed to).   The problem is that 
as part of the feature configuration, the system needs to reboot one final time 
(Windowsright?). This is done automatically by the feature installer, 
outside of juju.

This causes the hook to error, but not because of an actual problem. For this 
scenario, its enough to retry once when the unit agent comes up.

A solution might be to make this feature configurable. Something like retry 
profiles:

* periodic (current behavior)
* one-shot (once at agent startup)
* disabled

Gabriel

On Mi, 2016-01-20 at 09:39 -0500, Charles Butler wrote:
I'm pretty sure that we have amenities to reboot the host without completely 
skewing the hook execution

https://jujucharms.com/docs/1.25/reference-hook-tools#juju-reboot-[--now]

This should have rebooted the machine after safely closing out of any hook 
context the charm was in, and upon reboot it should have resumed from the next 
context in queue.  I'm not a huge fan of a charm doing auto hook retries, for 
the reasons outlined by Rick, unless it is well understood and documented 
behavior.  Just chiming in with my 2 cents


Charles Butler 
> - Juju 
Charmer
Come see the future of datacenter orchestration: http://jujucharms.com

On Wed, Jan 20, 2016 at 9:22 AM, Rick Harding 
> wrote:
+1 retries are great, with backoff, when you know you're doing it because you 
have experience that certain api requests to clouds, or to other known failure 
points.

Blindly just saying "if at first you don't succeed, go go go" isn't a better 
UX. It adds another layer of complexity in debugging, and doesn't really 
improve the product. Only the charm author knows enough about what it's trying 
to achieve to do intelligent retry.

In this case, if there's something about unexpected reboots of machines, 
perhaps there's some specific case that Juju can grow some intelligence and 
hint at the charm author what happened. The charm can then react to that 
information as it deems necessary.

On Wed, Jan 20, 2016 at 8:42 AM Dean Henrichsmeyer 
> wrote:
Hi,

It seems the original point James was making is getting missed. No one is 
arguing over the value of being able to retry and/or idempotent hooks. Yes, you 
should be able to retry them and yes nothing should break if you run them over 
and over.

The point made is that Juju shouldn't be automatically retrying them. The 
argument of "no one knows what went wrong so Juju automatically retrying them 
is a better experience" doesn't work. The intelligence of the stack in 
question, regardless of what it is, goes in the charms. If you start conflating 
and mixing up where the intelligence goes then creating, running, and debugging 
those distributed systems will be a nightmare.

The magic should only be in Juju's ability to effectively drive the models and 
intelligence encoded in the charms. It shouldn't make assumptions about what 
that intelligence is or what those models require.

Thanks.


-Dean
--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev


--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev



--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Automatic retries of hooks

[Stuart Bishop]

On 20 January 2016 at 17:46, William Reade  wrote:

> On Wed, Jan 20, 2016 at 8:46 AM, Stuart Bishop 
> wrote:

>> It happens naturally if you structure your charm to have a single hook
>> that does everything that needs to be done, rather than trying to
>> craft individual hooks to deal with specific events.
>
> Independent of everything else, *this* should *excellent* advice for
> speeding up your deployments. Have you already been writing charms like
> this? I'd love to hear your experiences; and, in particular, if you've
> noticed any improvement in deployment speed. The theoretically achievable
> speedup is vast, but the hook runner wasn't written with this approach in
> mind; we might need to make a couple of small tweaks [0] to get the best out
> of the approach.

The PostgreSQL charm has now existed in three forms. Traditional,
services framework, and now reactive framework. Using the services
framework, deployment speed was slower than traditional. You ended up
with one very long string of steps, many of which were unnecessary. I
felt it easier to maintain and understand, but logs noisier and it was
slower. The reactive framework is much faster deployment wise than all
other versions, as you can easily have only the necessary steps
triggered for the current state. The execution thread is harder to
follow, since there isn't really one, but it still seems very
maintainable and understandable. There is less code than the other
versions. It does drive you to create separate handlers for each hook,
but advice is to keep hooks at the absolute bare minimum to adjust the
charms state based on the event and put all the actual logic in the
state driven handlers.


-- 
Stuart Bishop 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Sending binaries over relations

[Merlijn Sebrechts]

It seems rssh might be able to do this!

2016-01-20 16:25 GMT+01:00 Merlijn Sebrechts :

> I like the idea of rsync. Is there a way to restrict access to a single
> file on the rsync server?
>
> 2016-01-20 16:16 GMT+01:00 Marco Ceppi :
>
>> I don't think sending the binary via relation is a good idea. Either
>> spinning up a web service or using rsync would be a better bet
>>
>> On Wed, Jan 20, 2016, 10:10 AM Matthew Williams <
>> matthew.willi...@canonical.com> wrote:
>>
>>> Would it not be better for the charm to have a path the client can
>>> `wget` the libraries from - this path can be sent via the relation as a
>>> string
>>>
>>> Matty
>>>
>>> On Wed, Jan 20, 2016 at 2:30 PM, José Antonio Rey 
>>> wrote:
>>>
 Hey,

 One of the options would be to cat the file as a string and pass that
 string over the connection, finally echoing that string to foo.binary.

 What do others think?

 --
 José Antonio Rey

 On Wed, Jan 20, 2016, 08:25 Merlijn Sebrechts <
 merlijn.sebrec...@gmail.com> wrote:

> Hi
>
>
> I have a question I'd like to discuss, if you guys aren't to busy
> prepping for Ubucon.. :)
>
> I've found a number of Java projects where, in order to communicate
> for example with Kafka, they require the Kafka Java libraries for that
> specific version. For the moment, I solve this by downloading the 
> libraries
> from a deployed Kafka installation and include them in the Charm. However,
> this has the disadvantage that everytime the Kafka charm version changes, 
> I
> have to update the libraries in all the charms that connect to Kafka. It
> would be better if there was a way to send these libraries over the
> connection. This way, a Charm that can connect to one version of Kafka has
> a very high chance of being able to connect to the next version.
>
> So my question is: Is there a way to send large binary files between
> Charms? Or is this problem better solved by using a subordinate
> kafka-plugin Charm like the Hadoop Charms do?
>
>
>
> Kind regards
> Merlijn Sebrechts
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>

 --
 Juju mailing list
 Juju@lists.ubuntu.com
 Modify settings or unsubscribe at:
 https://lists.ubuntu.com/mailman/listinfo/juju


>>> --
>>> Juju mailing list
>>> Juju@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju
>>>
>>
>> --
>> Juju mailing list
>> Juju@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>>
>>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Automatic retries of hooks

[Dean Henrichsmeyer]

On Wed, Jan 20, 2016 at 11:41 AM, William Reade  wrote:

> On Wed, Jan 20, 2016 at 3:22 PM, Rick Harding 
> wrote:
>
>> +1 retries are great, with backoff, when you know you're doing it because
>> you have experience that certain api requests to clouds, or to other known
>> failure points.
>>
>
> If you're thinking about it in terms of "known failure points" you already
> understand that you need a wide net to catch all the retryable errors that
> could come out of a given operation. What makes hook execution different
> from any other code that we want to be reliable?
>
> Blindly just saying "if at first you don't succeed, go go go" isn't a
>> better UX. It adds another layer of complexity in debugging, and doesn't
>> really improve the product. Only the charm author knows enough about what
>> it's trying to achieve to do intelligent retry.
>>
>
> Empirically, it seems that the retries caused jamespage's charm succeed
> where it would have failed; and we have happy results from Gabriel's
> windows charms as well. That STM to be evidence that the product is
> improved...
>

You realize James was complaining and not celebrating the "success" ? The
fact that we can have a discussion trying to determine whether something is
a bug or a feature indicates a problem.

-D
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Automatic retries of hooks

[William Reade]

On Wed, Jan 20, 2016 at 8:01 PM, Dean Henrichsmeyer 
wrote:
>
> You realize James was complaining and not celebrating the "success" ? The
> fact that we can have a discussion trying to determine whether something is
> a bug or a feature indicates a problem.
>

Sorry, I didn't intend to disparage his experience; I took it as legitimate
and reasonable surprise at a change we evidently didn't communicate
adequately. But I don't think it's a misfeature; I think it's a necessary
approach, in service of global reliability in challenging environments.

But: if there are times it's inconvenient and not just surprising, we
should surely be able to disable it. Gabriel/Bogdan, would you be able to
address this?

Cheers
William
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Automatic retries of hooks

[Gabriel Samfira]

On Mi, 2016-01-20 at 21:31 +0100, William Reade wrote:
> On Wed, Jan 20, 2016 at 8:01 PM, Dean Henrichsmeyer <
> d...@canonical.com> wrote:
> > You realize James was complaining and not celebrating the "success"
> > ? The fact that we can have a discussion trying to determine
> > whether something is a bug or a feature indicates a problem.
> > 
> Sorry, I didn't intend to disparage his experience; I took it as
> legitimate and reasonable surprise at a change we evidently didn't
> communicate adequately. But I don't think it's a misfeature; I think
> it's a necessary approach, in service of global reliability in
> challenging environments.
> 
> But: if there are times it's inconvenient and not just surprising, we
> should surely be able to disable it. Gabriel/Bogdan, would you be
> able to address this?

Prioritizing it ASAP. Should be a simple change.

> 
> Cheers
> William
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

LXD and trusty-backports

[John Meinel]

For juju-2.0 we know that we want to be using LXD for containers.
However, I don't believe LXD itself is going to land in trusty-updates. It
*is* already in trusty-backports. My concern is that if we end up enabling
trusty-backports, that may cause conflicts with charms, as they may get
versions of code that they weren't expecting.

When we ran into this on Precise, we used 2 mechanisms.
cloud-archive[/tools] and a PPA for Juju (to give backported mongod).

It would seem for customer engagements, using the official archive is the
easiest sell, because if they are disconnected, we know they already have
to be mirroring archive.ubuntu.com. Anything else is one more thing we have
to teach them about.

I know we also ran into problems with enabling the cloud-archive
everywhere, because it started providing new versions of Django, etc that
MAAS needed, which broke charms.
We resolved that by pinning cloud-archive to lower priority and only
installing from it by direct request. Can we do the same for
trusty-backports?

Potentially we could also only enable it when you actually go to start an
LXD container on a machine, rather than always. Does that just cause
confusion when charm X fails because you have containers, but succeeds when
you don't?

Is there any plan to get LXD into trusty-updates so we don't have to worry
about this?

John
=:->
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: "environment" vs "model" in the code

[Kapil Thangavelu]

On Mon, Jan 18, 2016 at 8:24 PM, Rick Harding 
wrote:

> No, there's not been a public note yet. It's work going into the 2.0
> updates currently.
>
> The gist of the reason is that as support for things such as networking,
> storage, and workloads expand out the idea is that Juju is doing more to
> model your infrastructure and workloads vs an environment.
>

networking and storage are very much part of an application's deployment
environment. ie. there different from dev, stage, prod. not sure what
workloads are (renamed or built on actions i presume).


> So far it's helped one of the issue that Juju has had in that it takes
> time to explain what it's actually doing before folks 'get it'.
>
Starting from the point of 'take what you have running and let Juju model
> it' seems to be clicking with new folks more.
>

so its a verb, its an instance/noun, does it also apply to templates
(previously known as bundles)?

i'm curious to try out the re-branding on some guinea pigs. re what's
commonly running to model, autoscale groups, elbs, multiple networks,
security groups, iam roles, rds.

thanks,

Kapil



>
> On Mon, Jan 18, 2016 at 9:15 AM Kapil Thangavelu  wrote:
>
>> out of curiosity is there any public explanation on the reason for the
>> change? environments map fairly naturally to various service topology
>> stages, ie my prod, qa, dev environments. while model is a rather opaque
>> term that doesn't convey much.
>>
>> On Thu, Jan 14, 2016 at 7:16 PM, Menno Smits 
>> wrote:
>>
>>> Hi all,
>>>
>>> We've committed to renaming "environment" to "model" in Juju's CLI and
>>> API but what do we want to do in Juju's internals? I'm currently adding
>>> significant new model/environment related functionality to the state
>>> package which includes adding new database collections, structs and
>>> functions which could include either "env/environment" or "model" in their
>>> names.
>>>
>>> One approach could be that we only use the word "model" at the edges -
>>> the CLI, API and GUI - and continue to use "environment" internally. That
>>> way the naming of environment related things in most of Juju's code and
>>> database stays consistent.
>>>
>>> Another approach is to use "model" for new work[1] with a hope that
>>> it'll eventually become the dominant name for the concept. This will
>>> however result in a long period of widespread inconsistency, and it's
>>> unlikely that things we'll ever completely get rid of all uses of
>>> "environment".
>>>
>>> I think we need arrive at some sort of consensus on the way to tackle
>>> this. FWIW, I prefer the former approach. Having good, consistent names for
>>> things is important[2].
>>>
>>> Thoughts?
>>>
>>> - Menno
>>>
>>> [1] - but what defines "new" and what do we do when making significant
>>> changes to existing code?
>>> [2] - http://martinfowler.com/bliki/TwoHardThings.html
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: LXD and trusty-backports

[John Meinel]

So according to https://help.ubuntu.com/community/UbuntuBackports backports
is enabled by default, as long as you explicitly request the packages from
backports. I did test bootstrapping an lxd trusty machine and then running:
 apt-get install lxd/trusty-backports lxc/trusty-backports
liblxc1/trusty-backports python3-lxc/trusty-backports
lxc-templates/trusty-backports lxd-client/trusty-backports
lxd-tools/trusty-backports

I'm not sure if we need all of those things, but we do need to manually
specify a few of them. (backports is available for selectively installing
packages, but will not be used to resolve dependencies, so you *also* need
to specify to get the dependencies from backports as well.)

That's probably the best solution for us on trusty. Unless people have
other ideas.

John
=:->


On Wed, Jan 20, 2016 at 12:38 PM, John Meinel 
wrote:

> For juju-2.0 we know that we want to be using LXD for containers.
> However, I don't believe LXD itself is going to land in trusty-updates. It
> *is* already in trusty-backports. My concern is that if we end up
> enabling trusty-backports, that may cause conflicts with charms, as they
> may get versions of code that they weren't expecting.
>
> When we ran into this on Precise, we used 2 mechanisms.
> cloud-archive[/tools] and a PPA for Juju (to give backported mongod).
>
> It would seem for customer engagements, using the official archive is the
> easiest sell, because if they are disconnected, we know they already have
> to be mirroring archive.ubuntu.com. Anything else is one more thing we
> have to teach them about.
>
> I know we also ran into problems with enabling the cloud-archive
> everywhere, because it started providing new versions of Django, etc that
> MAAS needed, which broke charms.
> We resolved that by pinning cloud-archive to lower priority and only
> installing from it by direct request. Can we do the same for
> trusty-backports?
>
> Potentially we could also only enable it when you actually go to start an
> LXD container on a machine, rather than always. Does that just cause
> confusion when charm X fails because you have containers, but succeeds when
> you don't?
>
> Is there any plan to get LXD into trusty-updates so we don't have to worry
> about this?
>
> John
> =:->
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: LXD and trusty-backports

[Martin Packman]

On 20/01/2016, John Meinel  wrote:
> So according to https://help.ubuntu.com/community/UbuntuBackports backports
> is enabled by default, as long as you explicitly request the packages from
> backports.

Yeah, that's the case for fresh machines, backports just has a lower
priority so we're safe from interfering with charm package installs.

Martin

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Automatic retries of hooks

[Martin Packman]

Another common error we see in CI is apt mirrors being unhappy leading
to hook failures. Just retry later does tend to be the right option
there, though it will often be an our or two until the archive is in a
usable state again.

On 20/01/2016, William Reade  wrote:
>
> Are there any concerns that I've missed?

Automatic retries make debugging your charm harder, as James found. I
think we want an environment setting to disable this for both testing
and for charm authors.

Martin

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Automatic retries of hooks

[roger peppe]

On 20 January 2016 at 12:20, Martin Packman
 wrote:
> Another common error we see in CI is apt mirrors being unhappy leading
> to hook failures. Just retry later does tend to be the right option
> there, though it will often be an our or two until the archive is in a
> usable state again.
>
> On 20/01/2016, William Reade  wrote:
>>
>> Are there any concerns that I've missed?
>
> Automatic retries make debugging your charm harder, as James found. I
> think we want an environment setting to disable this for both testing
> and for charm authors.

This seems like a good idea.
Also perhaps it wouldn't be so bad if you at least were able
to find some record of the hook failures without delving into the
logs.

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Sending binaries over relations

[José Antonio Rey]

Hey,

One of the options would be to cat the file as a string and pass that
string over the connection, finally echoing that string to foo.binary.

What do others think?

--
José Antonio Rey

On Wed, Jan 20, 2016, 08:25 Merlijn Sebrechts 
wrote:

> Hi
>
>
> I have a question I'd like to discuss, if you guys aren't to busy prepping
> for Ubucon.. :)
>
> I've found a number of Java projects where, in order to communicate for
> example with Kafka, they require the Kafka Java libraries for that specific
> version. For the moment, I solve this by downloading the libraries from a
> deployed Kafka installation and include them in the Charm. However, this
> has the disadvantage that everytime the Kafka charm version changes, I have
> to update the libraries in all the charms that connect to Kafka. It would
> be better if there was a way to send these libraries over the connection.
> This way, a Charm that can connect to one version of Kafka has a very high
> chance of being able to connect to the next version.
>
> So my question is: Is there a way to send large binary files between
> Charms? Or is this problem better solved by using a subordinate
> kafka-plugin Charm like the Hadoop Charms do?
>
>
>
> Kind regards
> Merlijn Sebrechts
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Automatic retries of hooks

[Rick Harding]

+1 retries are great, with backoff, when you know you're doing it because
you have experience that certain api requests to clouds, or to other known
failure points.

Blindly just saying "if at first you don't succeed, go go go" isn't a
better UX. It adds another layer of complexity in debugging, and doesn't
really improve the product. Only the charm author knows enough about what
it's trying to achieve to do intelligent retry.

In this case, if there's something about unexpected reboots of machines,
perhaps there's some specific case that Juju can grow some intelligence and
hint at the charm author what happened. The charm can then react to that
information as it deems necessary.

On Wed, Jan 20, 2016 at 8:42 AM Dean Henrichsmeyer 
wrote:

> Hi,
>
> It seems the original point James was making is getting missed. No one is
> arguing over the value of being able to retry and/or idempotent hooks.
> Yes, you should be able to retry them and yes nothing should break if you
> run them over and over.
>
> The point made is that Juju shouldn't be automatically retrying them. The
> argument of "no one knows what went wrong so Juju automatically retrying
> them is a better experience" doesn't work. The intelligence of the stack in
> question, regardless of what it is, goes in the charms. If you start
> conflating and mixing up where the intelligence goes then creating,
> running, and debugging those distributed systems will be a nightmare.
>
> The magic should only be in Juju's ability to effectively drive the models
> and intelligence encoded in the charms. It shouldn't make assumptions about
> what that intelligence is or what those models require.
>
> Thanks.
>
>
> -Dean
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Sending binaries over relations

[Merlijn Sebrechts]

Hi


I have a question I'd like to discuss, if you guys aren't to busy prepping
for Ubucon.. :)

I've found a number of Java projects where, in order to communicate for
example with Kafka, they require the Kafka Java libraries for that specific
version. For the moment, I solve this by downloading the libraries from a
deployed Kafka installation and include them in the Charm. However, this
has the disadvantage that everytime the Kafka charm version changes, I have
to update the libraries in all the charms that connect to Kafka. It would
be better if there was a way to send these libraries over the connection.
This way, a Charm that can connect to one version of Kafka has a very high
chance of being able to connect to the next version.

So my question is: Is there a way to send large binary files between
Charms? Or is this problem better solved by using a subordinate
kafka-plugin Charm like the Hadoop Charms do?



Kind regards
Merlijn Sebrechts
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Automatic retries of hooks

[Dean Henrichsmeyer]

Hi,

It seems the original point James was making is getting missed. No one is
arguing over the value of being able to retry and/or idempotent hooks. Yes,
you should be able to retry them and yes nothing should break if you run
them over and over.

The point made is that Juju shouldn't be automatically retrying them. The
argument of "no one knows what went wrong so Juju automatically retrying
them is a better experience" doesn't work. The intelligence of the stack in
question, regardless of what it is, goes in the charms. If you start
conflating and mixing up where the intelligence goes then creating,
running, and debugging those distributed systems will be a nightmare.

The magic should only be in Juju's ability to effectively drive the models
and intelligence encoded in the charms. It shouldn't make assumptions about
what that intelligence is or what those models require.

Thanks.

-Dean
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev