Re: Faster LXD bootstraps and provisioning

[Reed O'Brien]

On Mon, Aug 15, 2016 at 10:30 PM John Meinel  wrote:

> ...
>>
>
>
>> +### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in
>> +-A ufw-user-input -p tcp --dport 8000 -j ACCEPT
>> +-A ufw-user-input -p udp --dport 8000 -j ACCEPT
>> +
>>
>>
> If I'm reading this one correctly, it also means that anyone from *any* IP
> address (not restricted to your local network). So anyone that can get to
> port 8000 on your machine can proxy to any other public website. Now, I'd
> guess that you also run a NAT router so this may not actually be opening up
> an open proxy for the world to access, but it seems a little bit iffy to
> put into a general guide.
>

Good eyes! I am behind a NAT, so it doesn't matter too much. My network is
IPv6 internally (and externally) and I am not 100% on ipv6 local vs global
links and avahi. So I just made a rule to allow the port from anywhere. I
hope to make it more robust and update the wiki RSN™.
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Faster LXD bootstraps and provisioning

[Reed O'Brien]

On Mon, Aug 15, 2016 at 10:30 PM John Meinel  wrote:

> ...
>>
>
>
>> +### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in
>> +-A ufw-user-input -p tcp --dport 8000 -j ACCEPT
>> +-A ufw-user-input -p udp --dport 8000 -j ACCEPT
>> +
>>
>>
> If I'm reading this one correctly, it also means that anyone from *any* IP
> address (not restricted to your local network). So anyone that can get to
> port 8000 on your machine can proxy to any other public website. Now, I'd
> guess that you also run a NAT router so this may not actually be opening up
> an open proxy for the world to access, but it seems a little bit iffy to
> put into a general guide.
>

Good eyes! I am behind a NAT, so it doesn't matter too much. My network is
IPv6 internally (and externally) and I am not 100% on ipv6 local vs global
links and avahi. So I just made a rule to allow the port from anywhere. I
hope to make it more robust and update the wiki RSN™.
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Faster LXD bootstraps and provisioning

[Casey Marshall]

I decided it'd be easier & safer to host squid-deb-proxy in a LXD container
rather than the host. My host doesn't route inbound to LXD from other
networks, and all the Juju machines can see it.

On Tue, Aug 16, 2016 at 12:30 AM, John Meinel 
wrote:

> ...
>>
>
>
>> +### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in
>> +-A ufw-user-input -p tcp --dport 8000 -j ACCEPT
>> +-A ufw-user-input -p udp --dport 8000 -j ACCEPT
>> +
>>
>>
> If I'm reading this one correctly, it also means that anyone from *any* IP
> address (not restricted to your local network). So anyone that can get to
> port 8000 on your machine can proxy to any other public website. Now, I'd
> guess that you also run a NAT router so this may not actually be opening up
> an open proxy for the world to access, but it seems a little bit iffy to
> put into a general guide.
>
> John
> =:->
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Faster LXD bootstraps and provisioning

[Casey Marshall]

I decided it'd be easier & safer to host squid-deb-proxy in a LXD container
rather than the host. My host doesn't route inbound to LXD from other
networks, and all the Juju machines can see it.

On Tue, Aug 16, 2016 at 12:30 AM, John Meinel 
wrote:

> ...
>>
>
>
>> +### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in
>> +-A ufw-user-input -p tcp --dport 8000 -j ACCEPT
>> +-A ufw-user-input -p udp --dport 8000 -j ACCEPT
>> +
>>
>>
> If I'm reading this one correctly, it also means that anyone from *any* IP
> address (not restricted to your local network). So anyone that can get to
> port 8000 on your machine can proxy to any other public website. Now, I'd
> guess that you also run a NAT router so this may not actually be opening up
> an open proxy for the world to access, but it seems a little bit iffy to
> put into a general guide.
>
> John
> =:->
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Faster LXD bootstraps and provisioning

[John Meinel]

>
> ...
>


> +### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in
> +-A ufw-user-input -p tcp --dport 8000 -j ACCEPT
> +-A ufw-user-input -p udp --dport 8000 -j ACCEPT
> +
>
>
If I'm reading this one correctly, it also means that anyone from *any* IP
address (not restricted to your local network). So anyone that can get to
port 8000 on your machine can proxy to any other public website. Now, I'd
guess that you also run a NAT router so this may not actually be opening up
an open proxy for the world to access, but it seems a little bit iffy to
put into a general guide.

John
=:->
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Faster LXD bootstraps and provisioning

[John Meinel]

>
> ...
>


> +### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in
> +-A ufw-user-input -p tcp --dport 8000 -j ACCEPT
> +-A ufw-user-input -p udp --dport 8000 -j ACCEPT
> +
>
>
If I'm reading this one correctly, it also means that anyone from *any* IP
address (not restricted to your local network). So anyone that can get to
port 8000 on your machine can proxy to any other public website. Now, I'd
guess that you also run a NAT router so this may not actually be opening up
an open proxy for the world to access, but it seems a little bit iffy to
put into a general guide.

John
=:->
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Faster LXD bootstraps and provisioning

[John Meinel]

My concern about your last configuration is that it appears to change your
squid deb proxy into an open proxy by which anything can make a request to
any website. At least that is my understanding of:
http_access allow all

(Its no longer a 'deb' proxy, its just a plain http proxy.)
That may be what you want, but I'm not sure I would put it in "everyone who
wants a faster experience, just do this" page.

John
=:->

On Tue, Aug 16, 2016 at 2:07 AM, Reed O'Brien 
wrote:

> Yes thanks for doing the work to share this menn0. It is much appreciated.
>
> I also needed to change the following in squid-deb-proxy (even though I
> added a bunch of domains to /etc/squid-deb-proxy/mirror-
> dstdomain.acl.d/10-default):
> --- a/squid-deb-proxy/squid-deb-proxy.conf
> +++ b/squid-deb-proxy/squid-deb-proxy.conf
> @@ -80,12 +80,12 @@ http_access deny blockedpkgs
>  # allow access only to official archive mirrors
>  # uncomment the third and fouth line to permit any unlisted domain
>  #http_access deny !to_archive_mirrors
> -http_access allow !to_archive_mirrors
> +http_access allow all #!to_archive_mirrors
>
>  # don't cache domains not listed in the mirrors file
>  # uncomment the third and fourth line to cache any unlisted domains
>  #cache deny !to_archive_mirrors
> -cache allow !to_archive_mirrors
> +cache allow all
>
>  # allow access from our network and localhost
>  http_access allow allowed_networks
>
> And update some firewall rules aside from the ones shipped with squid (tcp
> only is prolly fine here):
> diff --git a/ufw/user.rules b/ufw/user.rules
> index 121b5b5..4cca3b0 100644
> --- a/ufw/user.rules
> +++ b/ufw/user.rules
> @@ -47,6 +47,10 @@
>  ### tuple ### allow tcp 2048,3128,3130,3401,4827 0.0.0.0/0 any 0.0.0.0/0
> Squid - in
>  -A ufw-user-input -p tcp -m multiport --dports 2048,3128,3130,3401,4827
> -j ACCEPT -m comment --comment 'dapp_Squid'
>
> +### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in
> +-A ufw-user-input -p tcp --dport 8000 -j ACCEPT
> +-A ufw-user-input -p udp --dport 8000 -j ACCEPT
> +
>  ### END RULES ###
>
>  ### LOGGING ###
> diff --git a/ufw/user6.rules b/ufw/user6.rules
> index 2c1aac4..d48de60 100644
> --- a/ufw/user6.rules
> +++ b/ufw/user6.rules
> @@ -47,6 +47,10 @@
>  ### tuple ### allow tcp 2048,3128,3130,3401,4827 ::/0 any ::/0 Squid - in
>  -A ufw6-user-input -p tcp -m multiport --dports 2048,3128,3130,3401,4827
> -j ACCEPT -m comment --comment 'dapp_Squid'
>
> +### tuple ### allow any 8000 ::/0 any ::/0 in
> +-A ufw6-user-input -p tcp --dport 8000 -j ACCEPT
> +-A ufw6-user-input -p udp --dport 8000 -j ACCEPT
>
> I've also added the config items to clouds.yaml and they work there. I'll
> update the wiki later this or next week.
>
>
> On Mon, Aug 15, 2016 at 2:27 PM Casey Marshall <
> casey.marsh...@canonical.com> wrote:
>
>> Menno,
>> This is great and thanks for sharing!
>>
>> In case anyone else runs into this.. charms that install from PPAs will
>> fail with this squid-deb-proxy setup. You'll need to allow archive mirrors
>> for this to work. See https://1337.tips/ubuntu-
>> cache-packages-using-squid-deb-proxy/ for an example.
>>
>> On Mon, Aug 15, 2016 at 9:31 AM, Rafael Gonzalez <
>> rafael.gonza...@canonical.com> wrote:
>>
>>> Hi Menno,
>>>
>>> Thanks for putting this together, great tips.  I recently ran into an
>>> issue which others could see as well.
>>>
>>> One may need to adjust the following for large bundle deployments on
>>> LXD.  A bundle deployment fails with errors about "Too many files open."
>>>  This will increase number of max open files:
>>>
>>> echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
>>> && sudo sysctl -p
>>>
>>>
>>> Regards,
>>>
>>> Rafael O. Gonzalez
>>> Canonical, Solutions Architect
>>> rgo...@canonical.com
>>> 1-646-481-7232
>>>
>>>
>>>
>>> On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits 
>>> wrote:
>>>
 I've put together a few tips on the wiki for speeding up bootstrap and
 provisioning times when using the Juju lxd provider. I find these
 techniques helpful when checking my work or investigating bugs - situations
 where you end up bootstrapping and deploying many times.

 https://github.com/juju/juju/wiki/Faster-LXD

 If you have your own techniques, or improvements to what I'm doing,
 please update the article.

 - Menno







 --
 Juju-dev mailing list
 juju-...@lists.ubuntu.com
 Modify settings or unsubscribe at: https://lists.ubuntu.com/
 mailman/listinfo/juju-dev


>>>
>>> --
>>> Juju-dev mailing list
>>> juju-...@lists.ubuntu.com
>>> Modify settings or unsubscribe at: https://lists.ubuntu.com/
>>> mailman/listinfo/juju-dev
>>>
>>>
>> --
>> Juju-dev mailing list
>> juju-...@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/
>> mailman/listinfo/juju-dev
>>
>
> --
> Juju-dev mailing list
> juju-...@lists.ubuntu.com
> Modify 

Re: Faster LXD bootstraps and provisioning

[Reed O'Brien]

Yes thanks for doing the work to share this menn0. It is much appreciated.

I also needed to change the following in squid-deb-proxy (even though I
added a bunch of domains to
/etc/squid-deb-proxy/mirror-dstdomain.acl.d/10-default):
--- a/squid-deb-proxy/squid-deb-proxy.conf
+++ b/squid-deb-proxy/squid-deb-proxy.conf
@@ -80,12 +80,12 @@ http_access deny blockedpkgs
 # allow access only to official archive mirrors
 # uncomment the third and fouth line to permit any unlisted domain
 #http_access deny !to_archive_mirrors
-http_access allow !to_archive_mirrors
+http_access allow all #!to_archive_mirrors

 # don't cache domains not listed in the mirrors file
 # uncomment the third and fourth line to cache any unlisted domains
 #cache deny !to_archive_mirrors
-cache allow !to_archive_mirrors
+cache allow all

 # allow access from our network and localhost
 http_access allow allowed_networks

And update some firewall rules aside from the ones shipped with squid (tcp
only is prolly fine here):
diff --git a/ufw/user.rules b/ufw/user.rules
index 121b5b5..4cca3b0 100644
--- a/ufw/user.rules
+++ b/ufw/user.rules
@@ -47,6 +47,10 @@
 ### tuple ### allow tcp 2048,3128,3130,3401,4827 0.0.0.0/0 any 0.0.0.0/0
Squid - in
 -A ufw-user-input -p tcp -m multiport --dports 2048,3128,3130,3401,4827 -j
ACCEPT -m comment --comment 'dapp_Squid'

+### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in
+-A ufw-user-input -p tcp --dport 8000 -j ACCEPT
+-A ufw-user-input -p udp --dport 8000 -j ACCEPT
+
 ### END RULES ###

 ### LOGGING ###
diff --git a/ufw/user6.rules b/ufw/user6.rules
index 2c1aac4..d48de60 100644
--- a/ufw/user6.rules
+++ b/ufw/user6.rules
@@ -47,6 +47,10 @@
 ### tuple ### allow tcp 2048,3128,3130,3401,4827 ::/0 any ::/0 Squid - in
 -A ufw6-user-input -p tcp -m multiport --dports 2048,3128,3130,3401,4827
-j ACCEPT -m comment --comment 'dapp_Squid'

+### tuple ### allow any 8000 ::/0 any ::/0 in
+-A ufw6-user-input -p tcp --dport 8000 -j ACCEPT
+-A ufw6-user-input -p udp --dport 8000 -j ACCEPT

I've also added the config items to clouds.yaml and they work there. I'll
update the wiki later this or next week.


On Mon, Aug 15, 2016 at 2:27 PM Casey Marshall 
wrote:

> Menno,
> This is great and thanks for sharing!
>
> In case anyone else runs into this.. charms that install from PPAs will
> fail with this squid-deb-proxy setup. You'll need to allow archive mirrors
> for this to work. See
> https://1337.tips/ubuntu-cache-packages-using-squid-deb-proxy/ for an
> example.
>
> On Mon, Aug 15, 2016 at 9:31 AM, Rafael Gonzalez <
> rafael.gonza...@canonical.com> wrote:
>
>> Hi Menno,
>>
>> Thanks for putting this together, great tips.  I recently ran into an
>> issue which others could see as well.
>>
>> One may need to adjust the following for large bundle deployments on
>> LXD.  A bundle deployment fails with errors about "Too many files open."
>>  This will increase number of max open files:
>>
>> echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf &&
>> sudo sysctl -p
>>
>>
>> Regards,
>>
>> Rafael O. Gonzalez
>> Canonical, Solutions Architect
>> rgo...@canonical.com
>> 1-646-481-7232
>>
>>
>>
>> On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits 
>> wrote:
>>
>>> I've put together a few tips on the wiki for speeding up bootstrap and
>>> provisioning times when using the Juju lxd provider. I find these
>>> techniques helpful when checking my work or investigating bugs - situations
>>> where you end up bootstrapping and deploying many times.
>>>
>>> https://github.com/juju/juju/wiki/Faster-LXD
>>>
>>> If you have your own techniques, or improvements to what I'm doing,
>>> please update the article.
>>>
>>> - Menno
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Juju-dev mailing list
>>> juju-...@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>>
>>
>> --
>> Juju-dev mailing list
>> juju-...@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>>
> --
> Juju-dev mailing list
> juju-...@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Faster LXD bootstraps and provisioning

[Reed O'Brien]

Yes thanks for doing the work to share this menn0. It is much appreciated.

I also needed to change the following in squid-deb-proxy (even though I
added a bunch of domains to
/etc/squid-deb-proxy/mirror-dstdomain.acl.d/10-default):
--- a/squid-deb-proxy/squid-deb-proxy.conf
+++ b/squid-deb-proxy/squid-deb-proxy.conf
@@ -80,12 +80,12 @@ http_access deny blockedpkgs
 # allow access only to official archive mirrors
 # uncomment the third and fouth line to permit any unlisted domain
 #http_access deny !to_archive_mirrors
-http_access allow !to_archive_mirrors
+http_access allow all #!to_archive_mirrors

 # don't cache domains not listed in the mirrors file
 # uncomment the third and fourth line to cache any unlisted domains
 #cache deny !to_archive_mirrors
-cache allow !to_archive_mirrors
+cache allow all

 # allow access from our network and localhost
 http_access allow allowed_networks

And update some firewall rules aside from the ones shipped with squid (tcp
only is prolly fine here):
diff --git a/ufw/user.rules b/ufw/user.rules
index 121b5b5..4cca3b0 100644
--- a/ufw/user.rules
+++ b/ufw/user.rules
@@ -47,6 +47,10 @@
 ### tuple ### allow tcp 2048,3128,3130,3401,4827 0.0.0.0/0 any 0.0.0.0/0
Squid - in
 -A ufw-user-input -p tcp -m multiport --dports 2048,3128,3130,3401,4827 -j
ACCEPT -m comment --comment 'dapp_Squid'

+### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in
+-A ufw-user-input -p tcp --dport 8000 -j ACCEPT
+-A ufw-user-input -p udp --dport 8000 -j ACCEPT
+
 ### END RULES ###

 ### LOGGING ###
diff --git a/ufw/user6.rules b/ufw/user6.rules
index 2c1aac4..d48de60 100644
--- a/ufw/user6.rules
+++ b/ufw/user6.rules
@@ -47,6 +47,10 @@
 ### tuple ### allow tcp 2048,3128,3130,3401,4827 ::/0 any ::/0 Squid - in
 -A ufw6-user-input -p tcp -m multiport --dports 2048,3128,3130,3401,4827
-j ACCEPT -m comment --comment 'dapp_Squid'

+### tuple ### allow any 8000 ::/0 any ::/0 in
+-A ufw6-user-input -p tcp --dport 8000 -j ACCEPT
+-A ufw6-user-input -p udp --dport 8000 -j ACCEPT

I've also added the config items to clouds.yaml and they work there. I'll
update the wiki later this or next week.


On Mon, Aug 15, 2016 at 2:27 PM Casey Marshall 
wrote:

> Menno,
> This is great and thanks for sharing!
>
> In case anyone else runs into this.. charms that install from PPAs will
> fail with this squid-deb-proxy setup. You'll need to allow archive mirrors
> for this to work. See
> https://1337.tips/ubuntu-cache-packages-using-squid-deb-proxy/ for an
> example.
>
> On Mon, Aug 15, 2016 at 9:31 AM, Rafael Gonzalez <
> rafael.gonza...@canonical.com> wrote:
>
>> Hi Menno,
>>
>> Thanks for putting this together, great tips.  I recently ran into an
>> issue which others could see as well.
>>
>> One may need to adjust the following for large bundle deployments on
>> LXD.  A bundle deployment fails with errors about "Too many files open."
>>  This will increase number of max open files:
>>
>> echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf &&
>> sudo sysctl -p
>>
>>
>> Regards,
>>
>> Rafael O. Gonzalez
>> Canonical, Solutions Architect
>> rgo...@canonical.com
>> 1-646-481-7232
>>
>>
>>
>> On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits 
>> wrote:
>>
>>> I've put together a few tips on the wiki for speeding up bootstrap and
>>> provisioning times when using the Juju lxd provider. I find these
>>> techniques helpful when checking my work or investigating bugs - situations
>>> where you end up bootstrapping and deploying many times.
>>>
>>> https://github.com/juju/juju/wiki/Faster-LXD
>>>
>>> If you have your own techniques, or improvements to what I'm doing,
>>> please update the article.
>>>
>>> - Menno
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com
>>> Modify settings or unsubscribe at:
>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>>
>>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>>
>>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Faster LXD bootstraps and provisioning

[Menno Smits]

Thanks Rafael. Would you mind adding this to the wiki page?

On 16 August 2016 at 02:31, Rafael Gonzalez 
wrote:

> Hi Menno,
>
> Thanks for putting this together, great tips.  I recently ran into an
> issue which others could see as well.
>
> One may need to adjust the following for large bundle deployments on LXD.
> A bundle deployment fails with errors about "Too many files open."  This
> will increase number of max open files:
>
> echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf &&
> sudo sysctl -p
>
>
> Regards,
>
> Rafael O. Gonzalez
> Canonical, Solutions Architect
> rgo...@canonical.com
> 1-646-481-7232
>
>
>
> On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits 
> wrote:
>
>> I've put together a few tips on the wiki for speeding up bootstrap and
>> provisioning times when using the Juju lxd provider. I find these
>> techniques helpful when checking my work or investigating bugs - situations
>> where you end up bootstrapping and deploying many times.
>>
>> https://github.com/juju/juju/wiki/Faster-LXD
>>
>> If you have your own techniques, or improvements to what I'm doing,
>> please update the article.
>>
>> - Menno
>>
>>
>>
>>
>>
>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>> an/listinfo/juju-dev
>>
>>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Faster LXD bootstraps and provisioning

[Menno Smits]

Good catch Casey. I've just updated the config in the gist to allow access
to any mirror or PPA (in a cleaner way than in the blog article IMO). It
seems to work well (apt-get download is nice way to test).


On 16 August 2016 at 09:27, Casey Marshall 
wrote:

> Menno,
> This is great and thanks for sharing!
>
> In case anyone else runs into this.. charms that install from PPAs will
> fail with this squid-deb-proxy setup. You'll need to allow archive mirrors
> for this to work. See https://1337.tips/ubuntu-cache-packages-using-squid-
> deb-proxy/ for an example.
>
> On Mon, Aug 15, 2016 at 9:31 AM, Rafael Gonzalez <
> rafael.gonza...@canonical.com> wrote:
>
>> Hi Menno,
>>
>> Thanks for putting this together, great tips.  I recently ran into an
>> issue which others could see as well.
>>
>> One may need to adjust the following for large bundle deployments on
>> LXD.  A bundle deployment fails with errors about "Too many files open."
>>  This will increase number of max open files:
>>
>> echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
>> && sudo sysctl -p
>>
>>
>> Regards,
>>
>> Rafael O. Gonzalez
>> Canonical, Solutions Architect
>> rgo...@canonical.com
>> 1-646-481-7232
>>
>>
>>
>> On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits 
>> wrote:
>>
>>> I've put together a few tips on the wiki for speeding up bootstrap and
>>> provisioning times when using the Juju lxd provider. I find these
>>> techniques helpful when checking my work or investigating bugs - situations
>>> where you end up bootstrapping and deploying many times.
>>>
>>> https://github.com/juju/juju/wiki/Faster-LXD
>>>
>>> If you have your own techniques, or improvements to what I'm doing,
>>> please update the article.
>>>
>>> - Menno
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Juju-dev mailing list
>>> juju-...@lists.ubuntu.com
>>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>>> an/listinfo/juju-dev
>>>
>>>
>>
>> --
>> Juju-dev mailing list
>> juju-...@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>> an/listinfo/juju-dev
>>
>>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Faster LXD bootstraps and provisioning

[Menno Smits]

Good catch Casey. I've just updated the config in the gist to allow access
to any mirror or PPA (in a cleaner way than in the blog article IMO). It
seems to work well (apt-get download is nice way to test).


On 16 August 2016 at 09:27, Casey Marshall 
wrote:

> Menno,
> This is great and thanks for sharing!
>
> In case anyone else runs into this.. charms that install from PPAs will
> fail with this squid-deb-proxy setup. You'll need to allow archive mirrors
> for this to work. See https://1337.tips/ubuntu-cache-packages-using-squid-
> deb-proxy/ for an example.
>
> On Mon, Aug 15, 2016 at 9:31 AM, Rafael Gonzalez <
> rafael.gonza...@canonical.com> wrote:
>
>> Hi Menno,
>>
>> Thanks for putting this together, great tips.  I recently ran into an
>> issue which others could see as well.
>>
>> One may need to adjust the following for large bundle deployments on
>> LXD.  A bundle deployment fails with errors about "Too many files open."
>>  This will increase number of max open files:
>>
>> echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
>> && sudo sysctl -p
>>
>>
>> Regards,
>>
>> Rafael O. Gonzalez
>> Canonical, Solutions Architect
>> rgo...@canonical.com
>> 1-646-481-7232
>>
>>
>>
>> On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits 
>> wrote:
>>
>>> I've put together a few tips on the wiki for speeding up bootstrap and
>>> provisioning times when using the Juju lxd provider. I find these
>>> techniques helpful when checking my work or investigating bugs - situations
>>> where you end up bootstrapping and deploying many times.
>>>
>>> https://github.com/juju/juju/wiki/Faster-LXD
>>>
>>> If you have your own techniques, or improvements to what I'm doing,
>>> please update the article.
>>>
>>> - Menno
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Juju-dev mailing list
>>> Juju-dev@lists.ubuntu.com
>>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>>> an/listinfo/juju-dev
>>>
>>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>> an/listinfo/juju-dev
>>
>>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Faster LXD bootstraps and provisioning

[Casey Marshall]

Menno,
This is great and thanks for sharing!

In case anyone else runs into this.. charms that install from PPAs will
fail with this squid-deb-proxy setup. You'll need to allow archive mirrors
for this to work. See
https://1337.tips/ubuntu-cache-packages-using-squid-deb-proxy/ for an
example.

On Mon, Aug 15, 2016 at 9:31 AM, Rafael Gonzalez <
rafael.gonza...@canonical.com> wrote:

> Hi Menno,
>
> Thanks for putting this together, great tips.  I recently ran into an
> issue which others could see as well.
>
> One may need to adjust the following for large bundle deployments on LXD.
> A bundle deployment fails with errors about "Too many files open."  This
> will increase number of max open files:
>
> echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf &&
> sudo sysctl -p
>
>
> Regards,
>
> Rafael O. Gonzalez
> Canonical, Solutions Architect
> rgo...@canonical.com
> 1-646-481-7232
>
>
>
> On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits 
> wrote:
>
>> I've put together a few tips on the wiki for speeding up bootstrap and
>> provisioning times when using the Juju lxd provider. I find these
>> techniques helpful when checking my work or investigating bugs - situations
>> where you end up bootstrapping and deploying many times.
>>
>> https://github.com/juju/juju/wiki/Faster-LXD
>>
>> If you have your own techniques, or improvements to what I'm doing,
>> please update the article.
>>
>> - Menno
>>
>>
>>
>>
>>
>>
>>
>> --
>> Juju-dev mailing list
>> juju-...@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>> an/listinfo/juju-dev
>>
>>
>
> --
> Juju-dev mailing list
> juju-...@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju-dev
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Faster LXD bootstraps and provisioning

[Casey Marshall]

Menno,
This is great and thanks for sharing!

In case anyone else runs into this.. charms that install from PPAs will
fail with this squid-deb-proxy setup. You'll need to allow archive mirrors
for this to work. See
https://1337.tips/ubuntu-cache-packages-using-squid-deb-proxy/ for an
example.

On Mon, Aug 15, 2016 at 9:31 AM, Rafael Gonzalez <
rafael.gonza...@canonical.com> wrote:

> Hi Menno,
>
> Thanks for putting this together, great tips.  I recently ran into an
> issue which others could see as well.
>
> One may need to adjust the following for large bundle deployments on LXD.
> A bundle deployment fails with errors about "Too many files open."  This
> will increase number of max open files:
>
> echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf &&
> sudo sysctl -p
>
>
> Regards,
>
> Rafael O. Gonzalez
> Canonical, Solutions Architect
> rgo...@canonical.com
> 1-646-481-7232
>
>
>
> On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits 
> wrote:
>
>> I've put together a few tips on the wiki for speeding up bootstrap and
>> provisioning times when using the Juju lxd provider. I find these
>> techniques helpful when checking my work or investigating bugs - situations
>> where you end up bootstrapping and deploying many times.
>>
>> https://github.com/juju/juju/wiki/Faster-LXD
>>
>> If you have your own techniques, or improvements to what I'm doing,
>> please update the article.
>>
>> - Menno
>>
>>
>>
>>
>>
>>
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>> an/listinfo/juju-dev
>>
>>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju-dev
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Faster LXD bootstraps and provisioning

[Rafael Gonzalez]

Hi Menno,

Thanks for putting this together, great tips.  I recently ran into an issue
which others could see as well.

One may need to adjust the following for large bundle deployments on LXD.
A bundle deployment fails with errors about "Too many files open."  This
will increase number of max open files:

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf &&
sudo sysctl -p


Regards,

Rafael O. Gonzalez
Canonical, Solutions Architect
rgo...@canonical.com
1-646-481-7232



On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits 
wrote:

> I've put together a few tips on the wiki for speeding up bootstrap and
> provisioning times when using the Juju lxd provider. I find these
> techniques helpful when checking my work or investigating bugs - situations
> where you end up bootstrapping and deploying many times.
>
> https://github.com/juju/juju/wiki/Faster-LXD
>
> If you have your own techniques, or improvements to what I'm doing, please
> update the article.
>
> - Menno
>
>
>
>
>
>
>
> --
> Juju-dev mailing list
> juju-...@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju-dev
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Faster LXD bootstraps and provisioning

[Rafael Gonzalez]

Hi Menno,

Thanks for putting this together, great tips.  I recently ran into an issue
which others could see as well.

One may need to adjust the following for large bundle deployments on LXD.
A bundle deployment fails with errors about "Too many files open."  This
will increase number of max open files:

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf &&
sudo sysctl -p


Regards,

Rafael O. Gonzalez
Canonical, Solutions Architect
rgo...@canonical.com
1-646-481-7232



On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits 
wrote:

> I've put together a few tips on the wiki for speeding up bootstrap and
> provisioning times when using the Juju lxd provider. I find these
> techniques helpful when checking my work or investigating bugs - situations
> where you end up bootstrapping and deploying many times.
>
> https://github.com/juju/juju/wiki/Faster-LXD
>
> If you have your own techniques, or improvements to what I'm doing, please
> update the article.
>
> - Menno
>
>
>
>
>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju-dev
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Faster LXD bootstraps and provisioning

[John Meinel]

Interesting. With the new rework of some of the model config you can
probably set some of those in the cloud definition and set some with "juju
set-model-default" so it doesn't have to be set in every bootstrap and
add-model. I'm not sure if all of it has landed, but it should certainly be
in 2.0.

John
=:->

On Aug 15, 2016 4:07 AM, "Menno Smits"  wrote:

> I've put together a few tips on the wiki for speeding up bootstrap and
> provisioning times when using the Juju lxd provider. I find these
> techniques helpful when checking my work or investigating bugs - situations
> where you end up bootstrapping and deploying many times.
>
> https://github.com/juju/juju/wiki/Faster-LXD
>
> If you have your own techniques, or improvements to what I'm doing, please
> update the article.
>
> - Menno
>
>
>
>
>
>
>
> --
> Juju-dev mailing list
> juju-...@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju-dev
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Faster LXD bootstraps and provisioning

[John Meinel]

Interesting. With the new rework of some of the model config you can
probably set some of those in the cloud definition and set some with "juju
set-model-default" so it doesn't have to be set in every bootstrap and
add-model. I'm not sure if all of it has landed, but it should certainly be
in 2.0.

John
=:->

On Aug 15, 2016 4:07 AM, "Menno Smits"  wrote:

> I've put together a few tips on the wiki for speeding up bootstrap and
> provisioning times when using the Juju lxd provider. I find these
> techniques helpful when checking my work or investigating bugs - situations
> where you end up bootstrapping and deploying many times.
>
> https://github.com/juju/juju/wiki/Faster-LXD
>
> If you have your own techniques, or improvements to what I'm doing, please
> update the article.
>
> - Menno
>
>
>
>
>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju-dev
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev