Re: Faster LXD bootstraps and provisioning
On Mon, Aug 15, 2016 at 10:30 PM John Meinel wrote: > ... >> > > >> +### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in >> +-A ufw-user-input -p tcp --dport 8000 -j ACCEPT >> +-A ufw-user-input -p udp --dport 8000 -j ACCEPT >> + >> >> > If I'm reading this one correctly, it also means that anyone from *any* IP > address (not restricted to your local network). So anyone that can get to > port 8000 on your machine can proxy to any other public website. Now, I'd > guess that you also run a NAT router so this may not actually be opening up > an open proxy for the world to access, but it seems a little bit iffy to > put into a general guide. > Good eyes! I am behind a NAT, so it doesn't matter too much. My network is IPv6 internally (and externally) and I am not 100% on ipv6 local vs global links and avahi. So I just made a rule to allow the port from anywhere. I hope to make it more robust and update the wiki RSN™. -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Faster LXD bootstraps and provisioning
I decided it'd be easier & safer to host squid-deb-proxy in a LXD container rather than the host. My host doesn't route inbound to LXD from other networks, and all the Juju machines can see it. On Tue, Aug 16, 2016 at 12:30 AM, John Meinel wrote: > ... >> > > >> +### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in >> +-A ufw-user-input -p tcp --dport 8000 -j ACCEPT >> +-A ufw-user-input -p udp --dport 8000 -j ACCEPT >> + >> >> > If I'm reading this one correctly, it also means that anyone from *any* IP > address (not restricted to your local network). So anyone that can get to > port 8000 on your machine can proxy to any other public website. Now, I'd > guess that you also run a NAT router so this may not actually be opening up > an open proxy for the world to access, but it seems a little bit iffy to > put into a general guide. > > John > =:-> > > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Faster LXD bootstraps and provisioning
> > ... > > +### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in > +-A ufw-user-input -p tcp --dport 8000 -j ACCEPT > +-A ufw-user-input -p udp --dport 8000 -j ACCEPT > + > > If I'm reading this one correctly, it also means that anyone from *any* IP address (not restricted to your local network). So anyone that can get to port 8000 on your machine can proxy to any other public website. Now, I'd guess that you also run a NAT router so this may not actually be opening up an open proxy for the world to access, but it seems a little bit iffy to put into a general guide. John =:-> -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Faster LXD bootstraps and provisioning
My concern about your last configuration is that it appears to change your squid deb proxy into an open proxy by which anything can make a request to any website. At least that is my understanding of: http_access allow all (Its no longer a 'deb' proxy, its just a plain http proxy.) That may be what you want, but I'm not sure I would put it in "everyone who wants a faster experience, just do this" page. John =:-> On Tue, Aug 16, 2016 at 2:07 AM, Reed O'Brien wrote: > Yes thanks for doing the work to share this menn0. It is much appreciated. > > I also needed to change the following in squid-deb-proxy (even though I > added a bunch of domains to /etc/squid-deb-proxy/mirror- > dstdomain.acl.d/10-default): > --- a/squid-deb-proxy/squid-deb-proxy.conf > +++ b/squid-deb-proxy/squid-deb-proxy.conf > @@ -80,12 +80,12 @@ http_access deny blockedpkgs > # allow access only to official archive mirrors > # uncomment the third and fouth line to permit any unlisted domain > #http_access deny !to_archive_mirrors > -http_access allow !to_archive_mirrors > +http_access allow all #!to_archive_mirrors > > # don't cache domains not listed in the mirrors file > # uncomment the third and fourth line to cache any unlisted domains > #cache deny !to_archive_mirrors > -cache allow !to_archive_mirrors > +cache allow all > > # allow access from our network and localhost > http_access allow allowed_networks > > And update some firewall rules aside from the ones shipped with squid (tcp > only is prolly fine here): > diff --git a/ufw/user.rules b/ufw/user.rules > index 121b5b5..4cca3b0 100644 > --- a/ufw/user.rules > +++ b/ufw/user.rules > @@ -47,6 +47,10 @@ > ### tuple ### allow tcp 2048,3128,3130,3401,4827 0.0.0.0/0 any 0.0.0.0/0 > Squid - in > -A ufw-user-input -p tcp -m multiport --dports 2048,3128,3130,3401,4827 > -j ACCEPT -m comment --comment 'dapp_Squid' > > +### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in > +-A ufw-user-input -p tcp --dport 8000 -j ACCEPT > +-A ufw-user-input -p udp --dport 8000 -j ACCEPT > + > ### END RULES ### > > ### LOGGING ### > diff --git a/ufw/user6.rules b/ufw/user6.rules > index 2c1aac4..d48de60 100644 > --- a/ufw/user6.rules > +++ b/ufw/user6.rules > @@ -47,6 +47,10 @@ > ### tuple ### allow tcp 2048,3128,3130,3401,4827 ::/0 any ::/0 Squid - in > -A ufw6-user-input -p tcp -m multiport --dports 2048,3128,3130,3401,4827 > -j ACCEPT -m comment --comment 'dapp_Squid' > > +### tuple ### allow any 8000 ::/0 any ::/0 in > +-A ufw6-user-input -p tcp --dport 8000 -j ACCEPT > +-A ufw6-user-input -p udp --dport 8000 -j ACCEPT > > I've also added the config items to clouds.yaml and they work there. I'll > update the wiki later this or next week. > > > On Mon, Aug 15, 2016 at 2:27 PM Casey Marshall < > casey.marsh...@canonical.com> wrote: > >> Menno, >> This is great and thanks for sharing! >> >> In case anyone else runs into this.. charms that install from PPAs will >> fail with this squid-deb-proxy setup. You'll need to allow archive mirrors >> for this to work. See https://1337.tips/ubuntu- >> cache-packages-using-squid-deb-proxy/ for an example. >> >> On Mon, Aug 15, 2016 at 9:31 AM, Rafael Gonzalez < >> rafael.gonza...@canonical.com> wrote: >> >>> Hi Menno, >>> >>> Thanks for putting this together, great tips. I recently ran into an >>> issue which others could see as well. >>> >>> One may need to adjust the following for large bundle deployments on >>> LXD. A bundle deployment fails with errors about "Too many files open." >>> This will increase number of max open files: >>> >>> echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf >>> && sudo sysctl -p >>> >>> >>> Regards, >>> >>> Rafael O. Gonzalez >>> Canonical, Solutions Architect >>> rgo...@canonical.com >>> 1-646-481-7232 >>> >>> >>> >>> On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits >>> wrote: >>> I've put together a few tips on the wiki for speeding up bootstrap and provisioning times when using the Juju lxd provider. I find these techniques helpful when checking my work or investigating bugs - situations where you end up bootstrapping and deploying many times. https://github.com/juju/juju/wiki/Faster-LXD If you have your own techniques, or improvements to what I'm doing, please update the article. - Menno -- Juju-dev mailing list juju-...@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/ mailman/listinfo/juju-dev >>> >>> -- >>> Juju-dev mailing list >>> juju-...@lists.ubuntu.com >>> Modify settings or unsubscribe at: https://lists.ubuntu.com/ >>> mailman/listinfo/juju-dev >>> >>> >> -- >> Juju-dev mailing list >> juju-...@lists.ubuntu.com >> Modify settings or unsubscribe at: https://lists.ubuntu.com/ >> mailman/listinfo/juju-dev >> > > -- > Juju-dev mailing list > juju-...@lists.ubuntu.com > Modify settings or unsubscribe at: https://lists.ubuntu.com/ > m
Re: Faster LXD bootstraps and provisioning
Yes thanks for doing the work to share this menn0. It is much appreciated. I also needed to change the following in squid-deb-proxy (even though I added a bunch of domains to /etc/squid-deb-proxy/mirror-dstdomain.acl.d/10-default): --- a/squid-deb-proxy/squid-deb-proxy.conf +++ b/squid-deb-proxy/squid-deb-proxy.conf @@ -80,12 +80,12 @@ http_access deny blockedpkgs # allow access only to official archive mirrors # uncomment the third and fouth line to permit any unlisted domain #http_access deny !to_archive_mirrors -http_access allow !to_archive_mirrors +http_access allow all #!to_archive_mirrors # don't cache domains not listed in the mirrors file # uncomment the third and fourth line to cache any unlisted domains #cache deny !to_archive_mirrors -cache allow !to_archive_mirrors +cache allow all # allow access from our network and localhost http_access allow allowed_networks And update some firewall rules aside from the ones shipped with squid (tcp only is prolly fine here): diff --git a/ufw/user.rules b/ufw/user.rules index 121b5b5..4cca3b0 100644 --- a/ufw/user.rules +++ b/ufw/user.rules @@ -47,6 +47,10 @@ ### tuple ### allow tcp 2048,3128,3130,3401,4827 0.0.0.0/0 any 0.0.0.0/0 Squid - in -A ufw-user-input -p tcp -m multiport --dports 2048,3128,3130,3401,4827 -j ACCEPT -m comment --comment 'dapp_Squid' +### tuple ### allow any 8000 0.0.0.0/0 any 0.0.0.0/0 in +-A ufw-user-input -p tcp --dport 8000 -j ACCEPT +-A ufw-user-input -p udp --dport 8000 -j ACCEPT + ### END RULES ### ### LOGGING ### diff --git a/ufw/user6.rules b/ufw/user6.rules index 2c1aac4..d48de60 100644 --- a/ufw/user6.rules +++ b/ufw/user6.rules @@ -47,6 +47,10 @@ ### tuple ### allow tcp 2048,3128,3130,3401,4827 ::/0 any ::/0 Squid - in -A ufw6-user-input -p tcp -m multiport --dports 2048,3128,3130,3401,4827 -j ACCEPT -m comment --comment 'dapp_Squid' +### tuple ### allow any 8000 ::/0 any ::/0 in +-A ufw6-user-input -p tcp --dport 8000 -j ACCEPT +-A ufw6-user-input -p udp --dport 8000 -j ACCEPT I've also added the config items to clouds.yaml and they work there. I'll update the wiki later this or next week. On Mon, Aug 15, 2016 at 2:27 PM Casey Marshall wrote: > Menno, > This is great and thanks for sharing! > > In case anyone else runs into this.. charms that install from PPAs will > fail with this squid-deb-proxy setup. You'll need to allow archive mirrors > for this to work. See > https://1337.tips/ubuntu-cache-packages-using-squid-deb-proxy/ for an > example. > > On Mon, Aug 15, 2016 at 9:31 AM, Rafael Gonzalez < > rafael.gonza...@canonical.com> wrote: > >> Hi Menno, >> >> Thanks for putting this together, great tips. I recently ran into an >> issue which others could see as well. >> >> One may need to adjust the following for large bundle deployments on >> LXD. A bundle deployment fails with errors about "Too many files open." >> This will increase number of max open files: >> >> echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && >> sudo sysctl -p >> >> >> Regards, >> >> Rafael O. Gonzalez >> Canonical, Solutions Architect >> rgo...@canonical.com >> 1-646-481-7232 >> >> >> >> On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits >> wrote: >> >>> I've put together a few tips on the wiki for speeding up bootstrap and >>> provisioning times when using the Juju lxd provider. I find these >>> techniques helpful when checking my work or investigating bugs - situations >>> where you end up bootstrapping and deploying many times. >>> >>> https://github.com/juju/juju/wiki/Faster-LXD >>> >>> If you have your own techniques, or improvements to what I'm doing, >>> please update the article. >>> >>> - Menno >>> >>> >>> >>> >>> >>> >>> >>> -- >>> Juju-dev mailing list >>> juju-...@lists.ubuntu.com >>> Modify settings or unsubscribe at: >>> https://lists.ubuntu.com/mailman/listinfo/juju-dev >>> >>> >> >> -- >> Juju-dev mailing list >> juju-...@lists.ubuntu.com >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju-dev >> >> > -- > Juju-dev mailing list > juju-...@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Faster LXD bootstraps and provisioning
Thanks Rafael. Would you mind adding this to the wiki page? On 16 August 2016 at 02:31, Rafael Gonzalez wrote: > Hi Menno, > > Thanks for putting this together, great tips. I recently ran into an > issue which others could see as well. > > One may need to adjust the following for large bundle deployments on LXD. > A bundle deployment fails with errors about "Too many files open." This > will increase number of max open files: > > echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && > sudo sysctl -p > > > Regards, > > Rafael O. Gonzalez > Canonical, Solutions Architect > rgo...@canonical.com > 1-646-481-7232 > > > > On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits > wrote: > >> I've put together a few tips on the wiki for speeding up bootstrap and >> provisioning times when using the Juju lxd provider. I find these >> techniques helpful when checking my work or investigating bugs - situations >> where you end up bootstrapping and deploying many times. >> >> https://github.com/juju/juju/wiki/Faster-LXD >> >> If you have your own techniques, or improvements to what I'm doing, >> please update the article. >> >> - Menno >> >> >> >> >> >> >> >> -- >> Juju-dev mailing list >> juju-...@lists.ubuntu.com >> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm >> an/listinfo/juju-dev >> >> > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Faster LXD bootstraps and provisioning
Good catch Casey. I've just updated the config in the gist to allow access to any mirror or PPA (in a cleaner way than in the blog article IMO). It seems to work well (apt-get download is nice way to test). On 16 August 2016 at 09:27, Casey Marshall wrote: > Menno, > This is great and thanks for sharing! > > In case anyone else runs into this.. charms that install from PPAs will > fail with this squid-deb-proxy setup. You'll need to allow archive mirrors > for this to work. See https://1337.tips/ubuntu-cache-packages-using-squid- > deb-proxy/ for an example. > > On Mon, Aug 15, 2016 at 9:31 AM, Rafael Gonzalez < > rafael.gonza...@canonical.com> wrote: > >> Hi Menno, >> >> Thanks for putting this together, great tips. I recently ran into an >> issue which others could see as well. >> >> One may need to adjust the following for large bundle deployments on >> LXD. A bundle deployment fails with errors about "Too many files open." >> This will increase number of max open files: >> >> echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf >> && sudo sysctl -p >> >> >> Regards, >> >> Rafael O. Gonzalez >> Canonical, Solutions Architect >> rgo...@canonical.com >> 1-646-481-7232 >> >> >> >> On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits >> wrote: >> >>> I've put together a few tips on the wiki for speeding up bootstrap and >>> provisioning times when using the Juju lxd provider. I find these >>> techniques helpful when checking my work or investigating bugs - situations >>> where you end up bootstrapping and deploying many times. >>> >>> https://github.com/juju/juju/wiki/Faster-LXD >>> >>> If you have your own techniques, or improvements to what I'm doing, >>> please update the article. >>> >>> - Menno >>> >>> >>> >>> >>> >>> >>> >>> -- >>> Juju-dev mailing list >>> juju-...@lists.ubuntu.com >>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm >>> an/listinfo/juju-dev >>> >>> >> >> -- >> Juju-dev mailing list >> juju-...@lists.ubuntu.com >> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm >> an/listinfo/juju-dev >> >> > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Faster LXD bootstraps and provisioning
Menno, This is great and thanks for sharing! In case anyone else runs into this.. charms that install from PPAs will fail with this squid-deb-proxy setup. You'll need to allow archive mirrors for this to work. See https://1337.tips/ubuntu-cache-packages-using-squid-deb-proxy/ for an example. On Mon, Aug 15, 2016 at 9:31 AM, Rafael Gonzalez < rafael.gonza...@canonical.com> wrote: > Hi Menno, > > Thanks for putting this together, great tips. I recently ran into an > issue which others could see as well. > > One may need to adjust the following for large bundle deployments on LXD. > A bundle deployment fails with errors about "Too many files open." This > will increase number of max open files: > > echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && > sudo sysctl -p > > > Regards, > > Rafael O. Gonzalez > Canonical, Solutions Architect > rgo...@canonical.com > 1-646-481-7232 > > > > On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits > wrote: > >> I've put together a few tips on the wiki for speeding up bootstrap and >> provisioning times when using the Juju lxd provider. I find these >> techniques helpful when checking my work or investigating bugs - situations >> where you end up bootstrapping and deploying many times. >> >> https://github.com/juju/juju/wiki/Faster-LXD >> >> If you have your own techniques, or improvements to what I'm doing, >> please update the article. >> >> - Menno >> >> >> >> >> >> >> >> -- >> Juju-dev mailing list >> juju-...@lists.ubuntu.com >> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm >> an/listinfo/juju-dev >> >> > > -- > Juju-dev mailing list > juju-...@lists.ubuntu.com > Modify settings or unsubscribe at: https://lists.ubuntu.com/ > mailman/listinfo/juju-dev > > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Faster LXD bootstraps and provisioning
Hi Menno, Thanks for putting this together, great tips. I recently ran into an issue which others could see as well. One may need to adjust the following for large bundle deployments on LXD. A bundle deployment fails with errors about "Too many files open." This will increase number of max open files: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p Regards, Rafael O. Gonzalez Canonical, Solutions Architect rgo...@canonical.com 1-646-481-7232 On Sun, Aug 14, 2016 at 8:07 PM, Menno Smits wrote: > I've put together a few tips on the wiki for speeding up bootstrap and > provisioning times when using the Juju lxd provider. I find these > techniques helpful when checking my work or investigating bugs - situations > where you end up bootstrapping and deploying many times. > > https://github.com/juju/juju/wiki/Faster-LXD > > If you have your own techniques, or improvements to what I'm doing, please > update the article. > > - Menno > > > > > > > > -- > Juju-dev mailing list > juju-...@lists.ubuntu.com > Modify settings or unsubscribe at: https://lists.ubuntu.com/ > mailman/listinfo/juju-dev > > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Faster LXD bootstraps and provisioning
Interesting. With the new rework of some of the model config you can probably set some of those in the cloud definition and set some with "juju set-model-default" so it doesn't have to be set in every bootstrap and add-model. I'm not sure if all of it has landed, but it should certainly be in 2.0. John =:-> On Aug 15, 2016 4:07 AM, "Menno Smits" wrote: > I've put together a few tips on the wiki for speeding up bootstrap and > provisioning times when using the Juju lxd provider. I find these > techniques helpful when checking my work or investigating bugs - situations > where you end up bootstrapping and deploying many times. > > https://github.com/juju/juju/wiki/Faster-LXD > > If you have your own techniques, or improvements to what I'm doing, please > update the article. > > - Menno > > > > > > > > -- > Juju-dev mailing list > juju-...@lists.ubuntu.com > Modify settings or unsubscribe at: https://lists.ubuntu.com/ > mailman/listinfo/juju-dev > > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Faster LXD bootstraps and provisioning
I've put together a few tips on the wiki for speeding up bootstrap and provisioning times when using the Juju lxd provider. I find these techniques helpful when checking my work or investigating bugs - situations where you end up bootstrapping and deploying many times. https://github.com/juju/juju/wiki/Faster-LXD If you have your own techniques, or improvements to what I'm doing, please update the article. - Menno -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju