Re: juju hangs during bootstrapping

[Andrew Wilkins]

On Fri, Sep 29, 2017 at 11:55 PM  wrote:

>
>
>
>
> *From:* Andrew Wilkins [mailto:andrew.wilk...@canonical.com]
> *Sent:* Saturday, September 30, 2017 12:23 AM
> *To:* Chen2, Dave ; juju@lists.ubuntu.com
> *Subject:* Re: juju hangs during bootstrapping
>
>
>
> On Fri, Sep 29, 2017 at 10:43 AM  wrote:
>
> Hi All,
>
>
>
> I am trying to bootstrap a MAAS cloud based on juju’s official guide (
> https://jujucharms.com/docs/2.2/clouds-maas), everything seems correct
> but after the Operation System (Ubuntu 16.04 or Ubuntu14.0) has been
> installed, juju hangs when attempting to connect to the MAAS node, here is
> what I can see from the terminal,
>
>
>
> $ juju bootstrap maas-cloud
>
> Creating Juju controller "maas-cloud" on maas-cloud
>
> Looking for packaged Juju agent version 2.2.4 for amd64
>
> Launching controller instance(s) on maas-cloud...
>
> - cka68p (arch=amd64 mem=32G cores=12)
>
> Fetching Juju GUI 2.9.2
>
> Waiting for address
>
> Attempting to connect to 10.20.3.254:22 (JUJU hangs here!)
>
>
>
> And it’s pending here forever, so I tried it again with the debug mode,
>
> $ juju bootstrap --show-log --debug --bootstrap-series=trusty maas-cloud
> maas-cloud-controller
>
>
>
> I saw some detail information like below,
>
> Attempting to connect to 10.20.3.254:22
>
> 19:33:11 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:16 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:21 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:56 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:34:32 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:35:08 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:35:43 INFO  juju.cloudconfig userdatacfg_unix.go:410 Fetching agent:
> curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time
> %{time_total}s; size %{size_download} bytes; speed %{speed_download}
> bytes/s ' --retry 10 -o $bin/tools.tar.gz <[
> https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz]>
>
>
>
> Is this the last thing logged? Try running that curl command on the
> machine manually. Perhaps there's an issue getting out to the internet.
>
> *[Dave] Yes, this is the last line I saw, our network topology is MAAS
> server can access internet (dual NIC with one NIC can access outer
> network), but each node that is deployed by MAAS/JUJU only get the IP from
> internal DHCP service, do you mean each deployed node also need access
> outer internet?*
>

The bootstrap machine requires access to the Juju agent repository, which
is on  "streams.canonical.com". It's possible to point Juju at a private
mirror to avoid the need to go to the internet.

*And… Do you have any idea about the  ssh connection refused error?*
>

That's just log spam, we should clean that up. Juju requests the MAAS
server, and then immediately starts attempting to connect via SSH. When the
server is still starting, we expect to see connection refused errors.


>
> I have no idea what’s going wrong since I can telnet to the node and ssh
> to that node is also possible, I just need type “yes” then I can login to
> the node,
>
> $ ssh ubuntu@10.20.3.254
>
> The authenticity of host ' 10.20.3.254 (10.20.3.254)' can't be established.
>
> ECDSA key fingerprint is
> SHA256:4FVm21s4dx7gc0/yDgz0+QAMGK4qWODoIqeoWtZg9RI.
>
> Are you sure you want to continue connecting (yes/no)?
>
>
>
> From the console of that node, I can find the controller’s public key has
> been injected to the node,
>
> -BEGIN SSH HOST KEY KEYS---
>
> …
>
> -END SSH KEY FINGERPRINTS
>
> …
>
> Cloud-init v. 0.7.9 finished at … Datasource DataSourceMAAS 
> [http://...:5240/MAAS/metadata/].
> Up 153.77 seconds.   (cloud-init hangs here!)
>
>
>
>
>
> I googled it and found someone said it is because “authorized-keys-path”
> is commented out in the “environments.yaml” [1], but the juju version I am
> using is “2.2.4-xenial-amd64”, the MAAS version is 2.2.2,
>
> Initially, I installed juju 1.25 and configured environments.yaml, but now
> I have uninstalled juju 1.25, removed all those file in $home/.juju/ and
> start it over again with juju 2.2.4.
>
> I really cannot figure out why it always hangs at this step, is there any
> cache persisted anywhere that masked the  “authorized-keys-path” even after
> the uninstallation of juju1.25? or there is any step I missed with juju
> 

Re: juju hangs during bootstrapping

[Andrew Wilkins]

On Sat, Sep 30, 2017 at 5:55 AM  wrote:

> Hi Narinder,
>
>
>
> Here is the log from the deployed node, actually, I can deploy Operation
> System successfully with MAAS or “juju bootstrap” but failed at some final
> steps, Our external network is exactly broken for some reason yesterday and
> won’t be recovered in short term, but I guess the network broken is
>  happened after I saw the below error message during bootstrapping.
>
> “Waiting for address
>
> Attempting to connect to 10.20.3.254:22”
>
>
>
> I can see apt-get update works from the log at the beginning, and the
> network is broken couple of hours after I saw those error message.
>
>
>
> Does the error message like below has any connection with connection
> refused error message? I am not quite sure.
>

No, they're not connected, but the failing curl command is what is
preventing bootstrap from proceeding. That's the server trying to download
the Juju agent.


>
>
> “0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>
> Fetching Juju agent version 2.2.4 for amd64
>
> Attempt 1 to download tools from
> https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz.
> ..
>
> curl: (6) Could not resolve host: streams.canonical.com
>
> “
>
>
>
> Anyway, I will try it again when our network back to normal, it would be
> great if you can see any other issue beside the network, thanks again for
> your help!
>
>
>
> Best Regards,
>
> Dave Chen
>
>
>
> *From:* Narinder Gupta [mailto:narinder.gu...@canonical.com]
> *Sent:* Friday, September 29, 2017 10:52 PM
> *To:* Chen2, Dave 
> *Cc:* juju 
> *Subject:* Re: juju hangs during bootstrapping
>
>
>
> Hi Dave,
>
> May I know which division of Dell you are working on? As i have setup
> Openstack deployed t Dell multiple time with MAAS and have not seen this
> issue so far.
>
>
>
> So please send me log /var/log/cloud-init-output.log which will let us
> know what is wrong. Also try sudo apt-get update on the bootstrap node to
> confirm you have external access.
>
>
>
> In MAAS you can always add the ssh keys to land into the installed nodes
> though.
>
>
>
>
> Thanks and Regards,
>
> Narinder Gupta (PMP)   narinder.gu...@canonical.com
>
> Canonical, Ltd.narindergupta [irc.freenode.net]
>
> +1.281.736.5150 <(281)%20736-5150>
> narindergupta2007[skype]
>
>
>
> Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com
>
>
>
> On Fri, Sep 29, 2017 at 9:42 AM,  wrote:
>
> Hi All,
>
>
>
> I am trying to bootstrap a MAAS cloud based on juju’s official guide (
> https://jujucharms.com/docs/2.2/clouds-maas), everything seems correct
> but after the Operation System (Ubuntu 16.04 or Ubuntu14.0) has been
> installed, juju hangs when attempting to connect to the MAAS node, here is
> what I can see from the terminal,
>
>
>
> $ juju bootstrap maas-cloud
>
> Creating Juju controller "maas-cloud" on maas-cloud
>
> Looking for packaged Juju agent version 2.2.4 for amd64
>
> Launching controller instance(s) on maas-cloud...
>
> - cka68p (arch=amd64 mem=32G cores=12)
>
> Fetching Juju GUI 2.9.2
>
> Waiting for address
>
> Attempting to connect to 10.20.3.254:22 (JUJU hangs here!)
>
>
>
> And it’s pending here forever, so I tried it again with the debug mode,
>
> $ juju bootstrap --show-log --debug --bootstrap-series=trusty maas-cloud
> maas-cloud-controller
>
>
>
> I saw some detail information like below,
>
> Attempting to connect to 10.20.3.254:22
>
> 19:33:11 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:16 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:21 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:56 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:34:32 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:35:08 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:35:43 INFO  juju.cloudconfig userdatacfg_unix.go:410 Fetching agent:
> curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time
> %{time_total}s; size %{size_download} bytes; speed %{speed_download}
> bytes/s ' --retry 10 -o $bin/tools.tar.gz <[
> https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz]>
>
>
>
> I have no idea what’s going wrong since I can telnet to the node and ssh
> to that node is also possible, I just need type “yes” then I 

RE: juju hangs during bootstrapping

[Narinder Gupta]

Can u try nslookup for archive.ubuntu.com looks like upstream dns issue as
well apart from network as you mentioned it was broken.

Thanks and regards,
Narinder Gupta
1-2817365150

On Sep 30, 2017 04:54,  wrote:

> Hi Narinder,
>
>
>
> Here is the log from the deployed node, actually, I can deploy Operation
> System successfully with MAAS or “juju bootstrap” but failed at some final
> steps, Our external network is exactly broken for some reason yesterday and
> won’t be recovered in short term, but I guess the network broken is
>  happened after I saw the below error message during bootstrapping.
>
> “Waiting for address
>
> Attempting to connect to 10.20.3.254:22”
>
>
>
> I can see apt-get update works from the log at the beginning, and the
> network is broken couple of hours after I saw those error message.
>
>
>
> Does the error message like below has any connection with connection
> refused error message? I am not quite sure.
>
>
>
> “0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>
> Fetching Juju agent version 2.2.4 for amd64
>
> Attempt 1 to download tools from https://streams.canonical.com/
> juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz...
>
> curl: (6) Could not resolve host: streams.canonical.com
>
> “
>
>
>
> Anyway, I will try it again when our network back to normal, it would be
> great if you can see any other issue beside the network, thanks again for
> your help!
>
>
>
> Best Regards,
>
> Dave Chen
>
>
>
> *From:* Narinder Gupta [mailto:narinder.gu...@canonical.com]
> *Sent:* Friday, September 29, 2017 10:52 PM
> *To:* Chen2, Dave 
> *Cc:* juju 
> *Subject:* Re: juju hangs during bootstrapping
>
>
>
> Hi Dave,
>
> May I know which division of Dell you are working on? As i have setup
> Openstack deployed t Dell multiple time with MAAS and have not seen this
> issue so far.
>
>
>
> So please send me log /var/log/cloud-init-output.log which will let us
> know what is wrong. Also try sudo apt-get update on the bootstrap node to
> confirm you have external access.
>
>
>
> In MAAS you can always add the ssh keys to land into the installed nodes
> though.
>
>
>
>
> Thanks and Regards,
>
> Narinder Gupta (PMP)   narinder.gu...@canonical.com
>
> Canonical, Ltd.narindergupta [irc.freenode.net]
>
> +1.281.736.5150 <(281)%20736-5150>
> narindergupta2007[skype]
>
>
>
> Ubuntu- Linux for human beings | www.ubuntu.com | www.canonical.com
>
>
>
> On Fri, Sep 29, 2017 at 9:42 AM,  wrote:
>
> Hi All,
>
>
>
> I am trying to bootstrap a MAAS cloud based on juju’s official guide (
> https://jujucharms.com/docs/2.2/clouds-maas), everything seems correct
> but after the Operation System (Ubuntu 16.04 or Ubuntu14.0) has been
> installed, juju hangs when attempting to connect to the MAAS node, here is
> what I can see from the terminal,
>
>
>
> $ juju bootstrap maas-cloud
>
> Creating Juju controller "maas-cloud" on maas-cloud
>
> Looking for packaged Juju agent version 2.2.4 for amd64
>
> Launching controller instance(s) on maas-cloud...
>
> - cka68p (arch=amd64 mem=32G cores=12)
>
> Fetching Juju GUI 2.9.2
>
> Waiting for address
>
> Attempting to connect to 10.20.3.254:22 (JUJU hangs here!)
>
>
>
> And it’s pending here forever, so I tried it again with the debug mode,
>
> $ juju bootstrap --show-log --debug --bootstrap-series=trusty maas-cloud
> maas-cloud-controller
>
>
>
> I saw some detail information like below,
>
> Attempting to connect to 10.20.3.254:22
>
> 19:33:11 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:16 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:21 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22:
> Connection refused
>
> 19:33:56 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:34:32 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:35:08 DEBUG juju.provider.common bootstrap.go:497 connection attempt
> for 10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist
>
> 19:35:43 INFO  juju.cloudconfig userdatacfg_unix.go:410 Fetching agent:
> curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time
> %{time_total}s; size %{size_download} bytes; speed %{speed_download}
> bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.
> com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz]>
>
>
>
> I have no idea what’s going wrong since I can telnet to the node and ssh
> to that node is also possible, I just need type “yes” 

RE: juju hangs during bootstrapping

[Dave.Chen]

Hi Narinder,

Here is the log from the deployed node, actually, I can deploy Operation System 
successfully with MAAS or “juju bootstrap” but failed at some final steps, Our 
external network is exactly broken for some reason yesterday and won’t be 
recovered in short term, but I guess the network broken is  happened after I 
saw the below error message during bootstrapping.
“Waiting for address
Attempting to connect to 10.20.3.254:22”

I can see apt-get update works from the log at the beginning, and the network 
is broken couple of hours after I saw those error message.

Does the error message like below has any connection with connection refused 
error message? I am not quite sure.

“0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Fetching Juju agent version 2.2.4 for amd64
Attempt 1 to download tools from 
https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz...
curl: (6) Could not resolve host: streams.canonical.com
“

Anyway, I will try it again when our network back to normal, it would be great 
if you can see any other issue beside the network, thanks again for your help!

Best Regards,
Dave Chen

From: Narinder Gupta [mailto:narinder.gu...@canonical.com]
Sent: Friday, September 29, 2017 10:52 PM
To: Chen2, Dave 
Cc: juju 
Subject: Re: juju hangs during bootstrapping

Hi Dave,
May I know which division of Dell you are working on? As i have setup Openstack 
deployed t Dell multiple time with MAAS and have not seen this issue so far.

So please send me log /var/log/cloud-init-output.log which will let us know 
what is wrong. Also try sudo apt-get update on the bootstrap node to confirm 
you have external access.

In MAAS you can always add the ssh keys to land into the installed nodes though.



Thanks and Regards,

Narinder Gupta (PMP)   
narinder.gu...@canonical.com

Canonical, Ltd.narindergupta 
[irc.freenode.net]

+1.281.736.5150narindergupta2007[skype]



Ubuntu- Linux for human beings | www.ubuntu.com | 
www.canonical.com

On Fri, Sep 29, 2017 at 9:42 AM, 
> wrote:
Hi All,

I am trying to bootstrap a MAAS cloud based on juju’s official guide 
(https://jujucharms.com/docs/2.2/clouds-maas), everything seems correct but 
after the Operation System (Ubuntu 16.04 or Ubuntu14.0) has been installed, 
juju hangs when attempting to connect to the MAAS node, here is what I can see 
from the terminal,


$ juju bootstrap maas-cloud

Creating Juju controller "maas-cloud" on maas-cloud

Looking for packaged Juju agent version 2.2.4 for amd64

Launching controller instance(s) on maas-cloud...

- cka68p (arch=amd64 mem=32G cores=12)

Fetching Juju GUI 2.9.2

Waiting for address

Attempting to connect to 10.20.3.254:22 (JUJU hangs 
here!)

And it’s pending here forever, so I tried it again with the debug mode,

$ juju bootstrap --show-log --debug --bootstrap-series=trusty maas-cloud 
maas-cloud-controller



I saw some detail information like below,

Attempting to connect to 10.20.3.254:22

19:33:11 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 
10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22: Connection refused

19:33:16 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 
10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22: Connection refused

19:33:21 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 
10.20.3.254 failed: ssh: connect to host 10.20.3.254 port 22: Connection refused

19:33:56 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 
10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist

19:34:32 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 
10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist

19:35:08 DEBUG juju.provider.common bootstrap.go:497 connection attempt for 
10.20.3.254 failed: /var/lib/juju/nonce.txt does not exist

19:35:43 INFO  juju.cloudconfig userdatacfg_unix.go:410 Fetching agent: curl 
-sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time 
%{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' 
--retry 10 -o $bin/tools.tar.gz 
<[https://streams.canonical.com/juju/tools/agent/2.2.4/juju-2.2.4-ubuntu-amd64.tgz]>

I have no idea what’s going wrong since I can telnet to the node and ssh to 
that node is also possible, I just need type “yes” then I can login to the node,
$ ssh ubuntu@10.20.3.254
The authenticity of host ' 10.20.3.254 (10.20.3.254)' can't be established.
ECDSA key fingerprint is SHA256:4FVm21s4dx7gc0/yDgz0+QAMGK4qWODoIqeoWtZg9RI.
Are you sure you