Re: master/CiaB: Wait for juju services to have open ports - Timeout when waiting for services
Some update. I tried to remove all the relations and re-add, but the status remains. Then I tried destroy-machine, but when adding it back it reports: ERROR machine is already provisioned. I tried to recover it but failed. Trying to get another install. Revisting the bad juju status, most of the services are missing relations. I'm wondering if it's because the related service is not ready itself? e.g. most of the service is waiting for database - mongodb, but mangodb is waiting for agent init finsih. And what does the agent mean? the charm? On Wed, Jan 4, 2017 at 8:48 PM, Mac Lin <mkl0...@gmail.com> wrote: > This is the content of juju-bad.log, result of "juju status" on my server. > Or please let me know what command should I gave. > > It's difficult for me to try them out in ansible playbook. *sigh* > > environment: manual > machines: > "0": > agent-state: started > agent-state-info: (started) > agent-version: 1.25.9 > dns-name: juju.cord.lab > instance-id: 'manual:' > series: trusty > hardware: arch=amd64 cpu-cores=8 mem=24112M > state-server-member-status: has-vote > "1": > agent-state: started > agent-state-info: (started) > agent-version: 1.25.9 > dns-name: keystone.cord.lab > instance-id: manual:keystone.cord.lab > series: trusty > hardware: arch=amd64 cpu-cores=8 mem=24112M > "2": > agent-state: started > agent-state-info: (started) > agent-version: 1.25.9 > dns-name: percona-cluster.cord.lab > instance-id: manual:percona-cluster.cord.lab > series: trusty > hardware: arch=amd64 cpu-cores=8 mem=24112M > "3": > agent-state: started > agent-state-info: (started) > agent-version: 1.25.9 > dns-name: nagios.cord.lab > instance-id: manual:nagios.cord.lab > series: trusty > hardware: arch=amd64 cpu-cores=8 mem=24112M > "4": > agent-state: started > agent-state-info: (started) > agent-version: 1.25.9 > dns-name: neutron-api.cord.lab > instance-id: manual:neutron-api.cord.lab > series: trusty > hardware: arch=amd64 cpu-cores=8 mem=24112M > "5": > agent-state: started > agent-state-info: (started) > agent-version: 1.25.9 > dns-name: nova-cloud-controller.cord.lab > instance-id: manual:nova-cloud-controller.cord.lab > series: trusty > hardware: arch=amd64 cpu-cores=8 mem=24112M > "6": > agent-state: started > agent-state-info: (started) > agent-version: 1.25.9 > dns-name: openstack-dashboard.cord.lab > instance-id: manual:openstack-dashboard.cord.lab > series: trusty > hardware: arch=amd64 cpu-cores=8 mem=24112M > "7": > agent-state: started > agent-state-info: (started) > agent-version: 1.25.9 > dns-name: rabbitmq-server.cord.lab > instance-id: manual:rabbitmq-server.cord.lab > series: trusty > hardware: arch=amd64 cpu-cores=8 mem=24112M > "8": > agent-state: started > agent-state-info: (started) > agent-version: 1.25.9 > dns-name: mongodb.cord.lab > instance-id: manual:mongodb.cord.lab > series: trusty > hardware: arch=amd64 cpu-cores=8 mem=24112M > "9": > agent-state: started > agent-state-info: (started) > agent-version: 1.25.9 > dns-name: ceilometer.cord.lab > instance-id: manual:ceilometer.cord.lab > series: trusty > hardware: arch=amd64 cpu-cores=8 mem=24112M > "10": > agent-state: started > agent-state-info: (started) > agent-version: 1.25.9 > dns-name: glance.cord.lab > instance-id: manual:glance.cord.lab > series: trusty > hardware: arch=amd64 cpu-cores=8 mem=24112M > services: > > ceilometer: > charm: cs:trusty/ceilometer-17 > exposed: false > service-status: > current: blocked > message: 'Missing relations: messaging, identity, database' > since: 03 Jan 2017 19:39:55Z > relations: > amqp: > - rabbitmq-server > ceilometer-service: > - ceilometer-agent > cluster: > - ceilometer > identity-service: > - keystone > juju-info: > - nagios > nrpe-external-master: > - nrpe > shared-db: > - mongodb > units: > ceilometer/0: > workload-status: > current: blocked > message: 'Missing relations: messaging, identity, database' > since: 03 Jan 2017 19:39:55Z > agent-status: > current: executi
Re: master/CiaB: Wait for juju services to have open ports - Timeout when waiting for services
since: 03 Jan 2017 15:06:02Z agent-status: current: allocating since: 03 Jan 2017 15:06:02Z version: 1.25.9 agent-state: pending agent-version: 1.25.9 machine: "6" public-address: openstack-dashboard.cord.lab percona-cluster: charm: cs:trusty/percona-cluster-31 exposed: false service-status: current: active message: Unit is ready since: 03 Jan 2017 17:47:40Z relations: cluster: - percona-cluster juju-info: - nrpe shared-db: - glance - keystone - neutron-api - nova-cloud-controller units: percona-cluster/0: workload-status: current: active message: Unit is ready since: 03 Jan 2017 17:47:40Z agent-status: current: executing message: running install hook since: 03 Jan 2017 14:56:46Z version: 1.25.9 agent-state: started agent-version: 1.25.9 machine: "2" public-address: percona-cluster.cord.lab rabbitmq-server: charm: cs:trusty/rabbitmq-server-42 exposed: false service-status: current: unknown message: Waiting for agent initialization to finish since: 03 Jan 2017 23:06:51Z relations: amqp: - ceilometer - neutron-api - nova-cloud-controller cluster: - rabbitmq-server nrpe-external-master: - nrpe units: rabbitmq-server/0: workload-status: current: unknown message: Waiting for agent initialization to finish since: 03 Jan 2017 23:06:51Z agent-status: current: allocating since: 03 Jan 2017 23:06:51Z version: 1.25.9 agent-state: pending agent-version: 1.25.9 machine: "7" public-address: rabbitmq-server.cord.lab On Wed, Jan 4, 2017 at 7:32 PM, Rick Harding <rick.hard...@canonical.com> wrote: > Can you paste the relations part of the juju status output that shows > those relations in place? Perhaps there's an issue with the charm logic and > breaking and re-adding the relations might kick it into gear? > > > > On Wed, Jan 4, 2017 at 2:48 AM Mac Lin <mkl0...@gmail.com> wrote: > >> Really appreciate for the help. Have been stuck here for weeks. >> >> AFAIK, no. I attached two result of "juju status" on CloudLab(good) and >> my server (bad). It's supposed to work without expose the ports. And I just >> noticed that on my server, most of the services are in blocked status. But >> it seems the missing parts are fulfilled. For example, the ceilometer is >> complaining missing messaging, identity, and database relations, but the >> relation does exist. >> >> Please let me know if any info needed. >> >> ceilometer: >> charm: cs:trusty/ceilometer-17 >> exposed: false >> service-status: >> current: blocked >> message: 'Missing relations: messaging, identity, database' >> since: 03 Jan 2017 19:39:55Z >> relations: >> amqp: >> - rabbitmq-server >> ceilometer-service: >> - ceilometer-agent >> cluster: >> - ceilometer >> identity-service: >> - keystone >> juju-info: >> - nagios >> nrpe-external-master: >> - nrpe >> shared-db: >> - mongodb >> units: >> ceilometer/0: >> workload-status: >> current: blocked >> message: 'Missing relations: messaging, identity, database' >> since: 03 Jan 2017 19:39:55Z >> agent-status: >> current: executing >> message: running install hook >> since: 03 Jan 2017 14:53:31Z >> version: 1.25.9 >> agent-state: started >> agent-version: 1.25.9 >> machine: "9" >> open-ports: >> - 8777/tcp >> public-address: ceilometer.cord.lab >> >> >> >> On Tue, Jan 3, 2017 at 8:23 PM, Rick Harding <rick.hard...@canonical.com> >> wrote: >> >> Has juju expose been run on the applications? The charm can declare that >> these ports are the ones that are opened if exposed, it's not actually set >> until the operator runs the juju expose command on the application. >> >> On Sun, Jan 1, 2017 at 2:58 AM Mac Lin <mkl0...@gmail.com> wrote: >> >> >> Hi, >> >> I'm running CORD master/cord-in-a-box.sh on an x86_64 server. The same >> script on CloudLab has no problem. >> >> If I lo
Re: master/CiaB: Wait for juju services to have open ports - Timeout when waiting for services
Really appreciate for the help. Have been stuck here for weeks. AFAIK, no. I attached two result of "juju status" on CloudLab(good) and my server (bad). It's supposed to work without expose the ports. And I just noticed that on my server, most of the services are in blocked status. But it seems the missing parts are fulfilled. For example, the ceilometer is complaining missing messaging, identity, and database relations, but the relation does exist. Please let me know if any info needed. ceilometer: charm: cs:trusty/ceilometer-17 exposed: false service-status: current: blocked message: 'Missing relations: messaging, identity, database' since: 03 Jan 2017 19:39:55Z relations: amqp: - rabbitmq-server ceilometer-service: - ceilometer-agent cluster: - ceilometer identity-service: - keystone juju-info: - nagios nrpe-external-master: - nrpe shared-db: - mongodb units: ceilometer/0: workload-status: current: blocked message: 'Missing relations: messaging, identity, database' since: 03 Jan 2017 19:39:55Z agent-status: current: executing message: running install hook since: 03 Jan 2017 14:53:31Z version: 1.25.9 agent-state: started agent-version: 1.25.9 machine: "9" open-ports: - 8777/tcp public-address: ceilometer.cord.lab On Tue, Jan 3, 2017 at 8:23 PM, Rick Harding <rick.hard...@canonical.com> wrote: > Has juju expose been run on the applications? The charm can declare that > these ports are the ones that are opened if exposed, it's not actually set > until the operator runs the juju expose command on the application. > > On Sun, Jan 1, 2017 at 2:58 AM Mac Lin <mkl0...@gmail.com> wrote: > >> >> Hi, >> >> I'm running CORD master/cord-in-a-box.sh on an x86_64 server. The same >> script on CloudLab has no problem. >> >> If I log into each failed service(lxc), I could found the port is not >> listened because service is not up, which is due to not configured >> properly. The configuration files are mostly at its initial state. >> >> How could I let juju to generate the configuration for the services >> manually? >> >> TASK [juju-finish : Wait for juju services to have open ports] >> * >> Saturday 31 December 2016 12:04:22 + (0:00:04.738) 0:09:13.981 >> * >> failed: [10.100.198.201] (item={u'service': u'ceilometer', >> u'ipv4_last_octet': 20, u'name': u'ceilometer-1', u'forwarded_ports': >> [{u'int': 8777, u'ext': 8777}], u'aliase >> s': [u'ceilometer']}) => {"elapsed": 1800, "failed": true, "item": >> {"aliases": ["ceilometer"], "forwarded_ports": [{"ext": 8777, "int": >> 8777}], "ipv4_last_octet": 20, "n >> ame": "ceilometer-1", "service": "ceilometer"}, "msg": "Timeout when >> waiting for ceilometer-1:8777"} >> failed: [10.100.198.201] (item={u'service': u'glance', >> u'ipv4_last_octet': 30, u'name': u'glance-1', u'forwarded_ports': [{u'int': >> 9292, u'ext': 9292}], u'aliases': [u'g >> lance']}) => {"elapsed": 1800, "failed": true, "item": {"aliases": >> ["glance"], "forwarded_ports": [{"ext": 9292, "int": 9292}], >> "ipv4_last_octet": 30, "name": "glance-1" >> , "service": "glance"}, "msg": "Timeout when waiting for glance-1:9292"} >> failed: [10.100.198.201] (item={u'service': u'keystone', >> u'ipv4_last_octet': 40, u'name': u'keystone-1', u'forwarded_ports': >> [{u'int': 35357, u'ext': 35357}, {u'int': 49 >> 90, u'ext': 4990}, {u'int': 5000, u'ext': 5000}], u'aliases': >> [u'keystone']}) => {"elapsed": 1800, "failed": true, "item": {"aliases": >> ["keystone"], "forwarded_ports": [ >> {"ext": 35357, "int": 35357}, {"ext": 4990, "int": 4990}, {"ext": 5000, >> "int": 5000}], "ipv4_last_octet": 40, "name": "keystone-1", "service": >> "keystone"}, "msg": "Timeo >> ut when waiting for keystone-1:35357"} >> ok: [10.100.198.201] => (item={u'service': u'nagios', u'ipv4_last_octet': >> 60, u'name': u'nagios-1', u'forwarded_ports': [{u'int': 80, u'ext': 3128}], >> u'aliases': [u'nagi >> os']}) >> fail
Fwd: master/CiaB: Wait for juju services to have open ports - Timeout when waiting for services
Hi, I'm running CORD master/cord-in-a-box.sh on an x86_64 server. The same script on CloudLab has no problem. If I log into each failed service(lxc), I could found the port is not listened because service is not up, which is due to not configured properly. The configuration files are mostly at its initial state. How could I let juju to generate the configuration for the services manually? TASK [juju-finish : Wait for juju services to have open ports] * Saturday 31 December 2016 12:04:22 + (0:00:04.738) 0:09:13.981 * failed: [10.100.198.201] (item={u'service': u'ceilometer', u'ipv4_last_octet': 20, u'name': u'ceilometer-1', u'forwarded_ports': [{u'int': 8777, u'ext': 8777}], u'aliase s': [u'ceilometer']}) => {"elapsed": 1800, "failed": true, "item": {"aliases": ["ceilometer"], "forwarded_ports": [{"ext": 8777, "int": 8777}], "ipv4_last_octet": 20, "n ame": "ceilometer-1", "service": "ceilometer"}, "msg": "Timeout when waiting for ceilometer-1:8777"} failed: [10.100.198.201] (item={u'service': u'glance', u'ipv4_last_octet': 30, u'name': u'glance-1', u'forwarded_ports': [{u'int': 9292, u'ext': 9292}], u'aliases': [u'g lance']}) => {"elapsed": 1800, "failed": true, "item": {"aliases": ["glance"], "forwarded_ports": [{"ext": 9292, "int": 9292}], "ipv4_last_octet": 30, "name": "glance-1" , "service": "glance"}, "msg": "Timeout when waiting for glance-1:9292"} failed: [10.100.198.201] (item={u'service': u'keystone', u'ipv4_last_octet': 40, u'name': u'keystone-1', u'forwarded_ports': [{u'int': 35357, u'ext': 35357}, {u'int': 49 90, u'ext': 4990}, {u'int': 5000, u'ext': 5000}], u'aliases': [u'keystone']}) => {"elapsed": 1800, "failed": true, "item": {"aliases": ["keystone"], "forwarded_ports": [ {"ext": 35357, "int": 35357}, {"ext": 4990, "int": 4990}, {"ext": 5000, "int": 5000}], "ipv4_last_octet": 40, "name": "keystone-1", "service": "keystone"}, "msg": "Timeo ut when waiting for keystone-1:35357"} ok: [10.100.198.201] => (item={u'service': u'nagios', u'ipv4_last_octet': 60, u'name': u'nagios-1', u'forwarded_ports': [{u'int': 80, u'ext': 3128}], u'aliases': [u'nagi os']}) failed: [10.100.198.201] (item={u'service': u'neutron-api', u'ipv4_last_octet': 70, u'name': u'neutron-api-1', u'forwarded_ports': [{u'int': 9696, u'ext': 9696}], u'alia ses': [u'neutron-api']}) => {"elapsed": 1800, "failed": true, "item": {"aliases": ["neutron-api"], "forwarded_ports": [{"ext": 9696, "int": 9696}], "ipv4_last_octet": 70 , "name": "neutron-api-1", "service": "neutron-api"}, "msg": "Timeout when waiting for neutron-api-1:9696"} failed: [10.100.198.201] (item={u'service': u'nova-cloud-controller', u'ipv4_last_octet': 80, u'name': u'nova-cloud-controller-1', u'forwarded_ports': [{u'int': 8774, u' ext': 8774}], u'aliases': [u'nova-cloud-controller']}) => {"elapsed": 1800, "failed": true, "item": {"aliases": ["nova-cloud-controller"], "forwarded_ports": [{"ext": 87 74, "int": 8774}], "ipv4_last_octet": 80, "name": "nova-cloud-controller-1", "service": "nova-cloud-controller"}, "msg": "Timeout when waiting for nova-cloud-controller- 1:8774"} failed: [10.100.198.201] (item={u'service': u'openstack-dashboard', u'ipv4_last_octet': 90, u'name': u'openstack-dashboard-1', u'forwarded_ports': [{u'int': 80, u'ext': 8080}], u'aliases': [u'openstack-dashboard']}) => {"elapsed": 1800, "failed": true, "item": {"aliases": ["openstack-dashboard"], "forwarded_ports": [{"ext": 8080, "int": 80}], "ipv4_last_octet": 90, "name": "openstack-dashboard-1", "service": "openstack-dashboard"}, "msg": "Timeout when waiting for openstack-dashboard-1:80"} to retry, use: --limit @/cord/build/platform-install/ cord-deploy-openstack.retry I tried to reset juju and lxc, but then it stuck at "Obtain Juju Facts for creating machines" sudo juju destroy-environment manual sudo lxc delete $(sudo lxc list | grep ^\| | grep NAME -v | cut -d' ' -f 2) --force -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju