Thanks Michael for the detailed answer.
My setup has 2 k8s nodes with each around 50G RAM and a master node running
rancher & the cluster was deployed using the scripts.
kubectl get pods -all-namespaces -o wide:
NAMESPACE NAME READY
STATUS RESTARTS AGE IP NODE
default beijing-aaf-cm-8448ddc4bf-z2lz6 0/1
Running 1 2d 10.42.99.143 oom
default beijing-aaf-cs-5d4968ff8b-qcmxg 0/1
Running 0 2d 10.42.211.8 oom
default beijing-aaf-fs-6c7f5458db-694d6 1/1
Running 0 2d 10.42.95.164 oom
default beijing-aaf-gui-64cd5c45dd-v2ghf 1/1
Running 1 2d 10.42.114.239 oom
default beijing-aaf-hello-64cc6f9775-xlgx2 1/1
Running 1 2d 10.42.198.193 oom
default beijing-aaf-locate-746c4dbbf5-x6lxd 1/1
Running 0 2d 10.42.177.144 oom
default beijing-aaf-oauth-cbc4dd94-46bs7 0/1
Running 2 2d 10.42.145.153 oom
default beijing-aaf-service-7477475f78-25t49 0/1
Running 2 2d 10.42.178.98 oom
default beijing-aaf-sms-674d5b9bdd-fj7xt 1/1
Running 3 2d 10.42.234.206 oom
default beijing-aaf-sms-quorumclient-0 1/1
Running 0 2d 10.42.252.144 oom
default beijing-aaf-sms-quorumclient-1 1/1
Running 0 2d 10.42.18.61 oom-1
default beijing-aaf-sms-quorumclient-2 1/1
Running 0 2d 10.42.183.35 oom-1
default beijing-aaf-sms-vault-0 2/2
Running 1 2d 10.42.82.54 oom
default beijing-aai-795958c985-92krm 1/1
Running 3 2d 10.42.90.176 oom
default beijing-aai-babel-555c485c8f-zlq65 2/2
Running 0 2d 10.42.148.162 oom
default beijing-aai-cassandra-0 0/1
Running 1 2d 10.42.51.213 oom
default beijing-aai-cassandra-1 1/1
Running 0 2d 10.42.114.132 oom-1
default beijing-aai-cassandra-2 1/1
Running 0 2d 10.42.209.20 oom-1
default beijing-aai-champ-79fff8f5b9-nj5pn 1/1
Running 0 2d 10.42.18.51 oom
default beijing-aai-data-router-9b86999fb-7f75h 1/1
Running 0 2d 10.42.184.131 oom
default beijing-aai-elasticsearch-56f6588bc6-kxjjg 1/1
Running 3 2d 10.42.110.10 oom
default beijing-aai-gizmo-76c5bdb676-85cv4 2/2
Running 3 2d 10.42.129.250 oom
default beijing-aai-hbase-6b95d5df68-zqgbs 1/1
Running 0 2d 10.42.173.156 oom
default beijing-aai-modelloader-bbfcf8d47-8frl7 2/2
Running 0 2d 10.42.62.160 oom
default beijing-aai-resources-6d789d65d7-nrcmj 2/2
Running 0 2d 10.42.207.22 oom
default beijing-aai-search-data-75bd845479-zjpvj 2/2
Running 2 2d 10.42.83.89 oom
default beijing-aai-sparky-be-7f786b5445-hw4z6 2/2
Running 2 2d 10.42.181.219 oom
default beijing-aai-traversal-699575dc59-zcmq7 2/2
Running 0 2d 10.42.191.223 oom
default beijing-aai-traversal-update-query-data-55cmk 0/1
Init:0/1 3 2d 10.42.126.230 oom
default beijing-appc-0 1/2
Running 0 2d 10.42.94.217 oom
default beijing-appc-cdt-7984476bb9-nxm45 1/1
Running 2 2d 10.42.117.183 oom
default beijing-appc-db-0 2/2
Running 1 2d 10.42.67.225 oom
default beijing-appc-dgbuilder-67bffd9767-vv5sz 1/1
Running 0 2d 10.42.95.143 oom
default beijing-brmsgw-7cff4dbc49-vj54q 1/1
Running 0 2d 10.42.67.190 oom
default beijing-clamp-6d77fdb84c-5c585 1/2
Running 3 2d 10.42.136.198 oom
default beijing-clamp-dash-es-5ff5b8d4df-qlhfx 1/1
Running 7 2d 10.42.164.20 oom
default beijing-clamp-dash-kibana-7bc8448868-dtgzg 1/1
Running 3 2d 10.42.164.253 oom
default beijing-clamp-dash-logstash-5d9655d9b4-bf2jl 1/1
Running 1 2d 10.42.154.56 oom
[70/4769]
default beijing-clampdb-7fbf7bb7cd-44999 1/1
Running 0 2d 10.42.118.44 oom
default beijing-cli-56b88c4777-rpwf4 1/1
Running 1 2d 10.42.5.110 oom
default beijing-consul-6cbf889c7-2hfph 1/1
Running 0 2d 10.42.194.238 oom
default beijing-consul-server-0 1/1
Running 1 2d 10.42.60.25 oom
default beijing-consul-server-1 1/1
Running 0 2d 10.42.101.16 oom-1
default beijing-consul-server-2 1/1
Running 0 2d 10.42.202.136 oom-1
default beijing-dbc-pg-0 1/1
Running 0 2d 10.42.53.66 oom
default beijing-dbc-pg-1 1/1
Running 0 2d 10.42.196.245 oom-1
default beijing-dcae-bootstrap-tg8vv 1/1
Running 0 2d 10.42.113.150 oom-1
default beijing-dcae-cloudify-manager-84ff9dc4cf-scvgp 1/1
Running 0 2d 10.42.70.160 oom
default beijing-dcae-db-0 1/1
Running 0 2d 10.42.169.130 oom
default beijing-dcae-db-1 1/1
Running 0 2d 10.42.73.229 oom-1
default beijing-dcae-healthcheck-75f4f8b749-p422m 1/1
Running 0 2d 10.42.70.7 oom
default beijing-dcae-redis-0 1/1
Running 1 2d 10.42.157.112 oom
default beijing-dcae-redis-1 1/1
Running 0 2d 10.42.128.196 oom-1
default beijing-dcae-redis-2 1/1
Running 0 2d 10.42.68.233 oom-1
default beijing-dcae-redis-3 1/1
Running 0 2d 10.42.126.97 oom-1
default beijing-dcae-redis-4 1/1
Running 0 2d 10.42.158.207 oom-1
default beijing-dcae-redis-5 1/1
Running 0 2d 10.42.124.108 oom-1
default beijing-dmaap-bus-controller-6576695765-96dzt 1/1
Running 0 2d 10.42.255.83 oom
default beijing-drools-0 1/1
Running 0 2d 10.42.42.24 oom
default beijing-esr-7d6bcbf866-48p4p 2/2
Running 0 2d 10.42.252.204 oom
default beijing-esr-gui-5565d89d97-wfqp9 1/1
Running 1 2d 10.42.132.172 oom
default beijing-kube2msb-55f6dbccb4-n9pgc 1/1
Running 0 2d 10.42.68.202 oom
default beijing-log-elasticsearch-744bc5d76c-7hprs 1/1
Running 2 2d 10.42.237.207 oom
default beijing-log-kibana-7759cb4d97-gxzh8 1/1
Running 0 2d 10.42.247.177 oom
default beijing-log-logstash-n7p8c 1/1
Running 0 2d 10.42.103.140 oom-1
default beijing-log-logstash-s8n2p 1/1
Running 1 2d 10.42.228.48 oom
default beijing-message-router-7bbdf95554-5rkmm 1/1
Running 3 2d 10.42.33.104 oom
default beijing-message-router-kafka-769dd687c9-mdlgt 1/1
Running 1 2d 10.42.12.171 oom
default beijing-message-router-zookeeper-56775b8d99-mgxr2 1/1
Running 0 2d 10.42.221.8 oom
default beijing-msb-consul-79586bcfdb-dg6kt 1/1
Running 1 2d 10.42.179.32 oom
default beijing-msb-discovery-5df747d7b5-vhb62 2/2
Running 1 2d 10.42.171.54 oom
default beijing-msb-eag-7fb589f867-7lbwh 2/2
Running 0 2d 10.42.227.205 oom
default beijing-msb-iag-76fc598878-758ps 2/2
Running 1 2d 10.42.52.101 oom
default beijing-multicloud-696ccb4cf9-jvk9v 2/2
Running 1 2d 10.42.231.202 oom
default beijing-multicloud-ocata-679bf7f7b7-5bjrk 2/2
Running 1 2d 10.42.23.149 oom
default beijing-multicloud-vio-585fb94b8b-q9g6j 2/2
Running 1 2d 10.42.133.221 oom
default beijing-multicloud-windriver-6d964f5769-rnldg 2/2
Running 1 2d 10.42.38.128 oom
default beijing-nbi-6966768f56-zz27n 1/1
Running 0 2d 10.42.200.32 oom
default beijing-nbi-mariadb-649597584b-swvlz 1/1
Running 1 2d 10.42.51.19 oom
default beijing-nbi-mongo-0 1/1
Running 1 2d 10.42.245.111 oom
default beijing-nexus-596cd78d9f-rmdgl 1/1
Running 0 2d 10.42.40.170 oom
default beijing-oof-85c9fb6c6b-zlkbd 0/1
Init:0/1 3 2d 10.42.69.139 oom
default beijing-oof-has-api-5875c799db-l85p7 1/1
Running 1 2d 10.42.132.208 oom
default beijing-oof-has-cassandra-75bdb757dc-pdrpv 0/1
Running 2 2d 10.42.77.162 oom
default beijing-oof-has-controller-5d768d668c-m55rw 0/1
Running 1 2d 10.42.10.202 oom
default beijing-oof-has-data-5c54f59f77-j8lls 0/1
Running 1 2d 10.42.254.220 oom
default beijing-oof-has-music-779c98bf5f-v6kc4 1/1
Running 0 2d 10.42.243.240 oom
[21/4769]
default beijing-oof-has-reservation-57755c5fdf-cm2vd 0/1
Running 0 2d 10.42.143.249 oom
default beijing-oof-has-solver-7bb46cb9d7-lwtm6 0/1
Running 0 2d 10.42.49.92 oom
default beijing-oof-has-zookeeper-b6484d9c7-fd9hx 1/1
Running 1 2d 10.42.205.236 oom
default beijing-pap-85f4c6b799-8r286 2/2
Running 0 2d 10.42.102.54 oom
default beijing-pdp-0 2/2
Running 1 2d 10.42.2.112 oom
default beijing-policydb-7b5cd6f7f5-r6sw5 1/1
Running 0 2d 10.42.109.76 oom
default beijing-portal-app-76988c76b6-ftm89 2/2
Running 1 2d 10.42.19.217 oom
default beijing-portal-cassandra-5db6df777b-74vr2 0/1
Running 1 2d 10.42.198.7 oom
default beijing-portal-db-747946fb6b-zhwn2 1/1
Running 1 2d 10.42.252.108 oom
default beijing-portal-sdk-86b5589674-6lgdb 2/2
Running 1 2d 10.42.173.203 oom
default beijing-portal-widget-58dc6f974-4nvqv 1/1
Running 3 2d 10.42.23.108 oom
default beijing-portal-zookeeper-59bf48d575-chjt5 1/1
Running 0 2d 10.42.36.241 oom
default beijing-robot-8569b89875-zcqxq 1/1
Running 1 2d 10.42.12.225 oom
default beijing-sdc-be-585c789ccb-fld49 1/2
Running 3 2d 10.42.107.255 oom
default beijing-sdc-be-config-backend-n8s5b 0/1
Running 0 2d 10.42.69.12 oom-1
default beijing-sdc-cs-549486b896-wr4vf 0/1
Running 1 2d 10.42.56.196 oom
default beijing-sdc-es-75bb7c4f6c-wgnv2 1/1
Running 1 2d 10.42.69.89 oom
default beijing-sdc-fe-6f9b95d949-djgv7 2/2
Running 2 2d 10.42.171.17 oom
default beijing-sdc-kb-5475847f94-k2nzr 1/1
Running 0 2d 10.42.48.141 oom
default beijing-sdc-onboarding-be-7b57d854f7-l8bcl 1/2
Running 0 2d 10.42.48.177 oom
default beijing-sdc-wfd-5d656797cf-n9l7r 1/1
Running 0 2d 10.42.78.190 oom
default beijing-sdnc-0 2/2
Running 0 2d 10.42.134.121 oom
default beijing-sdnc-ansible-server-67f7b45f77-8985s 0/1
Init:0/1 3 2d 10.42.243.243 oom-1
default beijing-sdnc-db-0 2/2
Running 1 2d 10.42.91.177 oom
default beijing-sdnc-dgbuilder-69c8dcc4ff-djwjv 1/1
Running 0 2d 10.42.58.67 oom
default beijing-sdnc-dmaap-listener-57ddf5f77f-q47h4 0/1
Init:0/1 3 2d 10.42.238.22 oom-1
default beijing-sdnc-portal-5bb8c9cfbb-9qbjc 0/1
Init:0/1 3 2d 10.42.219.235 oom-1
default beijing-sdnc-ueb-listener-6cd9595d86-99jl4 0/1
Init:0/1 3 2d 10.42.63.137 oom-1
default beijing-sniro-emulator-85ff675fc7-p4mqx 1/1
Running 0 2d 10.42.73.74 oom-1
default beijing-so-5df9bfbb95-r9kfh 2/2
Running 0 2d 10.42.77.78 oom-1
default beijing-so-db-54c78d8dc4-s8l99 1/1
Running 0 2d 10.42.9.98 oom-1
default beijing-uui-69ffdfb987-g6wjn 1/1
Running 0 2d 10.42.244.166 oom-1
default beijing-uui-server-7b7b8d5767-wk9g7 1/1
Running 0 2d 10.42.247.150 oom-1
default beijing-vfc-catalog-69895874bd-ffrxw 2/2
Running 0 2d 10.42.254.221 oom-1
default beijing-vfc-ems-driver-665848f646-gcplw 1/1
Running 268 2d 10.42.42.159 oom-1
default beijing-vfc-generic-vnfm-driver-5dcbf4bcd7-9d4xx 2/2
Running 0 2d 10.42.37.200 oom-1
default beijing-vfc-huawei-vnfm-driver-69cf6c5cb8-t2pm6 2/2
Running 0 2d 10.42.203.233 oom-1
default beijing-vfc-juju-vnfm-driver-5cc9fdf89c-dw7gq 2/2
Running 0 2d 10.42.229.131 oom-1
default beijing-vfc-multivim-proxy-5d9495d65c-lccn2 1/1
Running 262 2d 10.42.184.226 oom-1
default beijing-vfc-nokia-v2vnfm-driver-6fd57475d-n5krh 1/1
Running 0 2d 10.42.60.153 oom-1
default beijing-vfc-nokia-vnfm-driver-55f9fd6dbb-h9c9c 2/2
Running 0 2d 10.42.252.174 oom-1
default beijing-vfc-nslcm-79cc864765-sw8v4 2/2
Running 0 2d 10.42.42.118 oom-1
default beijing-vfc-resmgr-7477c95994-phk4z 2/2
Running 0 2d 10.42.37.218 oom-1
default beijing-vfc-vnflcm-fdb8cc87c-fzs2x 2/2
Running 0 2d 10.42.152.72 oom-1
default beijing-vfc-vnfmgr-777bc7b89c-kr6s4 2/2
Running 0 2d 10.42.113.189 oom-1
default beijing-vfc-vnfres-6dcfdcc58-8gfhq 2/2
Running 0 2d 10.42.199.120 oom-1
default beijing-vfc-workflow-5c657b7fff-86g2t 1/1
Running 11 2d 10.42.96.213 oom-1
default beijing-vfc-workflow-engine-64746f6cc6-ql5hl 1/1
Running 11 2d 10.42.58.177 oom-1
default beijing-vfc-zte-sdnc-driver-69d9d5bb8f-lmnl9 1/1
Running 263 2d 10.42.139.138 oom-1
default beijing-vfc-zte-vnfm-driver-589d4dd5fd-fn9qt 2/2
Running 0 2d 10.42.221.94 oom-1
default beijing-vid-7658fcdbb5-7tvbh 2/2
Running 0 2d 10.42.198.126 oom-1
default beijing-vid-mariadb-galera-0 0/1
Running 0 2d 10.42.226.147 oom
default beijing-vnfsdk-79dbb78cd7-njldf 1/1
Running 0 2d 10.42.15.6 oom-1
default beijing-vnfsdk-postgres-7d599d77d8-ghsgq 1/1
Running 0 2d 10.42.181.214 oom-1
kube-system heapster-76b8cd7b5-d2k95 1/1
Running 2 2d 10.42.241.163 oom
kube-system kube-dns-5d7b4487c9-wzn74 3/3
Running 8 2d 10.42.202.20 oom
kube-system kubernetes-dashboard-f9577fffd-k26gw 1/1
Running 5 2d 10.42.31.13 oom
kube-system monitoring-grafana-997796fcf-hzkl2 1/1
Running 1 2d 10.42.119.143 oom
kube-system monitoring-influxdb-56fdcd96b-4fsfj 1/1
Running 1 2d 10.42.231.153 oom
kube-system tiller-deploy-54bcc55dd5-svgzl 1/1
Running 1 2d 10.42.194.88 oom
kubectl top nodes
root@oom:/home/vagrant/beijing/oom/kubernetes# kubectl top nodes
error: failed to unmarshall heapster response: unexpected end of JSON input
I will try your suggestion of bring up single node.
Thanks
Deepak
From: onap-discuss@lists.onap.org [mailto:onap-discuss@lists.onap.org] On
Behalf Of Michael O'Brien
Sent: Thursday, July 5, 2018 7:50 AM
To: S, Deepak <deepa...@intel.com>; onap-discuss@lists.onap.org
Subject: Re: [onap-discuss] Getting cluster unavailable error if OOM is
deployed in more than 1 node. <eom>
Hi,
Could you provide a bit more info on your cluster branch (assume
master/Casablanca oom), undercloud versions (docker/kubernetes/helm/rancher) =
(17.03/1.8.10.2.8.2/1.6.14 or 1.6.18).
How you deployed the cluster and attached the hosts - manually or via a
script.
Post details before/after you deploy some pods across the cluster like a top
output (if one of the cluster is detached - it will show here)
And a snapshot of the default(k8s+rancher) and onap namespaces "kubectl get
pods -all-namespaces -o wide
ubuntu@ip-172-31-28-152:~$ kubectl top nodes
NAME CPU(cores) CPU%
MEMORY(bytes) MEMORY%
ip-172-31-28-145.us-west-1.compute.internal 3699m 92% 26034Mi
85%
ip-172-31-21-240.us-west-1.compute.internal 3741m 93% 3872Mi
12%
ip-172-31-16-85.us-west-1.compute.internal 3997m 99% 23160Mi
75%
ip-172-31-25-15.us-west-1.compute.internal 3998m 99% 27076Mi
88%
The error you are quoting with etcd is independent of deploying ONAP pods or
any pod - if the clean cluster is not healthy then nothing will deploy
including onap containers.
Provisioning a Kubernetes cluster and deploying ONAP itself are two distinct
issues that can be done independently - I would recommend a two phase approach
1) Bring up a single node - etcd will always work in this case - run parts
of ONAP - bring up AAI or Logging for example (under the 110 pod limit)
2) Provision a cluster - keep it clean - verify all kubernetes services
are up (either via the rancher, kubernetes gui or via kubectl) - again bring up
a single pod that will scale across the nodes (use the disable all yaml and
apply a -set <component>.enabled=true on either a helm install or helm upgrade.
- use anything that has a replicaSet greater than 1 or deploy the log pod which
has a DeamonSet (container/VM).
For your reference:
There are clusters currently running in windriver/tlab and Amazon ok as a
reference.
The procedure/scripts for an openstack cluster is on
http://onap.readthedocs.io/en/latest/submodules/oom.git/docs/oom_setup_kubernetes_rancher.html
The procedure/scripts for an Amazon AWS cluster is on
https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-4NodeKubernetesClusteronAWS
thank you
/michael
From: onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org>
<onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org>> On Behalf Of
Deepak S
Sent: Tuesday, July 3, 2018 8:29 PM
To: onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org>
Subject: [onap-discuss] Getting cluster unavailable error if OOM is deployed in
more than 1 node. <eom>
Hi All,
I am getting below error if I deploy OOM in more than 1 node
Error from server: client: etcd cluster is unavailable or misconfigured; error
#0: client: etcd member http://etcd.kubernetes.rancher.internal:2379 has no
leader
Thanks
Deepak
This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,
you may review at https://www.amdocs.com/about/email-disclaimer
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#10847): https://lists.onap.org/g/onap-discuss/message/10847
Mute This Topic: https://lists.onap.org/mt/23035460/21656
Group Owner: onap-discuss+ow...@lists.onap.org
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
