----- Original Message ----- > From: "Luke Meyer" <lme...@redhat.com> > To: "Matt Wringe" <mwri...@redhat.com> > Cc: "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com>, "users" > <users@lists.openshift.redhat.com> > Sent: Tuesday, June 14, 2016 5:10:05 PM > Subject: Re: Metrics deployment > > The readiness probe status seems like an important indicator to me: > > Readiness probe failed: cat: /etc/ld.so.conf.d/*.conf: No such file or > directory > > What could cause that failure? Or is that a red herring...
red herring. Its on the readiness probe and happens when the readiness probe should be failing for Cassandra. It looks like we have a bug in our readiness probe script, but its benign and can be ignored. > > On Tue, Jun 14, 2016 at 1:53 PM, Matt Wringe <mwri...@redhat.com> wrote: > > > ----- Original Message ----- > > > From: "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com> > > > To: "Matt Wringe" <mwri...@redhat.com> > > > Cc: users@lists.openshift.redhat.com > > > Sent: Tuesday, June 14, 2016 1:37:01 PM > > > Subject: Re: Metrics deployment > > > > > > I removed readiness probes from both hawkular-cassandra-1 & > > hawkular-metrics > > > as both status shows probes failed. > > > > You should not have to remove the probes, this indicates that something is > > wrong with your installation. > > > > > > > > It looks good now. Both containers looks and running > > > (hawkular-cassandra-1-kr8ka , hawkular-metrics-vhe3u) however > > heapster-7yl34 > > > logs still shows Could not connect to > > > https://hawkular-metrics:443/hawkular/metrics/status. Curl exit code: 6. > > > Status Code 000. > > > > > > Are we good or still had issues? > > > > > > > > > # oc get pods > > > NAME READY STATUS RESTARTS AGE > > > hawkular-cassandra-1-kr8ka 1/1 Running 0 6m > > > hawkular-metrics-vhe3u 1/1 Running 2 5m > > > heapster-7yl34 0/1 Running 2 5m > > > > > > > > > > > > > > > > > > -- > > > Srinivas Kotaru > > > > > > On 6/14/16, 10:07 AM, "Srinivas Naga Kotaru (skotaru)" < > > skot...@cisco.com> > > > wrote: > > > > > > >Matt > > > > > > > >Just want to share more info by running describe pod. > > > > > > > >It seems to be health probe failing. Do you think it is the issue? > > > > > > > > > > > > > > > ># oc describe pod hawkular-cassandra-1-it5uh > > > >Name: hawkular-cassandra-1-it5uh > > > >Namespace: openshift-infra > > > >Node: l3inpn-id2-003.cisco.com/173.36.96.16 > > > >Start Time: Tue, 14 Jun 2016 16:36:21 +0000 > > > >Labels: > > > > > > > > metrics-infra=hawkular-cassandra,name=hawkular-cassandra-1,type=hawkular-cassandra > > > >Status: Running > > > >IP: 10.1.9.2 > > > >Controllers: ReplicationController/hawkular-cassandra-1 > > > >Containers: > > > > hawkular-cassandra-1: > > > > Container ID: > > > > > > docker://17a9575eb655145859a9207f5c4bde7456f947e27188a056ff2bd08c4ce6ae5d > > > > Image: > > registry.access.redhat.com/openshift3/metrics-cassandra:latest > > > > Image ID: > > > > > > docker://ee2117c9848298ca5a0cbbce354fd4adff370435225324ab9d60cd9cd9a95c53 > > > > Ports: 9042/TCP, 9160/TCP, 7000/TCP, 7001/TCP > > > > Command: > > > > /opt/apache-cassandra/bin/cassandra-docker.sh > > > > --cluster_name=hawkular-metrics > > > > --data_volume=/cassandra_data > > > > --internode_encryption=all > > > > --require_node_auth=true > > > > --enable_client_encryption=true > > > > --require_client_auth=true > > > > --keystore_file=/secret/cassandra.keystore > > > > --keystore_password_file=/secret/cassandra.keystore.password > > > > --truststore_file=/secret/cassandra.truststore > > > > --truststore_password_file=/secret/cassandra.truststore.password > > > > --cassandra_pem_file=/secret/cassandra.pem > > > > QoS Tier: > > > > cpu: BestEffort > > > > memory: BestEffort > > > > State: Running > > > > Started: Tue, 14 Jun 2016 16:37:01 +0000 > > > > Ready: True > > > > Restart Count: 0 > > > > Readiness: exec > > [/opt/apache-cassandra/bin/cassandra-docker-ready.sh] > > > > delay=0s timeout=1s period=10s #success=1 #failure=3 > > > > Environment Variables: > > > > CASSANDRA_MASTER: true > > > > POD_NAMESPACE: openshift-infra (v1:metadata.namespace) > > > >Conditions: > > > > Type Status > > > > Ready True > > > >Volumes: > > > > cassandra-data: > > > > Type: PersistentVolumeClaim (a reference to a > > PersistentVolumeClaim in > > > > the same namespace) > > > > ClaimName: metrics-cassandra-1 > > > > ReadOnly: false > > > > hawkular-cassandra-secrets: > > > > Type: Secret (a volume populated by a Secret) > > > > SecretName: hawkular-cassandra-secrets > > > > cassandra-token-4urfd: > > > > Type: Secret (a volume populated by a Secret) > > > > SecretName: cassandra-token-4urfd > > > >Events: > > > > FirstSeen LastSeen Count From > > SubobjectPath Type Reason > > Message > > > > --------- -------- ----- ---- > > ------------- -------- ------ > > > > ------- > > > > 27m 27m 1 {default-scheduler } > > Normal > > Scheduled Successfully > > > > assigned hawkular-cassandra-1-it5uh to l3inpn-id2-003.cisco.com > > > > 27m 27m 1 {kubelet > > l3inpn-id2-003.cisco.com} > > > > spec.containers{hawkular-cassandra-1} Normal Pulling > > pulling image > > > > "registry.access.redhat.com/openshift3/metrics-cassandra:latest" > > > > 27m 27m 1 {kubelet > > l3inpn-id2-003.cisco.com} > > > > spec.containers{hawkular-cassandra-1} Normal Pulled > > Successfully pulled > > > > image "registry.access.redhat.com/openshift3/metrics-cassandra:latest > > " > > > > 27m 27m 1 {kubelet > > l3inpn-id2-003.cisco.com} > > > > spec.containers{hawkular-cassandra-1} Normal Created > > Created container > > > > with docker id 17a9575eb655 > > > > 27m 27m 1 {kubelet > > l3inpn-id2-003.cisco.com} > > > > spec.containers{hawkular-cassandra-1} Normal Started > > Started container > > > > with docker id 17a9575eb655 > > > > 27m 26m 3 {kubelet > > l3inpn-id2-003.cisco.com} > > > > spec.containers{hawkular-cassandra-1} Warning Unhealthy > > Readiness probe > > > > failed: cat: /etc/ld.so.conf.d/*.conf: No such file or directory > > > >nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: > > > >'Connection refused'. > > > >Cassandra not in the up and normal state. Current state is > > > >/opt/apache-cassandra/bin/cassandra-docker-ready.sh: line 28: [: =: > > unary > > > >operator expected > > > > > > > > > > > > > > > > > > > > > > > >-- > > > >Srinivas Kotaru > > > > > > > >On 6/14/16, 10:00 AM, "Srinivas Naga Kotaru (skotaru)" < > > skot...@cisco.com> > > > >wrote: > > > > > > > >>Matt > > > >> > > > >>DNS service is working > > > >> > > > >> > > > >>~ dig +short @master-01 kubernetes.default.svc.cluster.local > > > >>172.30.0.1 > > > >>~ dig +short @master-01 jenkins.alln-test.svc.cluster.local > > > >>172.30.85.148 > > > >>~ dig +short @master-01 cakephp-example.alln-test.svc.cluster.local > > > >>172.30.31.6 > > > >> > > > >>I captured hawkular-metrics, it shows the problem. It seems to be > > unable to > > > >>connect Cassandra cluster > > > >> > > > >># oc exec hawkular-metrics-qldd8 cat /opt/eap/standalone/log/server.log > > > >> > > > >>12:40:58,848 INFO [org.jboss.as.clustering.infinispan] (ServerService > > > >>Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem. > > > >>12:40:58,898 INFO [org.jboss.as.webservices] (ServerService Thread > > Pool -- > > > >>48) JBAS015537: Activating WebServices Extension > > > >>12:40:58,901 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 46) > > > >>JBAS010153: Node identifier property is set to the default value. > > Please > > > >>make sure it is unique. > > > >>12:40:58,902 INFO [org.jboss.as.security] (ServerService Thread Pool > > -- > > > >>44) JBAS013371: Activating Security Subsystem > > > >>12:40:58,912 INFO [org.jboss.as.naming] (ServerService Thread Pool -- > > 39) > > > >>JBAS011800: Activating Naming Subsystem > > > >>12:40:58,928 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 37) > > > >>JBAS012615: Activated the following JSF Implementations: [main, 1.2] > > > >>12:40:58,939 INFO [org.jboss.as.connector.logging] (MSC service thread > > > >>1-2) JBAS010408: Starting JCA Subsystem (IronJacamar > > > >>1.0.32.Final-redhat-1) > > > >>12:40:58,981 INFO [org.jboss.as.security] (MSC service thread 1-2) > > > >>JBAS013370: Current PicketBox version=4.1.1.Final-redhat-1 > > > >>12:40:58,961 INFO [org.jboss.as.connector.subsystems.datasources] > > > >>(ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant > > > >>driver class org.h2.Driver (version 1.3) > > > >>12:40:59,039 INFO [org.jboss.as.naming] (MSC service thread 1-2) > > > >>JBAS011802: Starting Naming Service > > > >>12:40:59,052 INFO [org.jboss.as.mail.extension] (MSC service thread > > 1-4) > > > >>JBAS015400: Bound mail session [java:jboss/mail/Default] > > > >>12:40:59,614 INFO [org.apache.coyote.http11.Http11Protocol] (MSC > > service > > > >>thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on : > > > >>http-/0.0.0.0:8444 > > > >>12:40:59,620 INFO [org.apache.coyote.http11.Http11Protocol] (MSC > > service > > > >>thread 1-2) JBWEB003001: Coyote HTTP/1.1 initializing on : > > > >>http-/0.0.0.0:8080 > > > >>12:40:59,621 INFO [org.apache.coyote.http11.Http11Protocol] (MSC > > service > > > >>thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on: http-/ > > 0.0.0.0:8444 > > > >>12:40:59,627 INFO [org.apache.coyote.http11.Http11Protocol] (MSC > > service > > > >>thread 1-2) JBWEB003000: Coyote HTTP/1.1 starting on: http-/ > > 0.0.0.0:8080 > > > >>12:40:59,654 INFO [org.jboss.as.connector.subsystems.datasources] (MSC > > > >>service thread 1-3) JBAS010400: Bound data source > > > >>[java:jboss/datasources/ExampleDS] > > > >>12:40:59,985 INFO [org.jboss.as.server.deployment] (MSC service thread > > > >>1-2) JBAS015876: Starting deployment of "activemq-rar.rar" > > (runtime-name: > > > >>"activemq-rar.rar") > > > >>12:40:59,986 INFO [org.jboss.as.server.deployment] (MSC service thread > > > >>1-1) JBAS015876: Starting deployment of > > "hawkular-metrics-api-jaxrs.war" > > > >>(runtime-name: "hawkular-metrics-api-jaxrs.war") > > > >>12:40:59,987 INFO [org.jboss.as.server.deployment.scanner] (MSC > > service > > > >>thread 1-3) JBAS015012: Started FileSystemDeploymentService for > > directory > > > >>/opt/eap/standalone/deployments > > > >>12:41:00,011 INFO [org.jboss.as.remoting] (MSC service thread 1-2) > > > >>JBAS017100: Listening on 127.0.0.1:9999 > > > >>12:41:00,030 WARN [org.jboss.as.domain.http.api] (MSC service thread > > 1-3) > > > >>JBAS015102: Unable to load console module for slot eap, disabling > > console > > > >>12:41:00,042 INFO [org.jboss.ws.common.management] (MSC service thread > > > >>1-4) JBWS022052: Starting JBoss Web Services - Stack CXF Server > > > >>4.3.5.Final-redhat-3 > > > >>12:41:00,045 INFO [org.jboss.as.remoting] (MSC service thread 1-4) > > > >>JBAS017100: Listening on 0.0.0.0:4447 > > > >>12:41:01,361 INFO [org.jboss.as.connector.deployers.RADeployer] (MSC > > > >>service thread 1-3) IJ020001: Required license terms for > > > > > >>file:/opt/eap/standalone/tmp/vfs/temp/temp912bb9d1372c33d3/activemq-rar.rar-a722089a00658182/contents/ > > > >>12:41:01,784 WARN [org.jboss.as.dependency.private] (MSC service > > thread > > > >>1-1) JBAS015867: Deployment > > "deployment.hawkular-metrics-api-jaxrs.war" is > > > >>using a private module ("org.codehaus.jackson.jackson-core-asl:main") > > > >>which may be changed or removed in future versions without notice. > > > >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service > > thread > > > >>1-1) JBAS015867: Deployment > > "deployment.hawkular-metrics-api-jaxrs.war" is > > > >>using a private module ("org.codehaus.jackson.jackson-core-asl:main") > > > >>which may be changed or removed in future versions without notice. > > > >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service > > thread > > > >>1-1) JBAS015867: Deployment > > "deployment.hawkular-metrics-api-jaxrs.war" is > > > >>using a private module ("org.codehaus.jackson.jackson-mapper-asl:main") > > > >>which may be changed or removed in future versions without notice. > > > >>12:41:01,785 WARN [org.jboss.as.dependency.private] (MSC service > > thread > > > >>1-1) JBAS015867: Deployment > > "deployment.hawkular-metrics-api-jaxrs.war" is > > > >>using a private module ("org.codehaus.jackson.jackson-mapper-asl:main") > > > >>which may be changed or removed in future versions without notice. > > > >>12:41:01,843 INFO [org.jboss.weld.deployer] (MSC service thread 1-1) > > > >>JBAS016002: Processing weld deployment hawkular-metrics-api-jaxrs.war > > > >>12:41:01,941 INFO [org.jboss.weld.deployer] (MSC service thread 1-1) > > > >>JBAS016005: Starting Services for CDI deployment: > > > >>hawkular-metrics-api-jaxrs.war > > > >>12:41:01,986 INFO [org.jboss.weld.Version] (MSC service thread 1-1) > > > >>WELD-000900 1.1.31 (redhat) > > > >>12:41:02,013 INFO [org.jboss.weld.deployer] (MSC service thread 1-3) > > > >>JBAS016008: Starting weld service for deployment > > > >>hawkular-metrics-api-jaxrs.war > > > >>12:41:02,820 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:02,835 INFO [org.jboss.web] (ServerService Thread Pool -- 51) > > > >>JBAS018210: Register web context: /hawkular/metrics > > > >>12:41:02,958 INFO [org.jboss.as.server] (ServerService Thread Pool -- > > 28) > > > >>JBAS015859: Deployed "hawkular-metrics-api-jaxrs.war" (runtime-name : > > > >>"hawkular-metrics-api-jaxrs.war") > > > >>12:41:02,959 INFO [org.jboss.as.server] (ServerService Thread Pool -- > > 28) > > > >>JBAS015859: Deployed "activemq-rar.rar" (runtime-name : > > > >>"activemq-rar.rar") > > > >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) > > JBAS015961: Http > > > >>management interface listening on http://127.0.0.1:9990/management > > > >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: > > > >>Admin console listening on http://127.0.0.1:9990 > > > >>12:41:02,994 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: > > > >>JBoss EAP 6.4.4.GA (AS 7.5.4.Final-redhat-4) started in 6593ms - > > Started > > > >>267 of 303 services (62 services are lazy, passive or on-demand) > > > >>12:41:02,999 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: > > unknown > > > >>error > > > >>12:41:02,999 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [1] Retrying > > > >>connecting to Cassandra cluster in [1]s... > > > >>12:41:04,000 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:04,001 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra > > > >>12:41:04,001 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [2] Retrying > > > >>connecting to Cassandra cluster in [2]s... > > > >>12:41:06,002 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:06,002 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra > > > >>12:41:06,003 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [3] Retrying > > > >>connecting to Cassandra cluster in [3]s... > > > >>12:41:08,203 INFO [org.jboss.resteasy.cdi.i18n] (http-/0.0.0.0:8080 > > -2) > > > >>RESTEASY006050: Found BeanManager at java:comp/BeanManager > > > >>12:41:08,299 INFO [org.hibernate.validator.internal.util.Version] > > > >>(http-/0.0.0.0:8080-2) HV000001: Hibernate Validator > > 4.3.2.Final-redhat-2 > > > >>12:41:08,418 INFO > > [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp] > > > >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting > > > >>12:41:08,421 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] > > > >>(http-/0.0.0.0:8080-2) RESTEASY000320: Deploying > > > >>javax.ws.rs.core.Application: class > > > > > >>org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp$Proxy$_$$_WeldClientProxy > > > >>12:41:08,421 INFO > > [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp] > > > >>(http-/0.0.0.0:8080-2) HAWKMETRICS200001: Hawkular Metrics starting > > > >>12:41:08,422 INFO > > [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp] > > > >>(http-/0.0.0.0:8080-2) Virtual clock is disabled > > > >>12:41:08,422 INFO > > [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp] > > > >>(http-/0.0.0.0:8080-2) Virtual clock is disabled > > > >>12:41:09,003 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:09,004 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra > > > >>12:41:09,004 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [4] Retrying > > > >>connecting to Cassandra cluster in [4]s... > > > >>12:41:13,004 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:13,123 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: > > unknown > > > >>error > > > >>12:41:13,124 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [5] Retrying > > > >>connecting to Cassandra cluster in [1]s... > > > >>12:41:14,124 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:14,124 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra > > > >>12:41:14,125 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [6] Retrying > > > >>connecting to Cassandra cluster in [2]s... > > > >>12:41:16,125 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:16,126 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra > > > >>12:41:16,126 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [7] Retrying > > > >>connecting to Cassandra cluster in [3]s... > > > >>12:41:19,126 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:19,127 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra > > > >>12:41:19,127 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [8] Retrying > > > >>connecting to Cassandra cluster in [4]s... > > > >>12:41:23,128 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:23,249 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: > > unknown > > > >>error > > > >>12:41:23,249 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [9] Retrying > > > >>connecting to Cassandra cluster in [1]s... > > > >>12:41:24,250 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:24,250 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra > > > >>12:41:24,250 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [10] Retrying > > > >>connecting to Cassandra cluster in [2]s... > > > >>12:41:26,251 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:26,251 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra > > > >>12:41:26,251 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [11] Retrying > > > >>connecting to Cassandra cluster in [3]s... > > > >>12:41:29,252 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:29,252 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra > > > >>12:41:29,253 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [12] Retrying > > > >>connecting to Cassandra cluster in [4]s... > > > >>12:41:33,253 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:33,376 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: > > unknown > > > >>error > > > >>12:41:33,377 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [13] Retrying > > > >>connecting to Cassandra cluster in [1]s... > > > >>12:41:34,377 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:34,378 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra > > > >>12:41:34,378 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [14] Retrying > > > >>connecting to Cassandra cluster in [2]s... > > > >>12:41:36,378 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:36,379 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra > > > >>12:41:36,379 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [15] Retrying > > > >>connecting to Cassandra cluster in [3]s... > > > >>12:41:39,379 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:39,380 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra > > > >>12:41:39,380 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [16] Retrying > > > >>connecting to Cassandra cluster in [4]s... > > > >>12:41:43,380 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:43,503 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra: > > unknown > > > >>error > > > >>12:41:43,504 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [17] Retrying > > > >>connecting to Cassandra cluster in [1]s... > > > >>12:41:44,504 INFO > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing > > metrics > > > >>service > > > >>12:41:44,505 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200003: Could not connect > > to > > > >>Cassandra cluster - assuming its not up yet: hawkular-cassandra > > > >>12:41:44,505 WARN > > [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] > > > >>(metricsservice-lifecycle-thread) HAWKMETRICS200004: [18] Retrying > > > >>connecting to Cassandra cluster in [2]s... > > > >> > > > >>-- > > > >>Srinivas Kotaru > > > >> > > > >>On 6/14/16, 6:06 AM, "Matt Wringe" <mwri...@redhat.com> wrote: > > > >> > > > >>>----- Original Message ----- > > > >>>> From: "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com> > > > >>>> To: "Matt Wringe" <mwri...@redhat.com> > > > >>>> Cc: users@lists.openshift.redhat.com > > > >>>> Sent: Monday, June 13, 2016 7:26:06 PM > > > >>>> Subject: Re: Metrics deployment > > > >>>> > > > >>>> Matt > > > >>>> > > > >>>> PV issue resolved. Was able to to see PV successfully bounded and > > > >>>> Casandra > > > >>>> container has been running. However, it seems puzzle not fully yet > > > >>>> solved. > > > >>> > > > >>>Are you sure the OpenShift DNS server is running? > > > >>> > > > >>>If you are running OSE 3.1, can you please follow this > > > >>>https://access.redhat.com/solutions/2329131 and see if you are now > > seeing > > > >>>errors in the Hawkular Metrics logs (essentially just run `oc exec > > > >>>hawkular-metrics-xxxxx cat /opt/eap/standalone/log/server.log`) > > > >>> > > > >>>> > > > >>>> I could see other container(heapster) not coming up, and seeing > > below > > > >>>> errors > > > >>>> > > > >>>> [skotaru@l3imas-id2-01 metrics]$ oc logs -f heapster-fnkdc > > > >>>> Endpoint Check in effect. Checking > > > >>>> https://hawkular-metrics:443/hawkular/metrics/status > > > >>>> Could not connect to > > > >>>> https://hawkular-metrics:443/hawkular/metrics/status. > > > >>>> Curl exit code: 6. Status Code 000 > > > >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not > > accessible > > > >>>> [HTTP status code: 000. Curl exit code 6]. Retrying. > > > >>>> Could not connect to > > > >>>> https://hawkular-metrics:443/hawkular/metrics/status. > > > >>>> Curl exit code: 6. Status Code 000 > > > >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not > > accessible > > > >>>> [HTTP status code: 000. Curl exit code 6]. Retrying. > > > >>>> > > > >>>> > > > >>>> # oc get pv > > > >>>> pv-5gb-0011 5Gi RWO Bound > > > >>>> openshift-infra/metrics-cassandra-1 22m > > > >>>> > > > >>>> > > > >>>> $ oc get pods > > > >>>> NAME READY STATUS RESTARTS AGE > > > >>>> hawkular-cassandra-1-2pzd7 1/1 Running 0 20m > > > >>>> hawkular-metrics-mf5qf 0/1 Running 7 20m > > > >>>> heapster-fnkdc 0/1 Error 6 20m > > > >>>> metrics-deployer-cvep0 0/1 Completed 0 21m > > > >>>> > > > >>>> # oc logs -f hawkular-metrics-mf5qf > > > >>>> > > > >>>> 19:20:00,819 INFO [org.xnio] (MSC service thread 1-2) XNIO Version > > > >>>> 3.0.14.GA-redhat-1 > > > >>>> 19:20:00,831 INFO [org.jboss.as.server] (Controller Boot Thread) > > > >>>> JBAS015888: > > > >>>> Creating http management service using socket-binding > > (management-http) > > > >>>> 19:20:00,834 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO > > > >>>> Implementation Version 3.0.14.GA-redhat-1 > > > >>>> 19:20:00,844 INFO [org.jboss.remoting] (MSC service thread 1-2) > > JBoss > > > >>>> Remoting version 3.3.5.Final-redhat-1 > > > >>>> > > > >>>> $ oc logs -f heapster-fnkdc > > > >>>> Endpoint Check in effect. Checking > > > >>>> https://hawkular-metrics:443/hawkular/metrics/status > > > >>>> Could not connect to > > > >>>> https://hawkular-metrics:443/hawkular/metrics/status. > > > >>>> Curl exit code: 6. Status Code 000 > > > >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not > > accessible > > > >>>> [HTTP status code: 000. Curl exit code 6]. Retrying. > > > >>>> Could not connect to > > > >>>> https://hawkular-metrics:443/hawkular/metrics/status. > > > >>>> Curl exit code: 6. Status Code 000 > > > >>>> 'https://hawkular-metrics:443/hawkular/metrics/status' is not > > accessible > > > >>>> [HTTP status code: 000. Curl exit code 6]. Retrying. > > > >>>> Could not connect to > > > >>>> https://hawkular-metrics:443/hawkular/metrics/status. > > > >>>> Curl exit code: 6. Status Code 000 > > > >>>> > > > >>>> $ oc logs -f hawkular-cassandra-1-2pzd7 > > > >>>> INFO 23:00:24 Starting listening for CQL clients on > > > >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9042... > > > >>>> INFO 23:00:24 Binding thrift service to > > > >>>> hawkular-cassandra-1-2pzd7/10.1.6.2:9160 > > > >>>> INFO 23:00:24 enabling encrypted thrift connections between client > > and > > > >>>> server > > > >>>> INFO 23:00:24 Listening for thrift clients... > > > >>>> INFO 23:00:26 Created default superuser role 'cassandra' > > > >>>> > > > >>>> # oc get svc > > > >>>> NAME CLUSTER-IP EXTERNAL-IP PORT(S) > > > >>>> AGE > > > >>>> hawkular-cassandra 172.30.2.13 <none> > > > >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m > > > >>>> hawkular-cassandra-nodes None <none> > > > >>>> 9042/TCP,9160/TCP,7000/TCP,7001/TCP 25m > > > >>>> hawkular-metrics 172.30.117.176 <none> 443/TCP > > > >>>> 25m > > > >>>> heapster 172.30.107.135 <none> 80/TCP > > > >>>> 25m > > > >>>> > > > >>>> #curl -I 172.30.117.176:443//hawkular/metrics/status > > > >>>> > > > >>>> HTTP/1.1 504 Gateway Timeout > > > >>>> Mime-Version: 1.0 > > > >>>> Date: Mon, 13 Jun 2016 23:25:47 GMT > > > >>>> Content-Type: text/html > > > >>>> Connection: keep-alive > > > >>>> Proxy-Connection: keep-alive > > > >>>> Content-Length: 1572 > > > >>>> > > > >>>> -- > > > >>>> Srinivas Kotaru > > > >>>> > > > >>>> On 6/13/16, 2:33 PM, "Srinivas Naga Kotaru (skotaru)" > > > >>>> <skot...@cisco.com> > > > >>>> wrote: > > > >>>> > > > >>>> >Matt > > > >>>> > > > > >>>> >That is good catch. I ran without USE_PERSISTENT_STORAGE=false and > > > >>>> >working > > > >>>> > > > > >>>> >I adjusted PV to 5Gi and reran. Will update progress. > > > >>>> > > > > >>>> >Thanks you for your help so far. > > > >>>> > > > > >>>> >-- > > > >>>> >Srinivas Kotaru > > > >>>> > > > > >>>> >On 6/13/16, 2:27 PM, "Matt Wringe" <mwri...@redhat.com> wrote: > > > >>>> > > > > >>>> >> > > > >>>> >> > > > >>>> >>----- Original Message ----- > > > >>>> >>> From: "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com> > > > >>>> >>> To: "Matt Wringe" <mwri...@redhat.com> > > > >>>> >>> Cc: users@lists.openshift.redhat.com > > > >>>> >>> Sent: Monday, June 13, 2016 5:21:01 PM > > > >>>> >>> Subject: Re: Metrics deployment > > > >>>> >>> > > > >>>> >>> Oh ok > > > >>>> >>> > > > >>>> >>> Am using PV for metrics > > > >>>> >>> > > > >>>> >>> description: "The persistent volume size for each of the > > Cassandra > > > >>>> >>> nodes" > > > >>>> >>> name: CASSANDRA_PV_SIZE > > > >>>> >>> value: "10Gi" > > > >>>> >>> > > > >>>> >>> oc get pv > > > >>>> >>> NAME CAPACITY ACCESSMODES STATUS CLAIM > > > >>>> >>> REASON > > > >>>> >>> AGE > > > >>>> >>> pv-1gb-001 1Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> pv-1gb-002 1Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> pv-1gb-003 1Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> pv-1gb-004 1Gi RWO Bound > > thlatt/mongodb > > > >>>> >>> 4d > > > >>>> >>> pv-1gb-005 1Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> pv-2gb-0010 2Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> pv-2gb-006 2Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> pv-2gb-007 2Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> pv-2gb-008 2Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> pv-2gb-009 2Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> pv-5gb-0011 5Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> pv-5gb-0012 5Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> pv-5gb-0013 5Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> pv-5gb-0014 5Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> pv-5gb-0015 5Gi RWO Available > > > >>>> >>> 4d > > > >>>> >>> > > > >>>> >>> am running with below command > > > >>>> >>> > > > >>>> >>> $ oc new-app -f metrics-deployer.yaml ( hardcoded HOSTNAME, > > > >>>> >>> MASTER_API > > > >>>> >>> and > > > >>>> >>> PV info so not passing any parameters) > > > >>>> >>> > > > >>>> >> > > > >>>> >>I would suspect that Cassandra is blocked because its waiting for > > 10Gi > > > >>>> >>PV > > > >>>> >>to become available, and none of the PV listed above are big > > enough. > > > >>>> >> > > > >>>> >>> > > > >>>> >>> -- > > > >>>> >>> Srinivas Kotaru > > > >>>> >>> > > > >>>> >>> On 6/13/16, 2:12 PM, "Matt Wringe" <mwri...@redhat.com> wrote: > > > >>>> >>> > > > >>>> >>> >----- Original Message ----- > > > >>>> >>> >> From: "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com> > > > >>>> >>> >> To: "Matt Wringe" <mwri...@redhat.com> > > > >>>> >>> >> Cc: users@lists.openshift.redhat.com > > > >>>> >>> >> Sent: Monday, June 13, 2016 4:55:55 PM > > > >>>> >>> >> Subject: Re: Metrics deployment > > > >>>> >>> >> > > > >>>> >>> >> Matt > > > >>>> >>> >> > > > >>>> >>> >> Thanks for looking into. I rerun the setup, but had the same > > > >>>> >>> >> issue > > > >>>> >>> >> > > > >>>> >>> >> # oc get pods > > > >>>> >>> >> NAME READY STATUS > > > >>>> >>> >> RESTARTS > > > >>>> >>> >> AGE > > > >>>> >>> >> hawkular-cassandra-1-y2egy 0/1 ContainerCreating 0 > > > >>>> >>> >> 5m > > > >>>> >>> >> hawkular-metrics-4b16f 0/1 Running 1 > > > >>>> >>> >> 4m > > > >>>> >>> >> heapster-x2gj2 0/1 Running 2 > > > >>>> >>> >> 4m > > > >>>> >>> >> metrics-deployer-9v7vc 0/1 Completed 0 > > > >>>> >>> >> 6m > > > >>>> >>> >> > > > >>>> >>> >> $ oc logs -f hawkular-cassandra-1-y2egy > > > >>>> >>> >> Error from server: container "hawkular-cassandra-1" in pod > > > >>>> >>> >> "hawkular-cassandra-1-y2egy" is waiting to start: > > > >>>> >>> >> ContainerCreating > > > >>>> >>> > > > > >>>> >>> >Ok, so it looks like something is blocking the Cassandra pod > > from > > > >>>> >>> >starting. > > > >>>> >>> > > > > >>>> >>> >If you are using persistent storage, Cassandra will not start > > until > > > >>>> >>> >the > > > >>>> >>> >PV > > > >>>> >>> >is available. There may be some more information about > > Cassandra in > > > >>>> >>> >the > > > >>>> >>> >pod > > > >>>> >>> >section of the console under events. > > > >>>> >>> > > > > >>>> >>> >What command did you use when deploying the deployer? > > > >>>> >>> > > > > >>>> >>> >> > > > >>>> >>> >> $ oc logs -f hawkular-metrics-4b16f > > > >>>> >>> >> > > > >>>> >>> >> 16:54:25,703 DEBUG [org.jboss.as.config] (MSC service thread > > 1-4) > > > >>>> >>> >> VM > > > >>>> >>> >> Arguments: -Duser.home=/home/jboss -Duser.name=jboss > > > >>>> >>> >> -D[Standalone] > > > >>>> >>> >> -XX:+UseCompressedOops -verbose:gc > > > >>>> >>> >> -Xloggc:/opt/eap/standalone/log/gc.log > > > >>>> >>> >> -XX:+PrintGCDetails -XX:+PrintGCDateStamps > > > >>>> >>> >> -XX:+UseGCLogFileRotation > > > >>>> >>> >> -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M > > > >>>> >>> >> -XX:-TraceClassUnloading > > > >>>> >>> >> -Xms1303m -Xmx1303m -XX:MaxPermSize=256m > > > >>>> >>> >> -Djava.net.preferIPv4Stack=true > > > >>>> >>> >> -Djboss.modules.system.pkgs=org.jboss.logmanager > > > >>>> >>> >> -Djava.awt.headless=true > > > >>>> >>> >> -Djboss.modules.policy-permissions=true > > > >>>> >>> >> > > -Xbootclasspath/p:/opt/eap/jboss-modules.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.4.Final-redhat-1.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/javax.json-1.0.4.jar:/opt/eap/modules/system/layers/base/org/jboss/logmanager/ext/main/jboss-logmanager-ext-1.0.0.Alpha2-redhat-1.jar > > > >>>> >>> >> -Djava.util.logging.manager=org.jboss.logmanager.LogManager > > > >>>> >>> >> > > -javaagent:/opt/eap/jolokia.jar=port=8778,protocol=https,caCert=/var/run/secrets/ > > kubernetes.io/serviceaccount/ca.crt,clientPrincipal=cn=system:master-proxy,useSslClientAuthentication=true,extraClientCheck=true,host=0.0.0.0,discoveryEnabled=false > > > >>>> >>> >> -Djava.security.egd=file:/dev/./urandom > > > >>>> >>> >> -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log > > > >>>> >>> >> > > -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties > > > >>>> >>> >> 16:54:27,079 INFO [org.xnio] (MSC service thread 1-3) XNIO > > > >>>> >>> >> Version > > > >>>> >>> >> 3.0.14.GA-redhat-1 > > > >>>> >>> >> 16:54:27,083 INFO [org.xnio.nio] (MSC service thread 1-3) > > XNIO > > > >>>> >>> >> NIO > > > >>>> >>> >> Implementation Version 3.0.14.GA-redhat-1 > > > >>>> >>> >> 16:54:27,101 INFO [org.jboss.as.server] (Controller Boot > > Thread) > > > >>>> >>> >> JBAS015888: > > > >>>> >>> >> Creating http management service using socket-binding > > > >>>> >>> >> (management-http) > > > >>>> >>> >> 16:54:27,104 INFO [org.jboss.remoting] (MSC service thread > > 1-3) > > > >>>> >>> >> JBoss > > > >>>> >>> >> Remoting version 3.3.5.Final-redhat-1 > > > >>>> >>> >> > > > >>>> >>> >> $ oc logs -f heapster-x2gj2 > > > >>>> >>> >> Endpoint Check in effect. Checking > > > >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status > > > >>>> >>> >> Could not connect to > > > >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status. > > > >>>> >>> >> Curl exit code: 6. Status Code 000 > > > >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is > > not > > > >>>> >>> >> accessible > > > >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying. > > > >>>> >>> >> Could not connect to > > > >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status. > > > >>>> >>> >> Curl exit code: 6. Status Code 000 > > > >>>> >>> >> 'https://hawkular-metrics:443/hawkular/metrics/status' is > > not > > > >>>> >>> >> accessible > > > >>>> >>> >> [HTTP status code: 000. Curl exit code 6]. Retrying. > > > >>>> >>> >> Could not connect to > > > >>>> >>> >> https://hawkular-metrics:443/hawkular/metrics/status. > > > >>>> >>> >> Curl exit code: 6. Status Code 000 > > > >>>> >>> >> > > > >>>> >>> >> > > > >>>> >>> >> $ oc logs -f metrics-deployer-9v7vc > > > >>>> >>> >> > > > >>>> >>> >> ++ oc create -f - > > > >>>> >>> >> serviceaccount "heapster" created > > > >>>> >>> >> service "heapster" created > > > >>>> >>> >> replicationcontroller "heapster" created > > > >>>> >>> >> + echo 'Success!' > > > >>>> >>> >> Success! > > > >>>> >>> >> > > > >>>> >>> >> -- > > > >>>> >>> >> Srinivas Kotaru > > > >>>> >>> >> > > > >>>> >>> >> On 6/13/16, 1:49 PM, "Matt Wringe" <mwri...@redhat.com> > > wrote: > > > >>>> >>> >> > > > >>>> >>> >> > > > > >>>> >>> >> > > > > >>>> >>> >> >----- Original Message ----- > > > >>>> >>> >> >> From: "Srinivas Naga Kotaru (skotaru)" <skot...@cisco.com > > > > > > >>>> >>> >> >> To: users@lists.openshift.redhat.com > > > >>>> >>> >> >> Sent: Monday, June 13, 2016 3:58:12 PM > > > >>>> >>> >> >> Subject: Metrics deployment > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> Hi > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> Am trying to configure metrics in our newly installed > > > >>>> >>> >> >> clusters. Am > > > >>>> >>> >> >> seeing > > > >>>> >>> >> >> below errors once metrics-deploy script was successful. I > > used > > > >>>> >>> >> >> our > > > >>>> >>> >> >> environment specific HAWKULAR_METRICS_HOSTNAME and > > MASTER_URL > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> # oc new-app -f metrics-deployer.yaml > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> Note: customized, CASSANDARA PV, MASTER_URL, and > > > >>>> >>> >> >> HAWKULAR_METRICS_HOSTNAME > > > >>>> >>> >> >> ( > > > >>>> >>> >> >> hard coded as values) > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> template "hawkular-heapster" created > > > >>>> >>> >> >> > > > >>>> >>> >> >> Deploying the Heapster component > > > >>>> >>> >> >> > > > >>>> >>> >> >> ++ echo 'Deploying the Heapster component' > > > >>>> >>> >> >> > > > >>>> >>> >> >> ++ '[' -n '' ']' > > > >>>> >>> >> >> > > > >>>> >>> >> >> ++ oc create -f - > > > >>>> >>> >> >> > > > >>>> >>> >> >> ++ oc process hawkular-heapster -v > > > >>>> >>> >> >> IMAGE_PREFIX= > > registry.access.redhat.com/openshift3/,IMAGE_VERSION=latest,MASTER_URL=https://lae3-alln-int-idev01.cisco.com:443,NODE_ID=nodename > > > >>>> >>> >> >> > > > >>>> >>> >> >> serviceaccount "heapster" created > > > >>>> >>> >> >> > > > >>>> >>> >> >> service "heapster" created > > > >>>> >>> >> >> > > > >>>> >>> >> >> replicationcontroller "heapster" created > > > >>>> >>> >> >> > > > >>>> >>> >> >> + echo 'Success!' > > > >>>> >>> >> >> > > > >>>> >>> >> >> Success! > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> # oc get pods > > > >>>> >>> >> >> > > > >>>> >>> >> >> NAME READY STATUS RESTARTS AGE > > > >>>> >>> >> >> > > > >>>> >>> >> >> hawkular-cassandra-1-9nzio 0/1 ContainerCreating 0 4m > > > >>>> >>> >> >> > > > >>>> >>> >> >> hawkular-metrics-hi7mb 0/1 Running 1 4m > > > >>>> >>> >> >> > > > >>>> >>> >> >> heapster-e8gbu 0/1 Running 2 4m > > > >>>> >>> >> >> > > > >>>> >>> >> >> metrics-deployer-64703 0/1 ContainerCreating 0 3s > > > >>>> >>> >> >> > > > >>>> >>> >> >> metrics-deployer-cd1nf 0/1 Completed 0 5m > > > >>>> >>> >> >> > > > >>>> >>> >> > > > > >>>> >>> >> >It looks like none of your containers are fully up and > > running > > > >>>> >>> >> >yet. > > > >>>> >>> >> > > > > >>>> >>> >> >Without Cassandra running, Hawkular Metrics will not run, > > and > > > >>>> >>> >> >Heapster > > > >>>> >>> >> >will > > > >>>> >>> >> >wait until Hawkular Metrics is fully running. > > > >>>> >>> >> > > > > >>>> >>> >> >Do you see anything in the Cassandra logs? The first step > > will > > > >>>> >>> >> >be to > > > >>>> >>> >> >get > > > >>>> >>> >> >Cassandra running properly. > > > >>>> >>> >> > > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> $ oc logs -f heapster-e8gbu > > > >>>> >>> >> >> > > > >>>> >>> >> >> Endpoint Check in effect. Checking > > > >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status > > > >>>> >>> >> >> > > > >>>> >>> >> >> Could not connect to > > > >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status. > > > >>>> >>> >> >> Curl exit code: 6. Status Code 000 > > > >>>> >>> >> >> > > > >>>> >>> >> >> 'https://hawkular-metrics:443/hawkular/metrics/status' > > is not > > > >>>> >>> >> >> accessible > > > >>>> >>> >> >> [HTTP status code: 000. Curl exit code 6]. Retrying. > > > >>>> >>> >> >> > > > >>>> >>> >> >> Could not connect to > > > >>>> >>> >> >> https://hawkular-metrics:443/hawkular/metrics/status. > > > >>>> >>> >> >> Curl exit code: 6. Status Code 000 > > > >>>> >>> >> > > > > >>>> >>> >> >Heapster waits until Hawkular Metrics is started before > > trying > > > >>>> >>> >> >to > > > >>>> >>> >> >push > > > >>>> >>> >> >metrics to it. The issue that you are seeing is because > > Heapster > > > >>>> >>> >> >could > > > >>>> >>> >> >not > > > >>>> >>> >> >properly connect to Hawkular Metrics. Until the Hawkular > > Metrics > > > >>>> >>> >> >service > > > >>>> >>> >> >is > > > >>>> >>> >> >fully up, Heapster will not be able to connect to it. > > > >>>> >>> >> > > > > >>>> >>> >> > > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> What is the wrong? Why it checking just hawkular-metrics > > > >>>> >>> >> >> rather > > > >>>> >>> >> >> full > > > >>>> >>> >> >> routing > > > >>>> >>> >> >> URL which was provided as HAWKULAR_METRICS_HOSTNAME > > > >>>> >>> >> > > > > >>>> >>> >> >The Hawkular Metrics service has two hostnames: the internal > > > >>>> >>> >> >hostname > > > >>>> >>> >> >used > > > >>>> >>> >> >by the internal components (eg 'hawkular-metrics') and the > > > >>>> >>> >> >external > > > >>>> >>> >> >hostname (eg what is configured via > > HAWKULAR_METRICS_HOSTNAME). > > > >>>> >>> >> >The > > > >>>> >>> >> >OpenShift dns server will resolve hostnames to the name of > > > >>>> >>> >> >services, > > > >>>> >>> >> >which > > > >>>> >>> >> >is where the internal 'hawkular-metrics' comes from. > > > >>>> >>> >> > > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> -- > > > >>>> >>> >> >> > > > >>>> >>> >> >> > > > >>>> >>> >> >> Srinivas Kotaru > > > >>>> >>> >> >> > > > >>>> >>> >> >> _______________________________________________ > > > >>>> >>> >> >> users mailing list > > > >>>> >>> >> >> users@lists.openshift.redhat.com > > > >>>> >>> >> >> > > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > > > >>>> >>> >> >> > > > >>>> >>> >> > > > >>>> >>> >> > > > >>>> >>> > > > >>>> >>> > > > >>>> > > > > >>>> > > > >>>> > > > >> > > > > > > > > > > > > > > _______________________________________________ > > users mailing list > > users@lists.openshift.redhat.com > > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > > > _______________________________________________ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
